datavane / tis

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
https://tis.pub
Apache License 2.0
940 stars 211 forks source link

psgql同步到doris由于数据类型为geometry导致错误 #333

Closed chn-maich closed 1 week ago

chn-maich commented 1 week ago

不小心把测试环境删了,只能文字描述了。 pg库的数据库字段为geometry,将数据同步到doris时出现异常,提示应该是不支持geometry类型。

baisui1981 commented 1 week ago

可以提供一下PG的create table ddl不?

baisui1981 commented 1 week ago
CREATE TABLE test_geometries (
    id SERIAL PRIMARY KEY,
    description TEXT NOT NULL,
    geom geometry(Point, 4326) NOT NULL
);

-- 插入北京天安门位置的点数据
INSERT INTO test_geometries (description, geom)
VALUES ('Tiananmen Square, Beijing', ST_GeomFromText('POINT(116.3975 39.9085)', 4326));

-- 插入纽约时代广场位置的点数据
INSERT INTO test_geometries (description, geom)
VALUES ('Times Square, New York', ST_GeomFromText('POINT(-73.9857 40.7589)', 4326));

-- 插入巴黎埃菲尔铁塔位置的点数据
INSERT INTO test_geometries (description, geom)
VALUES ('Eiffel Tower, Paris', ST_GeomFromText('POINT(2.2945 48.8584)', 4326));

请注意,在实际应用中,如果你的PostgreSQL数据库没有启用postgis扩展,你将无法使用geometry类型和相关函数(如ST_GeomFromText)。确保已经安装并启用了postgis扩展:

CREATE EXTENSION IF NOT EXISTS postgis;
chn-maich commented 1 week ago
CREATE TABLE test_geometries (
    id SERIAL PRIMARY KEY,
    description TEXT NOT NULL,
    geom geometry(Point, 4326) NOT NULL
);

-- 插入北京天安门位置的点数据
INSERT INTO test_geometries (description, geom)
VALUES ('Tiananmen Square, Beijing', ST_GeomFromText('POINT(116.3975 39.9085)', 4326));

-- 插入纽约时代广场位置的点数据
INSERT INTO test_geometries (description, geom)
VALUES ('Times Square, New York', ST_GeomFromText('POINT(-73.9857 40.7589)', 4326));

-- 插入巴黎埃菲尔铁塔位置的点数据
INSERT INTO test_geometries (description, geom)
VALUES ('Eiffel Tower, Paris', ST_GeomFromText('POINT(2.2945 48.8584)', 4326));

请注意,在实际应用中,如果你的PostgreSQL数据库没有启用postgis扩展,你将无法使用geometry类型和相关函数(如ST_GeomFromText)。确保已经安装并启用了postgis扩展:

CREATE EXTENSION IF NOT EXISTS postgis;

可以提供一下PG的create table ddl不?

CREATE TABLE "public"."camera_device2" (
  "id" varchar(255) COLLATE "pg_catalog"."default" NOT NULL,
  "code" varchar(255) COLLATE "pg_catalog"."default",
  "name" varchar(255) COLLATE "pg_catalog"."default",
  "geom" geometry(GEOMETRY),
  CONSTRAINT "camera_device2_pkey" PRIMARY KEY ("id")
)
;

INSERT INTO "public"."camera_device2" ("id", "code", "name", "geom") VALUES ('1', '11', '测试', NULL);

异常信息:

RROR c.a.d.c.s.p.t.StdoutPluginCollector-com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-12], Description:[不支持的数据库类型. 请注意查看 DataX 已经支持的数据库类型以及数据库版本.].  - 您的配置文件中的列配置信息有误. 因为DataX 不支持数据库读取这种字段类型. 字段:[ColumnMetaData{key='geom', type=1111,2147483647,0, index=3, schemaFieldType=null, pk=false}]. 请尝试使用数据库函数将其转换datax支持的类型 或者不同步该字段 .
baisui1981 commented 1 week ago

已经修复了,com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader.Task 的 buildRecord方法中支持的列类型添加Types.OTHER 即可