search

alibaba / DataX

DataX是阿里云DataWorks数据集成的开源版本。
Other
16.03k stars 5.46k forks source link

请问支持人大金仓数据库么？ #531

Open GalaxyCenter opened 4 years ago

GalaxyCenter commented 4 years ago

我按说明用rdbmswriter，然后把jdbc驱动丢到 datax\plugin\writer\rdbmswriter\libs 下面 然后plugin.json 改成 { "name": "rdbmswriter", "class": "com.alibaba.datax.plugin.reader.rdbmswriter.RdbmsWriter", "description": "useScene: prod. mechanism: Jdbc connection using the database, execute select sql, retrieve data from the ResultSet. warn: The more you know about the database, the less problems you encounter.", "developer": "alibaba", "drivers":["dm.jdbc.driver.DmDriver", "com.sybase.jdbc3.jdbc.SybDriver", "com.edb.Driver", "com.kingbase.Driver"] }

我生成的job json如下： "writer": { "parameter": { "password": "1234", "column": ["Data_Store_Id", "Data_Store_Nm", "Data_Store_Type_Id", "Data_Store_Param", "Data_Srcid", "Tnmtid", "Data_Store_Dev_Id", "Data_Store_Status", "Create_Dt", "Update_Dt"], "connection": [{ "jdbcUrl": "jdbc:kingbase://127.0.0.1:54321/kingbase", "table": ["o_bj01_000_data_store"] }], "username": "SYSTEM" }, "name": "rdbmswriter" }

====================== 执行的时候 Description:[连接数据库失败. 请检查您的 账号、密码、数据库名称、IP、Port或者向 DBA 寻求帮助(注意网络环境).]. - 具体错误信息为：java.sql.SQLException: No suitable driver found for jdbc:kingbase://127.0.0.1:54321/kingbase

2019-12-09 10:19:27.108 [job-0] ERROR RetryUtil - Exception when calling callable, 即将尝试执行第1次重试.本次重试计划等待[1000]ms,实际等待[1001]ms, 异常Msg:[Code:[DBUtilErrorCode-10], Description:[连接数据库失败. 请检查您的 账号、密码、数据库名称、IP、Port或者向 DBA 寻求帮助(注意网络环境).]. - 具体错误信息为：java.sql.SQLException: No suitable driver found for jdbc:kingbase://127.0.0.1:54321/kingbase]

hwz9612 commented 4 years ago

支持人大金仓数据库，你这应该是驱动找不到的表现，具体地除了把jar放到rdmpswriter和rdmpsreader的libs下外，可以把jar包放到{YOUR_DATAX_HOME}/target/datax/datax/libs下面，另外看你配置"jdbcUrl": "jdbc:kingbase://127.0.0.1:54321/kingbase",人大金仓数据库V8需要把"jdbcUrl": "jdbc:kingbase8://127.0.0.1:54321/kingbase"

cy1331250 commented 4 years ago

可以支持吗

hwz9612 commented 4 years ago

可以支持

---原始邮件--- 发件人: "cy1331250"<notifications@github.com> 发送时间: 2020年8月11日(周二) 中午11:56 收件人: "alibaba/DataX"<DataX@noreply.github.com>; 抄送: "Comment"<comment@noreply.github.com>;"hwz9612"<1056957558@qq.com>; 主题: Re: [alibaba/DataX] 请问支持人大金仓数据库么？ (#531)

可以支持吗

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

cy1331250 commented 4 years ago

是使用RDBMS吗读和写吗

------------------ 原始邮件 ------------------ 发件人: "alibaba/DataX" <notifications@github.com>; 发送时间: 2020年8月11日(星期二) 中午12:05 收件人: "alibaba/DataX"<DataX@noreply.github.com>; 抄送: "开三轮拉宝马"<297041179@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [alibaba/DataX] 请问支持人大金仓数据库么？ (#531)

可以支持

---原始邮件--- 发件人: "cy1331250"<notifications@github.com&gt; 发送时间: 2020年8月11日(周二) 中午11:56 收件人: "alibaba/DataX"<DataX@noreply.github.com&gt;; 抄送: "Comment"<comment@noreply.github.com&gt;;"hwz9612"<1056957558@qq.com&gt;; 主题: Re: [alibaba/DataX] 请问支持人大金仓数据库么？ (#531)

可以支持吗

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

hwz9612 commented 4 years ago

是的

------------------ 原始邮件 ------------------ 发件人: "alibaba/DataX" <notifications@github.com>; 发送时间: 2020年8月11日(星期二) 中午12:06 收件人: "alibaba/DataX"<DataX@noreply.github.com>; 抄送: "●威╲振&"<1056957558@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [alibaba/DataX] 请问支持人大金仓数据库么？ (#531)

是使用RDBMS吗读和写吗

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "alibaba/DataX" <notifications@github.com&gt;; 发送时间:&nbsp;2020年8月11日(星期二) 中午12:05 收件人:&nbsp;"alibaba/DataX"<DataX@noreply.github.com&gt;; 抄送:&nbsp;"开三轮拉宝马"<297041179@qq.com&gt;;"Comment"<comment@noreply.github.com&gt;; 主题:&nbsp;Re: [alibaba/DataX] 请问支持人大金仓数据库么？ (#531)

可以支持

---原始邮件---
发件人: "cy1331250"<notifications@github.com&amp;gt;
发送时间: 2020年8月11日(周二) 中午11:56
收件人: "alibaba/DataX"<DataX@noreply.github.com&amp;gt;;
抄送: "Comment"<comment@noreply.github.com&amp;gt;;"hwz9612"<1056957558@qq.com&amp;gt;;
主题: Re: [alibaba/DataX] 请问支持人大金仓数据库么？ (#531)

可以支持吗

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

PastWind-Zhou commented 2 years ago

支持人大金仓数据库，你这应该是驱动找不到的表现，具体地除了把jar放到rdmpswriter和rdmpsreader的libs下外，可以把jar包放到{YOUR_DATAX_HOME}/target/datax/datax/libs下面，另外看你配置"jdbcUrl": "jdbc:kingbase://127.0.0.1:54321/kingbase",人大金仓数据库V8需要把"jdbcUrl": "jdbc:kingbase8://127.0.0.1:54321/kingbase"

您好！请问读取金仓写入mysql怎么实现增量同步呢？

huliang123456 commented 2 years ago

大佬 我金仓的报错 提示 2022-07-23 14:26:35.300 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null 2022-07-23 14:26:35.301 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0 2022-07-23 14:26:35.301 [main] INFO JobContainer - DataX jobContainer starts job. 2022-07-23 14:26:35.304 [main] INFO JobContainer - Set jobId = 0 2022-07-23 14:26:40.799 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:kingbase8://10.10.10.191:54321/lightcode. 2022-07-23 14:26:40.845 [job-0] INFO OriginalConfPretreatmentUtil - table:[test] all columns:[ name,pwd ]. 2022-07-23 14:26:40.878 [job-0] ERROR JobContainer - Exception when job run com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-01], Description:[获取表字段相关信息失败.]. - 获取表:test 的字段的元信息时失败. 请联系 DBA 核查该库、表信息. - com.kingbase8.util.KSQLException: 错误: 操作符不存在: ` varchar Hint: 没有匹配指定名称和参数类型的操作符. 您也许需要增加明确的类型转换. Position: 8 At Line: 1, Line Position: 8 at com.kingbase8.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2679) at com.kingbase8.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2367) at com.kingbase8.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:318) at com.kingbase8.jdbc.KbStatement.executeInternal(KbStatement.java:472) at com.kingbase8.jdbc.KbStatement.execute(KbStatement.java:392) at com.kingbase8.jdbc.KbStatement.executeWithFlags(KbStatement.java:333) at com.kingbase8.jdbc.KbStatement.executeCachedSql(KbStatement.java:319) at com.kingbase8.jdbc.KbStatement.executeWithFlags(KbStatement.java:296) at com.kingbase8.jdbc.KbStatement.executeQuery(KbStatement.java:246) at com.alibaba.datax.plugin.rdbms.util.DBUtil.getColumnMetaData(DBUtil.java:563) at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:125) at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:140) at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:35) at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41) at com.alibaba.datax.plugin.reader.rdbmswriter.RdbmsWriter$Job.init(RdbmsWriter.java:38) at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704) at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304) at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) at com.alibaba.datax.core.Engine.start(Engine.java:92) at com.alibaba.datax.core.Engine.entry(Engine.java:171) at com.alibaba.datax.core.Engine.main(Engine.java:204)

    at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:33) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.rdbms.util.DBUtil.getColumnMetaData(DBUtil.java:575) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:125) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:140) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:35) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.reader.rdbmswriter.RdbmsWriter$Job.init(RdbmsWriter.java:38) ~[rdbmswriter-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]

Caused by: com.kingbase8.util.KSQLException: 错误: 操作符不存在: ` varchar Hint: 没有匹配指定名称和参数类型的操作符. 您也许需要增加明确的类型转换. Position: 8 At Line: 1, Line Position: 8 at com.kingbase8.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2679) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.kingbase8.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2367) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.kingbase8.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:318) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.kingbase8.jdbc.KbStatement.executeInternal(KbStatement.java:472) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.kingbase8.jdbc.KbStatement.execute(KbStatement.java:392) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.kingbase8.jdbc.KbStatement.executeWithFlags(KbStatement.java:333) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.kingbase8.jdbc.KbStatement.executeCachedSql(KbStatement.java:319) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.kingbase8.jdbc.KbStatement.executeWithFlags(KbStatement.java:296) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.kingbase8.jdbc.KbStatement.executeQuery(KbStatement.java:246) ~[kingbase8-8.6.0.jar:V008R006B0301] at com.alibaba.datax.plugin.rdbms.util.DBUtil.getColumnMetaData(DBUtil.java:563) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] ... 11 common frames omitted 2022-07-23 14:26:40.884 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00% 2022-07-23 14:26:40.885 [job-0] ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是: com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-01], Description:[获取表字段相关信息失败.]. - 获取表:test 的字段的元信息时失败. 请联系 DBA 核查该库、表信息. - com.kingbase8.util.KSQLException: 错误: 操作符不存在: ` varchar Hint: 没有匹配指定名称和参数类型的操作符. 您也许需要增加明确的类型转换. Position: 8 At Line: 1, Line Position: 8 at com.kingbase8.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2679) at com.kingbase8.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2367) at com.kingbase8.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:318) at com.kingbase8.jdbc.KbStatement.executeInternal(KbStatement.java:472) at com.kingbase8.jdbc.KbStatement.execute(KbStatement.java:392) at com.kingbase8.jdbc.KbStatement.executeWithFlags(KbStatement.java:333) at com.kingbase8.jdbc.KbStatement.executeCachedSql(KbStatement.java:319) at com.kingbase8.jdbc.KbStatement.executeWithFlags(KbStatement.java:296) at com.kingbase8.jdbc.KbStatement.executeQuery(KbStatement.java:246) at com.alibaba.datax.plugin.rdbms.util.DBUtil.getColumnMetaData(DBUtil.java:563) at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:125) at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:140) at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:35) at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41) at com.alibaba.datax.plugin.reader.rdbmswriter.RdbmsWriter$Job.init(RdbmsWriter.java:38) at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704) at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304) at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) at com.alibaba.datax.core.Engine.start(Engine.java:92) at com.alibaba.datax.core.Engine.entry(Engine.java:171) at com.alibaba.datax.core.Engine.main(Engine.java:204)

    at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:33)
    at com.alibaba.datax.plugin.rdbms.util.DBUtil.getColumnMetaData(DBUtil.java:575)
    at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:125)
    at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.dealColumnConf(OriginalConfPretreatmentUtil.java:140)
    at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:35)
    at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41)
    at com.alibaba.datax.plugin.reader.rdbmswriter.RdbmsWriter$Job.init(RdbmsWriter.java:38)
    at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
    at com.alibaba.datax.core.Engine.start(Engine.java:92)
    at com.alibaba.datax.core.Engine.entry(Engine.java:171)
    at com.alibaba.datax.core.Engine.main(Engine.java:204)

Caused by: com.kingbase8.util.KSQLException: 错误: 操作符不存在: ` varchar Hint: 没有匹配指定名称和参数类型的操作符. 您也许需要增加明确的类型转换. Position: 8 At Line: 1, Line Position: 8 at com.kingbase8.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2679) at com.kingbase8.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2367) at com.kingbase8.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:318) at com.kingbase8.jdbc.KbStatement.executeInternal(KbStatement.java:472) at com.kingbase8.jdbc.KbStatement.execute(KbStatement.java:392) at com.kingbase8.jdbc.KbStatement.executeWithFlags(KbStatement.java:333) at com.kingbase8.jdbc.KbStatement.executeCachedSql(KbStatement.java:319) at com.kingbase8.jdbc.KbStatement.executeWithFlags(KbStatement.java:296) at com.kingbase8.jdbc.KbStatement.executeQuery(KbStatement.java:246) at com.alibaba.datax.plugin.rdbms.util.DBUtil.getColumnMetaData(DBUtil.java:563) ... 11 more

我就指定了两个

{ "job":{ "setting":{ "speed":{ "channel":2, "byte":1048576 }, "errorLimit":{ "record":0, "percentage":0.02 } }, "content":[ { "reader":{ "name":"rdbmsreader", "parameter":{ "username":"lightcode", "password":"simayi@123", "splitPk":"", "connection":[ { "querySql":[ "select name,archives_no from archives_basics" ], "jdbcUrl":[ "jdbc:kingbase8://10.10.10.191:54321/lightcode" ] } ] } }, "writer":{ "name":"rdbmswriter", "parameter":{ "username":"lightcode", "password":"simayi@123", "column":[ "name", "pwd" ], "connection":[ { "table":[ "test" ], "jdbcUrl":"jdbc:kingbase8://10.10.10.191:54321/lightcode" } ] } } } ] }, "core":{ "transport":{ "channel":{ "speed":{ "byte":524288 } } } } }

huliang123456 commented 2 years ago

抽取的两个字段 都是 varchar 写入的两个字段也是一样的

yaoleifly commented 2 years ago

路过

huliang123456 commented 2 years ago

可以支持

------------------ 原始邮件 ------------------ 发件人: "alibaba/DataX" @.>; 发送时间: 2022年11月17日(星期四) 下午2:42 @.>; @.**@.>; 主题: Re: [alibaba/DataX] 请问支持人大金仓数据库么？ (#531)

路过

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

jinruojia2000 commented 10 months ago

所以说现在的金仓支持的是8而不是7哈？

ARS-Rocket commented 5 months ago

@huliang123456 你好，我碰到了和你一样的问题，我的kingbase是v8r3的，用datax自带的kingbaseeswriter，报错“Caused by: com.kingbase8.util.KSQLException: 错误: 操作符不存在: ` varchar Hint: 没有匹配指定名称和参数类型的操作符. 您也许需要增加明确的类型转换.”想问下你是怎么解决的