search

alibaba / DataX

DataX是阿里云DataWorks数据集成的开源版本。
Other
16k stars 5.46k forks source link

使用 Datax 抽取 Tdengine数据时报错 #1846

Open acaee opened 1 year ago

acaee commented 1 year ago

报错信息如下：

[root@xxxxx ]# datax.py ./t2stream-2.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2023-07-19 14:10:07.654 [main] INFO  MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-07-19 14:10:07.656 [main] INFO  MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2023-07-19 14:10:07.695 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-07-19 14:10:07.699 [main] INFO  Engine - the machine info  => 

        osInfo: Oracle Corporation 1.8 25.371-b11
        jvmInfo:        Linux amd64 3.10.0-1160.el7.x86_64
        cpu num:        8

        totalPhysicalMemory:    -0.00G
        freePhysicalMemory:     -0.00G
        maxFileDescriptorCount: -1
        currentOpenFileDescriptorCount: -1

        GC Names        [PS MarkSweep, PS Scavenge]

        MEMORY_NAME                    | allocation_size                | init_size                      
        PS Eden Space                  | 256.00MB                       | 256.00MB                       
        Code Cache                     | 240.00MB                       | 2.44MB                         
        Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
        PS Survivor Space              | 42.50MB                        | 42.50MB                        
        PS Old Gen                     | 683.00MB                       | 683.00MB                       
        Metaspace                      | -0.00MB                        | 0.00MB                         

2023-07-19 14:10:07.710 [main] INFO  Engine - 
{
        "content":[
                {
                        "reader":{
                                "name":"tdenginereader",
                                "parameter":{
                                        "username":"root",
                                        "password":"********",
                                        "connection":[
                                                {
                                                        "querySql":[
                                                                "select * from meters where ts > '2017-07-14 00:00:00' and ts < '2017-07-15 00:00:00'"
                                                        ],
                                                        "jdbcUrl":[
                                                                "jdbc:TAOS-RS://127.0.0.1:6041/test"
                                                        ]
                                                }
                                        ]
                                }
                        },
                        "writer":{
                                "name":"streamwriter",
                                "parameter":{
                                        "encoding":"UTF-8",
                                        "print":true
                                }
                        }
                }
        ],
        "setting":{
                "speed":{
                        "channel":1
                }
        }
}

2023-07-19 14:10:07.726 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false
2023-07-19 14:10:07.726 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-07-19 14:10:07.727 [main] INFO  JobContainer - Set jobId = 0
2023-07-19 14:10:07.738 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-07-19 14:10:07.739 [job-0] INFO  JobContainer - DataX Reader.Job [tdenginereader] do prepare work .
2023-07-19 14:10:07.739 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2023-07-19 14:10:07.739 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-07-19 14:10:07.739 [job-0] INFO  JobContainer - Job set Channel-Number to 1 channels.
2023-07-19 14:10:07.740 [job-0] INFO  TDengineReader$Job - Configuration: [{"username":"root","password":"taosdata","jdbcUrl":"jdbc:TAOS-RS://127.0.0.1:6041/test","querySql":["select * from meters where ts > '2017-07-14 00:00:00' and ts < '2017-07-15 00:00:00'"]}]
2023-07-19 14:10:07.740 [job-0] INFO  JobContainer - DataX Reader.Job [tdenginereader] splits to [1] tasks.
2023-07-19 14:10:07.740 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2023-07-19 14:10:07.765 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-07-19 14:10:07.767 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-07-19 14:10:07.769 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-07-19 14:10:07.776 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-07-19 14:10:07.783 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-07-19 14:10:07.784 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-07-19 14:10:07.819 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-07-19 14:10:08.240 [0-0-0-reader] ERROR ReaderRunner - Reader runner Received Exceptions:
java.lang.NullPointerException: null
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71) ~[taos-jdbcdriver-2.0.39.jar:na]
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122) ~[taos-jdbcdriver-2.0.39.jar:na]
        at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_371]
        at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[na:1.8.0_371]
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159) ~[tdenginereader-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at java.lang.Thread.run(Thread.java:750) [na:1.8.0_371]
Exception in thread "taskGroup-0" com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .].  - java.lang.NullPointerException
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71)
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159)
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
        at java.lang.Thread.run(Thread.java:750)
 - java.lang.NullPointerException
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71)
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159)
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
        at java.lang.Thread.run(Thread.java:750)

        at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:48)
        at com.alibaba.datax.core.taskgroup.TaskGroupContainer.start(TaskGroupContainer.java:195)
        at com.alibaba.datax.core.taskgroup.runner.TaskGroupContainerRunner.run(TaskGroupContainerRunner.java:24)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71)
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159)
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
        ... 1 more
2023-07-19 14:10:17.783 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-07-19 14:10:17.785 [job-0] ERROR JobContainer - 运行scheduler 模式[standalone]出错.
2023-07-19 14:10:17.787 [job-0] ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .].  - java.lang.NullPointerException
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71)
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159)
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
        at java.lang.Thread.run(Thread.java:750)
 - java.lang.NullPointerException
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71)
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159)
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
        at java.lang.Thread.run(Thread.java:750)

        at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:48) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.Engine.start(Engine.java:86) [datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.Engine.entry(Engine.java:168) [datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.Engine.main(Engine.java:201) [datax-core-0.0.1-SNAPSHOT.jar:na]
java.lang.NullPointerException: null
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71) ~[na:na]
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122) ~[na:na]
        at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_371]
        at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[na:1.8.0_371]
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159) ~[na:na]
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_371]
2023-07-19 14:10:17.789 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-07-19 14:10:17.790 [job-0] ERROR Engine - 

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .].  - java.lang.NullPointerException
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71)
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159)
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
        at java.lang.Thread.run(Thread.java:750)
 - java.lang.NullPointerException
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71)
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159)
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
        at java.lang.Thread.run(Thread.java:750)

        at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:48)
        at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.dealFailedStat(ProcessInnerScheduler.java:39)
        at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:99)
        at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535)
        at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119)
        at com.alibaba.datax.core.Engine.start(Engine.java:86)
        at com.alibaba.datax.core.Engine.entry(Engine.java:168)
        at com.alibaba.datax.core.Engine.main(Engine.java:201)
Caused by: java.lang.NullPointerException
        at com.taosdata.jdbc.rs.RestfulStatement.execute(RestfulStatement.java:71)
        at com.taosdata.jdbc.rs.RestfulDriver.connect(RestfulDriver.java:122)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at com.alibaba.datax.plugin.reader.TDengineReader$Task.init(TDengineReader.java:159)
        at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
        at java.lang.Thread.run(Thread.java:750)

这是导出文件配置：

{
  "job": {
    "content": [
      {
        "reader": {
          "name": "tdenginereader",
          "parameter": {
            "username": "root",
            "password": "taosdata",
            "connection": [
              {
                "querySql": [
                  "select * from meters where ts > '2017-07-14 00:00:00' and ts < '2017-07-15 00:00:00'"
                ],
                "jdbcUrl": [
                  "jdbc:TAOS-RS://127.0.0.1:6041/test"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "streamwriter",
          "parameter": {
            "encoding": "UTF-8",
            "print": true
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "channel": 1
      }
    }
  }
}
subs-sun commented 1 year ago

请问一下，您在使用datax导出数据的时候有没有遇到ts丢失毫秒值的问题啊