search

alibaba / DataX

DataX是阿里云DataWorks数据集成的开源版本。
Other
15.94k stars 5.43k forks source link

opentsdbreader 配置导出多列时，只导出了第一列 #1969

Closed zipper-meng closed 1 year ago

zipper-meng commented 1 year ago

问题

当我从 opentsdb 中一次性导出多个 metric 时，DataX 只为第一个 metric 创建了 Task，第二个以及之后的 metric 没有创建 Task。

配置文件

我想要导出 cpu_usage_system 与 cpu_usage_user 两个 metrics。

{
  "job": {
    "content": [
      {
        "reader": {
          "name": "opentsdbreader",
          "parameter": {
            "endpoint": "http://127.0.0.1:4242",
            "column": [
              "cpu_usage_system",
              "cpu_usage_user"
            ],
            "beginDateTime": "2023-10-26 00:00:00",
            "endDateTime": "2023-10-27 00:00:00"
          }
        },
        "writer": {
          "name": "txtfilewriter",
          "parameter": {
            "path": "/home/fedora_38_user/datax",
            "fileName": "result.txt",
            "writeMode": "truncate",
            "dateFormat": "yyyy-MM-dd"
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "channel": 1
      }
    }
  }
}

日志

被折叠的日志

```log 2023-10-26 19:12:28.227 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false 2023-10-26 19:12:28.227 [main] INFO JobContainer - DataX jobContainer starts job. 2023-10-26 19:12:28.228 [main] INFO JobContainer - Set jobId = 0 2023-10-26 19:12:28.323 [job-0] INFO JobContainer - jobContainer starts to do prepare ... 2023-10-26 19:12:28.323 [job-0] INFO JobContainer - DataX Reader.Job [opentsdbreader] do prepare work . 2023-10-26 19:12:28.323 [job-0] INFO JobContainer - DataX Writer.Job [txtfilewriter] do prepare work . 2023-10-26 19:12:28.324 [job-0] INFO TxtFileWriter$Job - 由于您配置了writeMode truncate, 开始清理 [/home/fedora_38_user/datax] 下面以 [result.txt] 开头的内容 2023-10-26 19:12:28.325 [job-0] INFO JobContainer - jobContainer starts to do split ... 2023-10-26 19:12:28.325 [job-0] INFO JobContainer - Job set Channel-Number to 10 channels. 2023-10-26 19:12:28.347 [job-0] INFO OpenTSDBReader$Job - Configuration for column: cpu_usage_system 2023-10-26 19:12:28.364 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698249600000,"endDateTime":1698253199999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.366 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698253200000,"endDateTime":1698256799999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.366 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698256800000,"endDateTime":1698260399999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.366 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698260400000,"endDateTime":1698263999999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.366 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698264000000,"endDateTime":1698267599999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.367 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698267600000,"endDateTime":1698271199999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.367 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698271200000,"endDateTime":1698274799999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.367 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698274800000,"endDateTime":1698278399999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.367 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698278400000,"endDateTime":1698281999999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.367 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698282000000,"endDateTime":1698285599999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.367 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698285600000,"endDateTime":1698289199999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.367 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698289200000,"endDateTime":1698292799999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.368 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698292800000,"endDateTime":1698296399999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.368 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698296400000,"endDateTime":1698299999999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.368 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698300000000,"endDateTime":1698303599999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.368 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698303600000,"endDateTime":1698307199999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.368 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698307200000,"endDateTime":1698310799999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.368 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698310800000,"endDateTime":1698314399999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.369 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698314400000,"endDateTime":1698317999999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.369 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698318000000,"endDateTime":1698321599999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.369 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698321600000,"endDateTime":1698325199999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.369 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698325200000,"endDateTime":1698328799999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.369 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698328800000,"endDateTime":1698332399999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.370 [job-0] INFO OpenTSDBReader$Job - Configuration: {"internal":{"endpoint":"http://127.0.0.1:4242","column":["cpu_usage_system"],"beginDateTime":1698332400000,"endDateTime":1698335999999},"keys":["endpoint","column[0]","beginDateTime","endDateTime"],"secretKeyPathSet":[]} 2023-10-26 19:12:28.370 [job-0] INFO OpenTSDBReader$Job - Configuration for column: cpu_usage_user 2023-10-26 19:12:28.370 [job-0] INFO JobContainer - DataX Reader.Job [opentsdbreader] splits to [24] tasks. 2023-10-26 19:12:28.370 [job-0] INFO TxtFileWriter$Job - begin do split... 2023-10-26 19:12:28.381 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__fc50559d_f5b7_4fd1_bba7_8eeb03f7f15a] 2023-10-26 19:12:28.381 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__1472556c_52d5_41f7_8da4_d54f2900c2fe] 2023-10-26 19:12:28.381 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__e754d67c_3c7e_4420_b6de_e007ecc29630] 2023-10-26 19:12:28.381 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__427ea498_eb5b_4ad4_845d_52c48220498f] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__9ec9949b_8e96_4306_b106_f65c56d84ceb] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__435da16a_30a9_4ced_86b4_c8c23e9638a0] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__b21c14f0_828d_453f_b411_880b5e5b3e7d] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__5e4c3996_41dc_4303_847c_31f739e96949] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__0b1b580e_5ca6_446e_a211_c72562d348a9] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__43c994e6_ee7d_4f64_a4f9_874bf84b1350] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__3cfdfd27_51c4_40e8_a907_86db09eff605] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__cb83d669_1323_463a_b891_1cf7924ecc43] 2023-10-26 19:12:28.382 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__81ada6ba_6978_463a_b498_97e3cf922808] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__a073c6f1_41c3_4397_9fd5_181396838984] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__3fc85726_e7e1_4c10_a4c6_8b88d15de141] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__5a9697da_abc5_48bf_a4a4_ebfeaef0d60d] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__b3dc7a3b_60dc_474a_8daa_9df14d6e2066] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__6fca6a6f_21a7_4bf2_a54d_b77bfce3da93] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__2891cd75_142f_4b90_8121_49177d3a578b] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__03ebde6f_1bda_49f4_a040_41fac90a15db] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__89d58531_da1d_40dd_96c5_dcaac84d8101] 2023-10-26 19:12:28.383 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__751fcbbc_1a83_4e51_a7fc_cde0975addeb] 2023-10-26 19:12:28.384 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__a013d443_f439_4cc7_9a37_96f8bdffd62b] 2023-10-26 19:12:28.384 [job-0] INFO TxtFileWriter$Job - splited write file name:[result.txt__c9fb4309_f4d8_492a_b723_56cdf88e154a] 2023-10-26 19:12:28.384 [job-0] INFO TxtFileWriter$Job - end do split. 2023-10-26 19:12:28.384 [job-0] INFO JobContainer - DataX Writer.Job [txtfilewriter] splits to [24] tasks. 2023-10-26 19:12:28.391 [job-0] INFO JobContainer - jobContainer starts to do schedule ... 2023-10-26 19:12:28.398 [job-0] INFO JobContainer - Scheduler starts [2] taskGroups. 2023-10-26 19:12:28.401 [job-0] INFO JobContainer - Running by standalone Mode. ```

其他

我想这是因为下面的代码导致的，当迭代到下一个 column 时，startDateTime 没有被重置。

https://github.com/alibaba/DataX/blob/86b7935bb4c0570895e6ab97d4e2fe50435f50cb/opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/OpenTSDBReader.java#L143-L162

zipper-meng commented 1 year ago

我看到前面已经有人提出了相同的 Issue #455, 所以我要关闭这个 Issue。