search

datavane / tis

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
https://tis.pub
Apache License 2.0
1.04k stars 222 forks source link

4.0.0版本，从sqlserver-->doris，定时任务在运行一段时间后，出现同步失败，后续手动触发所有任务失败，日志无输出。 #393

Closed zzxVv closed 6 days ago

zzxVv commented 6 days ago

具体报错如下： 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 INFO c.a.d.c.s.c.c.j.StandAloneJobContainerCommunicator-Total 50112 records, 6762112 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1.688s | All Task WaitReaderTime 3.961s | Percentage 0.00% 11-19 04:15 10.244.1.16 taskid:5098,tablename:t_wl_cgdd_htxx_11.json,read:50112,all:176679 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 INFO c.a.d.c.s.c.c.j.StandAloneJobContainerCommunicator-Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00% 11-19 04:15 10.244.1.16 taskid:5098,tablename:t_fgs_xs_gj_12.json,read:0,all:585944 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 INFO c.a.d.c.s.c.c.j.StandAloneJobContainerCommunicator-Total 50112 records, 6762112 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 1.688s | All Task WaitReaderTime 3.961s | Percentage 0.00% 11-19 04:15 10.244.1.16 taskid:5098,tablename:t_wl_cgdd_htxx_11.json,read:50112,all:176679 11-19 04:15 10.244.1.16 INFO c.a.d.c.s.c.c.j.StandAloneJobContainerCommunicator-Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00% 11-19 04:15 10.244.1.16 taskid:5098,tablename:t_fgs_xs_gj_12.json,read:0,all:585944 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 INFO c.a.d.c.s.c.c.j.StandAloneJobContainerCommunicator-Total 118176 records, 17537917 bytes | Speed 3.43MB/s, 22688 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 6.423s | All Task WaitReaderTime 5.654s | Percentage 0.00% 11-19 04:15 10.244.1.16 INFO c.a.d.c.s.c.c.j.StandAloneJobContainerCommunicator-Total 105152 records, 5350850 bytes | Speed 1.70MB/s, 35050 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 6.156s | All Task WaitReaderTime 1.846s | Percentage 0.00% 11-19 04:15 10.244.1.16 taskid:5098,tablename:t_fgs_xs_gj_12.json,read:105152,all:585944 11-19 04:15 10.244.1.16 taskid:5098,tablename:t_wl_cgdd_htxx_11.json,read:118176,all:176679 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml 11-19 04:15 10.244.1.16 WARN com.qlangtech.tis.plugin.PluginStore-target xstream file is not exist:/opt/data/tis/cfg_repo/tis_plugin_config/ap/wmsHot_sync_WMSODS_per_5_mins/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml 11-19 04:15 10.244.1.16 WARN org.apache.velocity.deprecation-configuration key 'resource.loader' has been deprecated in favor of 'resource.loaders' 11-19 04:15 10.244.1.16 WARN org.apache.velocity.deprecation-configuration key 'tisLoader.resource.loader.class' has been deprecated in favor of 'resource.loader.tisLoader.class' 11-19 04:15 10.244.1.16 WARN org.apache.velocity.deprecation-configuration key 'resource.loader' has been deprecated in favor of 'resource.loaders' 11-19 04:15 10.244.1.16 WARN org.apache.velocity.deprecation-configuration key 'tisLoader.resource.loader.class' has been deprecated in favor of 'resource.loader.tisLoader.class' 同时，还存在bug，多个定时任务在相同时间无法并发执行，只能顺序执行。powerjob和tis均部署在同一个k8s集群中。

zzxVv commented 6 days ago

再排查了一下，发现是powerjob-work启不起来，server可正常运行。且work有多次重启动作。