datavane / tis

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
https://tis.pub
Apache License 2.0
1.04k stars 221 forks source link

dolphinscheduler任务同步成功,执行失败 #389

Closed iLay1678 closed 2 weeks ago

iLay1678 commented 2 weeks ago
java.lang.AbstractMethodError: Receiver class com.qlangtech.tis.plugin.dolphinscheduler.task.TISDatasyncTaskChannel does not define or inherit an implementation of the resolved method abstract parseParameters(Ljava/lang/String;)Lorg/apache/dolphinscheduler/plugin/task/api/parameters/AbstractParameters; of interface org.apache.dolphinscheduler.plugin.task.api.TaskChannel.
    at org.apache.dolphinscheduler.server.worker.utils.TaskExecutionContextUtils.downloadResourcesIfNeeded(TaskExecutionContextUtils.java:71)
    at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecutor.beforeExecute(WorkerTaskExecutor.java:226)
    at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecutor.run(WorkerTaskExecutor.java:165)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)

确认已替换worker节点libs中的jar包

iLay1678 commented 2 weeks ago

另外问一下支持的dolphinscheduler版本,但是dolphinscheduler jdk版本是8,修改的dolphinscheduler-task-datasync是用jdk11编译的

baisui1981 commented 2 weeks ago

确认已替换worker节点libs中的jar包

iLay1678 commented 2 weeks ago

使用jdk11会报Could not initialize class org.apache.dolphinscheduler.extract.base.client.SingletonJdkDynamicRpcClientProxyFactory

baisui1981 commented 2 weeks ago

使用jdk11会报Could not initialize class org.apache.dolphinscheduler.extract.base.client.SingletonJdkDynamicRpcClientProxyFactory

你的DS 啥版本的?

iLay1678 commented 2 weeks ago

docker部署的3.2.2,针对worker容器做了一些映射

dolphinscheduler-worker:
    image: ${HUB}/dolphinscheduler-worker:${TAG}
    profiles: ["all"]
    env_file: .env
    healthcheck:
      test: [ "CMD", "curl", "http://localhost:1235/actuator/health" ]
      interval: 30s
      timeout: 5s
      retries: 3
    depends_on:
      dolphinscheduler-zookeeper:
        condition: service_healthy
    volumes:
      - ./data/worker:/tmp/dolphinscheduler
      - ./logs:/opt/dolphinscheduler/logs
      - ./shared:/opt/soft
      - ./resource:/dolphinscheduler
      - ./worker-libs:/opt/dolphinscheduler/libs
      - ./worker-conf:/opt/dolphinscheduler/conf
      - ./jdk/jdk-11:/opt/java/openjdk

环境配置里设置了下java_home,现在能运行了,但是下载依赖报错

java.lang.RuntimeException: may be there is some errors when connect to TIS host, you would better to deploy the local infrastructure to enable TIS data synchronize by manual
        at com.qlangtech.tis.plugin.dolphinscheduler.task.TISDatasyncTask.init(TISDatasyncTask.java:101)
        at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecutor.beforeExecute(WorkerTaskExecutor.java:239)
        at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecutor.run(WorkerTaskExecutor.java:164)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.qlangtech.tis.lang.TisException: maxRetry:1,url:http://mirror.qlangtech.com/4.0.1/tis/tis-data.tar.gz
        at com.qlangtech.tis.lang.TisException.create(TisException.java:171)
        at com.qlangtech.tis.manage.common.ConfigFileContext.processContent(ConfigFileContext.java:136)
        at com.qlangtech.tis.manage.common.ConfigFileContext.processContent(ConfigFileContext.java:112)
        at com.qlangtech.tis.manage.common.ConfigFileContext.processContent(ConfigFileContext.java:108)
        at com.qlangtech.tis.manage.common.HttpUtils.get(HttpUtils.java:109)
        at com.qlangtech.tis.extension.model.UpdateCenter.copyTarToLocal(UpdateCenter.java:171)
        at com.qlangtech.tis.extension.model.UpdateCenter.copyDataTarToLocal(UpdateCenter.java:157)
        at com.qlangtech.tis.plugin.dolphinscheduler.task.TISDatasyncTask.init(TISDatasyncTask.java:80)
        ... 5 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.base/java.net.SocketInputStream.socketRead0(Native Method)
        at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
        at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:746)
        at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1604)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
        at com.qlangtech.tis.manage.common.ConfigFileContext.getNetInputStream(ConfigFileContext.java:183)
        at com.qlangtech.tis.manage.common.ConfigFileContext.processContent(ConfigFileContext.java:120)
baisui1981 commented 2 weeks ago

docker 所在网络环境能连公网不?ds 运行是会去公网下载 资源包 http://mirror.qlangtech.com/4.0.1/tis/tis-data.tar.gz

iLay1678 commented 2 weeks ago

可以

baisui1981 commented 2 weeks ago

可以

未必 ,你进入docker 容器 执行一下 wget http://mirror.qlangtech.com/4.0.1/tis/tis-data.tar.gz 试试看

iLay1678 commented 2 weeks ago

可以了,经排查发现是docker网络mtu问题,OpenStack开的机器mtu默认是1450,docker网络mtu是1500导致read timeout

baisui1981 commented 2 weeks ago

可以了,经排查发现是docker网络mtu问题,OpenStack开的机器mtu默认是1450,docker网络mtu是1500导致read timeout

完美,有啥问题随时交流。多提宝贵意见 :smiley: