apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.86k stars 4.62k forks source link

[Bug] zookeeper EndOfStreamException when using 2.0.5 #8942

Closed lordk911 closed 2 years ago

lordk911 commented 2 years ago

Search before asking

What happened

I have many etl task by calling hive beeline in dolphin shell task:

image

in dolphin 2.0.0 all works well. after upgrade to 2.0.5 , many task of this kind hung up by zookeeper's EndOfStreamException:

[INFO] 2022-03-17 05:00:47.669 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[83] - shell task params {"resourceList":[],"localParams":[],"rawScript":"#!/bin/sh\n\nbeeline  -u 'jdbc:hive2://worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver-prd' -e \"load data inpath '/tmp/datax/tmp/ods/ods_io_dm_lp_cloud_lock_his_ratio' overwrite into table ods.ods_io_dm_lp_cloud_lock_his_ratio\"","dependence":{},"conditionResult":{"successNode":[],"failedNode":[]},"waitStartTimeout":{},"switchResult":{}}
[INFO] 2022-03-17 05:00:47.671 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[137] - raw script : #!/bin/sh

beeline  -u 'jdbc:hive2://worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver-prd' -e "load data inpath '/tmp/datax/tmp/ods/ods_io_dm_lp_cloud_lock_his_ratio' overwrite into table ods.ods_io_dm_lp_cloud_lock_his_ratio"
[INFO] 2022-03-17 05:00:47.672 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[138] - task execute path : /tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650
[INFO] 2022-03-17 05:00:47.672 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[86] - tenantCode user:bigtop, task dir:36722_150650
[INFO] 2022-03-17 05:00:47.672 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[91] - create command file:/tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650/36722_150650.command
[INFO] 2022-03-17 05:00:47.673 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[117] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /data/soft/dolphinscheduler-2.0.5/conf/env/dolphinscheduler_env.sh
/tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650/36722_150650_node.sh
[INFO] 2022-03-17 05:00:47.678 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[287] - task run command: sudo -u bigtop sh /tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650/36722_150650.command
[INFO] 2022-03-17 05:00:47.679 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[178] - process start, process id is: 36353
[INFO] 2022-03-17 05:00:48.680 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] -  -> welcome to use bigdata scheduling system...
    Connecting to jdbc:hive2://worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver-prd
    22/03/17 05:00:48 INFO jdbc.Utils: Supplied authorities: worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181
[INFO] 2022-03-17 05:00:49.681 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] -  -> 22/03/17 05:00:48 INFO imps.CuratorFrameworkImpl: Starting
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:host.name=client-10-0-161-28
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_281
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_281-amd64/jre
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/data/soft/spark3/conf/:/data/soft/spark3/jars/HikariCP-2.5.1.jar:/data/soft/spark3/jars/JLargeArrays-1.5.jar:/data/soft/spark3/jars/JTransforms-3.1.jar:/data/soft/spark3/jars/RoaringBitmap-0.9.0.jar:/data/soft/spark3/jars/ST4-4.0.4.jar:/data/soft/spark3/jars/activation-1.1.1.jar:/data/soft/spark3/jars/aircompressor-0.21.jar:/data/soft/spark3/jars/algebra_2.12-2.0.1.jar:/data/soft/spark3/jars/annotations-17.0.0.jar:/data/soft/spark3/jars/antlr-runtime-3.5.2.jar:/data/soft/spark3/jars/antlr4-runtime-4.8.jar:/data/soft/spark3/jars/aopalliance-repackaged-2.6.1.jar:/data/soft/spark3/jars/arpack-2.2.1.jar:/data/soft/spark3/jars/arpack_combined_all-0.1.jar:/data/soft/spark3/jars/arrow-format-2.0.0.jar:/data/soft/spark3/jars/arrow-memory-core-2.0.0.jar:/data/soft/spark3/jars/arrow-memory-netty-2.0.0.jar:/data/soft/spark3/jars/arrow-vector-2.0.0.jar:/data/soft/spark3/jars/audience-annotations-0.5.0.jar:/data/soft/spark3/jars/automaton-1.11-8.jar:/data/soft/spark3/jars/avro-1.10.2.jar:/data/soft/spark3/jars/avro-ipc-1.10.2.jar:/data/soft/spark3/jars/avro-mapred-1.10.2.jar:/data/soft/spark3/jars/blas-2.2.1.jar:/data/soft/spark3/jars/bonecp-0.8.0.RELEASE.jar:/data/soft/spark3/jars/breeze-macros_2.12-1.2.jar:/data/soft/spark3/jars/breeze_2.12-1.2.jar:/data/soft/spark3/jars/cats-kernel_2.12-2.1.1.jar:/data/soft/spark3/jars/chill-java-0.10.0.jar:/data/soft/spark3/jars/chill_2.12-0.10.0.jar:/data/soft/spark3/jars/commons-cli-1.2.jar:/data/soft/spark3/jars/commons-codec-1.15.jar:/data/soft/spark3/jars/commons-collections-3.2.2.jar:/data/soft/spark3/jars/commons-compiler-3.0.16.jar:/data/soft/spark3/jars/commons-compress-1.21.jar:/data/soft/spark3/jars/commons-crypto-1.1.0.jar:/data/soft/spark3/jars/commons-dbcp-1.4.jar:/data/soft/spark3/jars/commons-io-2.8.0.jar:/data/soft/spark3/jars/commons-lang-2.6.jar:/data/soft/spark3/jars/commons-lang3-3.12.0.jar:/data/soft/spark3/jars/commons-logging-1.1.3.jar:/data/soft/spark3/jars/commons-math3-3.4.1.jar:/data/soft/spark3/jars/commons-net-3.1.jar:/data/soft/spark3/jars/commons-pool-1.5.4.jar:/data/soft/spark3/jars/commons-text-1.6.jar:/data/soft/spark3/jars/compress-lzf-1.0.3.jar:/data/soft/spark3/jars/core-1.1.2.jar:/data/soft/spark3/jars/curator-client-2.13.0.jar:/data/soft/spark3/jars/curator-framework-2.13.0.jar:/data/soft/spark3/jars/curator-recipes-2.13.0.jar:/data/soft/spark3/jars/datanucleus-api-jdo-4.2.4.jar:/data/soft/spark3/jars/datanucleus-core-4.1.17.jar:/data/soft/spark3/jars/datanucleus-rdbms-4.1.19.jar:/data/soft/spark3/jars/derby-10.14.2.0.jar:/data/soft/spark3/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/data/soft/spark3/jars/flatbuffers-java-1.9.0.jar:/data/soft/spark3/jars/generex-1.0.2.jar:/data/soft/spark3/jars/gson-2.2.4.jar:/data/soft/spark3/jars/guava-14.0.1.jar:/data/soft/spark3/jars/hadoop-client-api-3.3.1.jar:/data/soft/spark3/jars/hadoop-client-runtime-3.3.1.jar:/data/soft/spark3/jars/hadoop-shaded-guava-1.1.1.jar:/data/soft/spark3/jars/hadoop-yarn-server-web-proxy-3.3.1.jar:/data/soft/spark3/jars/hive-beeline-2.3.9.jar:/data/soft/spark3/jars/hive-cli-2.3.9.jar:/data/soft/spark3/jars/hive-common-2.3.9.jar:/data/soft/spark3/jars/hive-exec-2.3.9-core.jar:/data/soft/spark3/jars/hive-jdbc-2.3.9.jar:/data/soft/spark3/jars/hive-llap-common-2.3.9.jar:/data/soft/spark3/jars/hive-metastore-2.3.9.jar:/data/soft/spark3/jars/hive-serde-2.3.9.jar:/data/soft/spark3/jars/hive-service-rpc-3.1.2.jar:/data/soft/spark3/jars/hive-shims-0.23-2.3.9.jar:/data/soft/spark3/jars/hive-shims-2.3.9.jar:/data/soft/spark3/jars/hive-shims-common-2.3.9.jar:/data/soft/spark3/jars/hive-shims-scheduler-2.3.9.jar:/data/soft/spark3/jars/hive-storage-api-2.7.2.jar:/data/soft/spark3/jars/hive-vector-code-gen-2.3.9.jar:/data/soft/spark3/jars/hk2-api-2.6.1.jar:/data/soft/spark3/jars/hk2-locator-2.6.1.jar:/data/soft/spark3/jars/hk2-utils-2.6.1.jar:/data/soft/spark3/jars/htrace-core4-4.1.0-incubating.jar:/data/soft/spark3/jars/httpclient-4.5.13.jar:/data/soft/spark3/jars/httpcore-4.4.14.jar:/data/soft/spark3/jars/istack-commons-runtime-3.0.8.jar:/data/soft/spark3/jars/ivy-2.5.0.jar:/data/soft/spark3/jars/jackson-annotations-2.12.3.jar:/data/soft/spark3/jars/jackson-core-2.12.3.jar:/data/soft/spark3/jars/jackson-core-asl-1.9.13.jar:/data/soft/spark3/jars/jackson-databind-2.12.3.jar:/data/soft/spark3/jars/jackson-dataformat-yaml-2.12.3.jar:/data/soft/spark3/jars/jackson-datatype-jsr310-2.11.2.jar:/data/soft/spark3/jars/jackson-mapper-asl-1.9.13.jar:/data/soft/spark3/jars/jackson-module-scala_2.12-2.12.3.jar:/data/soft/spark3/jars/jakarta.annotation-api-1.3.5.jar:/data/soft/spark3/jars/jakarta.inject-2.6.1.jar:/data/soft/spark3/jars/jakarta.servlet-api-4.0.3.jar:/data/soft/spark3/jars/jakarta.validation-api-2.0.2.jar:/data/soft/spark3/jars/jakarta.ws.rs-api-2.1.6.jar:/data/soft/spark3/jars/jakarta.xml.bind-api-2.3.2.jar:/data/soft/spark3/jars/janino-3.0.16.jar:/data/soft/spark3/jars/javassist-3.25.0-GA.jar:/data/soft/spark3/jars/javax.jdo-3.2.0-m3.jar:/data/soft/spark3/jars/javolution-5.5.1.jar:/data/soft/spark3/jars/jaxb-api-2.2.11.jar:/data/soft/spark3/jars/jaxb-runtime-2.3.2.jar:/data/soft/spark3/jars/jcl-over-slf4j-1.7.30.jar:/data/soft/spark3/jars/jdo-api-3.0.1.jar:/data/soft/spark3/jars/jersey-client-2.34.jar:/data/soft/spark3/jars/jersey-common-2.34.jar:/data/soft/spark3/jars/jersey-container-servlet-2.34.jar:/data/soft/spark3/jars/jersey-container-servlet-core-2.34.jar:/data/soft/spark3/jars/jersey-hk2-2.34.jar:/data/soft/spark3/jars/jersey-server-2.34.jar:/data/soft/spark3/jars/jline-2.14.6.jar:/data/soft/spark3/jars/joda-time-2.10.10.jar:/data/soft/spark3/jars/jodd-core-3.5.2.jar:/data/soft/spark3/jars/jpam-1.1.jar:/data/soft/spark3/jars/json-1.8.jar:/data/soft/spark3/jars/json4s-ast_2.12-3.7.0-M11.jar:/data/soft/spark3/jars/json4s-core_2.12-3.7.0-M11.jar:/data/soft/spark3/jars/json4s-jackson_2.12-3.7.0-M11.jar:/data/soft/spark3/jars/json4s-scalap_2.12-3.7.0-M11.jar:/data/soft/spark3/jars/jsr305-3.0.0.jar:/data/soft/spark3/jars/jta-1.1.jar:/data/soft/spark3/jars/jul-to-slf4j-1.7.30.jar:/data/soft/spark3/jars/kryo-shaded-4.0.2.jar:/data/soft/spark3/jars/kubernetes-client-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-admissionregistration-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-apiextensions-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-apps-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-autoscaling-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-batch-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-certificates-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-common-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-coordination-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-core-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-discovery-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-events-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-extensions-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-flowcontrol-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-metrics-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-networking-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-node-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-policy-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-rbac-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-scheduling-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-storageclass-5.4.1.jar:/data/soft/spark3/jars/lapack-2.2.1.jar:/data/soft/spark3/jars/leveldbjni-all-1.8.jar:/data/soft/spark3/jars/libfb303-0.9.3.jar:/data/soft/spark3/jars/libthrift-0.12.0.jar:/data/soft/spark3/jars/log4j-1.2.17.jar:/data/soft/spark3/jars/logging-interceptor-3.12.12.jar:/data/soft/spark3/jars/lz4-java-1.7.1.jar:/data/soft/spark3/jars/macro-compat_2.12-1.1.1.jar:/data/soft/spark3/jars/mesos-1.4.0-shaded-protobuf.jar:/data/soft/spark3/jars/metrics-core-4.2.0.jar:/data/soft/spark3/jars/metrics-graphite-4.2.0.jar:/data/soft/spark3/jars/metrics-jmx-4.2.0.jar:/data/soft/spark3/jars/metrics-json-4.2.0.jar:/data/soft/spark3/jars/metrics-jvm-4.2.0.jar:/data/soft/spark3/jars/minlog-1.3.0.jar:/data/soft/spark3/jars/netty-all-4.1.68.Final.jar:/data/soft/spark3/jars/objenesis-2.6.jar:/data/soft/spark3/jars/okhttp-3.12.12.jar:/data/soft/spark3/jars/okio-1.14.0.jar:/data/soft/spark3/jars/opencsv-2.3.jar:/data/soft/spark3/jars/orc-core-1.6.12.jar:/data/soft/spark3/jars/orc-mapreduce-1.6.12.jar:/data/soft/spark3/jars/orc-shims-1.6.12.jar:/data/soft/spark3/jars/oro-2.0.8.jar:/data/soft/spark3/jars/osgi-resource-locator-1.0.3.jar:/data/soft/spark3/jars/paranamer-2.8.jar:/data/soft/spark3/jars/parquet-column-1.12.2.jar:/data/soft/spark3/jars/parquet-common-1.12.2.jar:/data/soft/spark3/jars/parquet-encoding-1.12.2.jar:/data/soft/spark3/jars/parquet-format-structures-1.12.2.jar:/data/soft/spark3/jars/parquet-hadoop-1.12.2.jar:/data/soft/spark3/jars/parquet-jackson-1.12.2.jar:/data/soft/spark3/jars/protobuf-java-2.5.0.jar:/data/soft/spark3/jars/py4j-0.10.9.3.jar:/data/soft/spark3/jars/pyrolite-4.30.jar:/data/soft/spark3/jars/rocksdbjni-6.20.3.jar:/data/soft/spark3/jars/scala-collection-compat_2.12-2.1.1.jar:/data/soft/spark3/jars/scala-compiler-2.12.15.jar:/data/soft/spark3/jars/scala-library-2.12.15.jar:/data/soft/spark3/jars/scala-parser-combinators_2.12-1.1.2.jar:/data/soft/spark3/jars/scala-reflect-2.12.15.jar:/data/soft/spark3/jars/scala-xml_2.12-1.2.0.jar:/data/soft/spark3/jars/shapeless_2.12-2.3.3.jar:/data/soft/spark3/jars/shims-0.9.0.jar:/data/soft/spark3/jars/slf4j-api-1.7.30.jar:/data/soft/spark3/jars/slf4j-log4j12-1.7.30.jar:/data/soft/spark3/jars/snakeyaml-1.27.jar:/data/soft/spark3/jars/snappy-java-1.1.8.4.jar:/data/soft/spark3/jars/spark-catalyst_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-core_2.12-3.2.1.jar:/data/soft/spark3/jars/stax-api-1.0.1.jar:/data/soft/spark3/jars/spark-graphx_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-hive-thriftserver_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-hive_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-kubernetes_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-kvstore_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-launcher_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-mesos_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-mllib-local_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-mllib_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-network-common_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-network-shuffle_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-repl_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-sketch_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-sql_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-streaming_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-tags_2.12-3.2.1-tests.jar:/data/soft/spark3/jars/spark-tags_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-unsafe_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-yarn_2.12-3.2.1.jar:/data/soft/spark3/jars/spire-macros_2.12-0.17.0.jar:/data/soft/spark3/jars/spire-platform_2.12-0.17.0.jar:/data/soft/spark3/jars/spire-util_2.12-0.17.0.jar:/data/soft/spark3/jars/spire_2.12-0.17.0.jar:/data/soft/spark3/jars/stream-2.9.6.jar:/data/soft/spark3/jars/super-csv-2.2.0.jar:/data/soft/spark3/jars/threeten-extra-1.5.0.jar:/data/soft/spark3/jars/tink-1.6.0.jar:/data/soft/spark3/jars/transaction-api-1.1.jar:/data/soft/spark3/jars/univocity-parsers-2.9.1.jar:/data/soft/spark3/jars/velocity-1.5.jar:/data/soft/spark3/jars/xbean-asm9-shaded-4.20.jar:/data/soft/spark3/jars/xz-1.8.jar:/data/soft/spark3/jars/zjsonpatch-0.3.0.jar:/data/soft/spark3/jars/zookeeper-3.6.2.jar:/data/soft/spark3/jars/zookeeper-jute-3.6.2.jar:/data/soft/spark3/jars/zstd-jni-1.5.0-4.jar:/data/soft/spark3/jars/submarine-spark-security-spark32-ranger12.jar:/data/soft/spark3/jars/ojdbc8-19.6.0.0.jar:/data/soft/spark3/jars/iceberg-spark-runtime-3.2_2.12-0.13.1.jar:/data/soft/spark3/jars/kyuubi-extension-spark-3-2_2.12-1.4.1-incubating.jar:/etc/hadoop/conf/
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/3.1.4.0-315/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1127.el7.x86_64
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:user.name=bigtop
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/bigtop
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:user.dir=/tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.memory.free=884MB
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.memory.max=981MB
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.memory.total=981MB
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@1ddf84b8
    22/03/17 05:00:48 INFO common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
    22/03/17 05:00:48 INFO zookeeper.ClientCnxnSocket: jute.maxbuffer value is 1048575 Bytes
    22/03/17 05:00:48 INFO zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled=false
    22/03/17 05:00:48 INFO zookeeper.ClientCnxn: Opening socket connection to server worker-10-0-161-21/10.0.161.21:2181.
    22/03/17 05:00:48 INFO zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
    22/03/17 05:00:48 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.161.28:58538, server: worker-10-0-161-21/10.0.161.21:2181
    22/03/17 05:00:48 INFO zookeeper.ClientCnxn: Session establishment complete on server worker-10-0-161-21/10.0.161.21:2181, session id = 0x37d3196dfaac391, negotiated timeout = 60000
    22/03/17 05:00:48 INFO state.ConnectionStateManager: State change: CONNECTED
    22/03/17 05:00:48 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
    22/03/17 05:00:48 WARN zookeeper.ClientCnxn: An exception was thrown while closing send thread for session 0x37d3196dfaac391.
    EndOfStreamException: Unable to read additional data from server sessionid 0x37d3196dfaac391, likely server has closed socket
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275)
    22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Session: 0x37d3196dfaac391 closed
    22/03/17 05:00:48 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x37d3196dfaac391
    22/03/17 05:00:48 INFO jdbc.Utils: Resolved authority: worker-10-0-161-21:10000
    22/03/17 05:00:49 INFO jdbc.HiveConnection: Connected to worker-10-0-161-21:10000
    Connected to: Apache Hive (version 3.1.0.3.1.4.0-315)
    Driver: Hive JDBC (version 2.3.9)
    Transaction isolation: TRANSACTION_REPEATABLE_READ
[INFO] 2022-03-17 07:08:05.125 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] -  -> . . . . . . . . . . . . . . . . . . . . . . .> /tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650/36722_150650_node.sh: line 3: 36362 Killed                  beeline  -u 'jdbc:hive2://worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver-prd' -e "load data inpath '/tmp/datax/tmp/ods/ods_io_dm_lp_cloud_lock_his_ratio' overwrite into table ods.ods_io_dm_lp_cloud_lock_his_ratio"
[INFO] 2022-03-17 07:08:05.096 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[202] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650, processId:36353 ,exitStatusCode:137 ,processWaitForStatus:true ,processExitValue:137
[INFO] 2022-03-17 07:08:05.128 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[60] - FINALIZE_SESSION

From the log we know task was hung up untill I kill it

What you expected to happen

1、I need some clue to find out the reason. 2、Is it because dolphin give to much pressure to zookeeper? 3、my dolphinscheduler deployed in cluster mode , if I build a new zookeeper cluster , and make dolphinscheduler to point the new zookeeper cluster , need I migration data between zookeeper?

How to reproduce

I'm not sure this problem is because of dolphinscheduler, But it real happen ofter a upgrade, I will make some test.

Anything else

No response

Version

2.0.5

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 2 years ago

Hi:

lordk911 commented 2 years ago

my env problem.