search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.13k stars 3.57k forks source link

[Bug] Zookeeper TLS breaks /bin/pulsar zookeeper-shell #17442

Closed yuweisung closed 1 year ago

yuweisung commented 2 years ago

Search before asking

Version

2.9 and 2.10

Minimal reproduce step

zookeeper server sslquorum and client TLS on (2281 port).

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=data/zookeeper
# the port at which the clients will connect
clientPort=2181

# the port at which the admin will listen
admin.enableServer=true
admin.serverPort=9990

# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=1000
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

# Requires updates to be synced to media of the transaction log before finishing
# processing the update. If this option is set to 'no', ZooKeeper will not require
# updates to be synced to the media.
# WARNING: it's not recommended to run a production ZK cluster with forceSync disabled.
forceSync=yes

# Enable Quorum TLS on each node.
# To Transition from nontls to tls without downtime: the system needs to be restarted several times.
# First, Restart zookeepers with portUnification: true.
# Next,  Restart zookeepers with sslQuorum: true
# Finally, Restart zookeepers with portUnification: false
# Default: false
sslQuorum=true

# Specifies that the client port should accept SSL connections
# (using the same configuration as the secure client port).
# Default: false
portUnification=false

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpPort=8000
metricsProvider.exportJvmInfo=true
globalOutstandingLimit=100000
peerType=participant
secureClientPort=2281
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
snapCount=1000000
ssl.keyStore.location=/pulsar/certs/zookeeper/tls-combined.pem
ssl.keyStore.type=PEM
ssl.quorum.keyStore.location=/pulsar/certs/zookeeper/tls-combined.pem
ssl.quorum.keyStore.type=PEM
ssl.quorum.trustStore.location=/pulsar/certs/ca/ca.crt
ssl.quorum.trustStore.type=PEM
ssl.trustStore.location=/pulsar/certs/ca/ca.crt
ssl.trustStore.type=PEM
zookeeper.client.secure=true
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
server.1=test1-sn-platform-zookeeper-0.test1-sn-platform-zookeeper.snp.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
server.2=test1-sn-platform-zookeeper-1.test1-sn-platform-zookeeper.snp.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
server.3=test1-sn-platform-zookeeper-2.test1-sn-platform-zookeeper.snp.svc.cluster.local:2888:3888:participant;0.0.0.0:2181

What did you expect to see?

root@test1-sn-platform-toolset-0:/pulsar# bin/pulsar zookeeper-shell -server test1-sn-platform-zookeeper:2281 --run-once "ls /"
Warning: Nashorn engine is planned to be removed from a future JDK release
Connecting to test1-sn-platform-zookeeper:2281
15:57:02.635 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.6.3--819541497c21cb28b0fa1ed456dee9f2ee6742f1-dirty, built on 01/28/2022 15:35 GMT
15:57:02.639 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=test1-sn-platform-toolset-0.test1-sn-platform-toolset.snp.svc.cluster.local
15:57:02.640 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=11.0.15
15:57:02.641 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Private Build
15:57:02.642 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64
15:57:02.643 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/io.netty-netty-buffer-4.1.77.Final.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.77.Final.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.4.32.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.14.5.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.43.v20210629.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.77.Final.jar:/pulsar/lib/org.apache.avro-avro-1.10.2.jar:/pulsar/lib/io.vertx-vertx-core-3.9.8.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.77.Final-linux-x86_64.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-3.2.5.jar:/pulsar/lib/io.streamnative-zookeeper-3.6.3.1.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.0.1.jar:/pulsar/lib/io.streamnative-pulsar-client-messagecrypto-bc-2.9.2.20.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.14.5.jar:/pulsar/lib/io.streamnative-pulsar-broker-common-2.9.2.20.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.14.5.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.17.1.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/io.vertx-vertx-bridge-common-3.9.8.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/org.apache.logging.log4j-log4j-1.2-api-2.17.1.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.2.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.13.2.1.jar:/pulsar/lib/io.streamnative-pulsar-functions-utils-2.9.2.20.jar:/pulsar/lib/io.kubernetes-client-java-12.0.1.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-windows-x86_64.jar:/pulsar/lib/com.google.http-client-google-http-client-jackson2-1.41.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.17.1.jar:/pulsar/lib/io.streamnative-pulsar-client-admin-api-2.9.2.20.jar:/pulsar/lib/io.vertx-vertx-web-3.9.8.jar:/pulsar/lib/io.streamnative-pulsar-functions-secrets-2.9.2.20.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.43.v20210629.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-osx-x86_64.jar:/pulsar/lib/org.rocksdb-rocksdbjni-6.10.2.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.14.5-tests.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/io.prometheus.jmx-collector-0.14.0.jar:/pulsar/lib/io.streamnative-pulsar-package-core-2.9.2.20.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.6.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.14.5.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.52.Final.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.14.5.jar:/pulsar/lib/io.kubernetes-client-java-proto-12.0.1.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/io.grpc-grpc-services-1.45.1.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.14.5.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.77.Final.jar:/pulsar/lib/io.netty-netty-codec-4.1.77.Final.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.43.v20210629.jar:/pulsar/lib/io.perfmark-perfmark-api-0.19.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.43.v20210629.jar:/pulsar/lib/io.streamnative-pulsar-transaction-common-2.9.2.20.jar:/pulsar/lib/io.netty-netty-transport-4.1.77.Final.jar:/pulsar/lib/com.beust-jcommander-1.78.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.77.Final.jar:/pulsar/lib/org.yaml-snakeyaml-1.30.jar:/pulsar/lib/io.netty-netty-common-4.1.77.Final.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.7.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.5.0.jar:/pulsar/lib/io.vertx-vertx-auth-common-3.9.8.jar:/pulsar/lib/io.streamnative-pulsar-client-auth-sasl-2.9.2.20.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.14.5.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.14.5.jar:/pulsar/lib/io.streamnative-pulsar-functions-runtime-2.9.2.20.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.17.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.13.2.jar:/pulsar/lib/io.streamnative-pulsar-config-validation-2.9.2.20.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.17.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/io.netty-netty-handler-4.1.77.Final.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-3.2.5.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-linux-aarch_64.jar:/pulsar/lib/io.streamnative-pulsar-zookeeper-utils-2.9.2.20.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.77.Final.jar:/pulsar/lib/io.streamnative-pulsar-package-bookkeeper-storage-2.9.2.20.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.13.2.jar:/pulsar/lib/io.streamnative-zookeeper-jute-3.6.3.1.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.43.v20210629.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/commons-collections-commons-collections-3.2.2.jar:/pulsar/lib/io.streamnative-pulsar-functions-worker-2.9.2.20.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.14.5.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.45.1.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.14.5.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.14.5.jar:/pulsar/lib/com.carrotsearch-hppc-0.7.3.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.14.5.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.2.jar:/pulsar/lib/com.squareup.okio-okio-2.8.0.jar:/pulsar/lib/io.streamnative-pulsar-client-admin-original-2.9.2.20.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.14.5.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/io.streamnative-pulsar-client-original-2.9.2.20.jar:/pulsar/lib/io.grpc-grpc-alts-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/io.streamnative-pulsar-broker-auth-sasl-2.9.2.20.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/io.grpc-grpc-all-1.45.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.43.v20210629.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.13.2.jar:/pulsar/lib/io.streamnative-pulsar-io-common-2.9.2.20.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.5.0.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.13.2.jar:/pulsar/lib/io.streamnative-pulsar-common-2.9.2.20.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/io.streamnative-pulsar-transaction-coordinator-2.9.2.20.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.14.5.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.43.v20210629.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.14.5.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/org.slf4j-jul-to-slf4j-1.7.32.jar:/pulsar/lib/io.prometheus-simpleclient-0.5.0.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/io.streamnative-managed-ledger-2.9.2.20.jar:/pulsar/lib/io.streamnative-pulsar-io-core-2.9.2.20.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.6.2-tests.jar:/pulsar/lib/io.streamnative-pulsar-client-api-2.9.2.20.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.77.Final.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.45.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/io.streamnative-pulsar-functions-local-runner-original-2.9.2.20.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.4.jar:/pulsar/lib/io.grpc-grpc-xds-1.45.1.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.13.2.jar:/pulsar/lib/io.streamnative-zookeeper-prometheus-metrics-3.6.3.1.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.77.Final.jar:/pulsar/lib/io.kubernetes-client-java-api-12.0.1.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.77.Final.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/io.streamnative-pulsar-functions-instance-2.9.2.20.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-linux-x86_64.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.5.0.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.43.v20210629.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.4.32.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/io.grpc-grpc-rls-1.45.1.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.4.32.jar:/pulsar/lib/com.google.guava-guava-30.1-jre.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.5.0.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/io.swagger-swagger-core-1.6.2.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.5.0.jar:/pulsar/lib/io.streamnative-pulsar-client-tools-2.9.2.20.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.7.6.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/io.grpc-grpc-core-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.43.v20210629.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.13.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.5.0.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/io.streamnative-pulsar-functions-api-2.9.2.20.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/io.grpc-grpc-api-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.14.5.jar:/pulsar/lib/io.vertx-vertx-web-common-3.9.8.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-osx-aarch_64.jar:/pulsar/lib/io.grpc-grpc-auth-1.45.1.jar:/pulsar/lib/org.checkerframework-checker-qual-3.5.0.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.1.jar:/pulsar/lib/io.streamnative-bouncy-castle-bc-2.9.2.20-pkg.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.14.5.jar:/pulsar/lib/io.swagger-swagger-models-1.6.2.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.43.v20210629.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.14.5.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.10.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.43.v20210629.jar:/pulsar/lib/io.grpc-grpc-context-1.45.1.jar:/pulsar/lib/net.java.dev.jna-jna-4.2.0.jar:/pulsar/lib/io.streamnative-pulsar-broker-2.9.2.20.jar:/pulsar/lib/io.streamnative-pulsar-proxy-2.9.2.20.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.4.32.jar:/pulsar/lib/io.grpc-grpc-stub-1.45.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/io.streamnative-pulsar-testclient-2.9.2.20.jar:/pulsar/lib/com.google.re2j-re2j-1.5.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.45.1.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.14.5.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.43.v20210629.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.43.v20210629.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/io.streamnative-pulsar-functions-proto-2.9.2.20.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.14.5.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-3.2.5.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.77.Final.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/io.streamnative-pulsar-metadata-2.9.2.20.jar:/pulsar/lib/io.streamnative-testmocks-2.9.2.20.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/io.netty-netty-resolver-4.1.77.Final.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.14.5.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.13.2.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.14.5.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.13.2.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.6.2.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.43.v20210629.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/io.grpc-grpc-netty-1.45.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.14.5.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.43.v20210629.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.2.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/io.streamnative-pulsar-websocket-2.9.2.20.jar:/pulsar/lib/pulsar-broker-auth-ldap-shaded-2.9.2.20.jar:/pulsar/lib/pulsar-broker-auth-oauth2-2.9.2.20-sources.jar:/pulsar/lib/pulsar-broker-auth-oauth2-2.9.2.20-javadoc.jar:/pulsar/lib/pulsar-package-storage-cloud-2.9.2.20-jar-with-dependencies.jar:/pulsar/lib/state-storage-server-2.9.2.20.jar:/pulsar/lib/pulsar-broker-auth-oauth2-2.9.2.20.jar:/pulsar/lib/pulsar-broker-auth-vault-shaded-2.9.2.20.jar:/pulsar/lib/oauth-client-2.9.2.20.jar:
15:57:02.646 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
15:57:02.647 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
15:57:02.647 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
15:57:02.647 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
15:57:02.648 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
15:57:02.649 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.10.109+
15:57:02.649 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
15:57:02.650 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
15:57:02.651 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar
15:57:02.651 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=7MB
15:57:02.652 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=128MB
15:57:02.652 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=68MB
15:57:02.668 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=test1-sn-platform-zookeeper:2281 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@46f699d5
15:57:02.674 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/pulsar/lib/io.netty-netty-common-4.1.77.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
15:57:03.021 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
15:57:03.155 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
15:57:03.218 [main-SendThread(test1-sn-platform-zookeeper:2281)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server test1-sn-platform-zookeeper/10.104.2.46:2281.
15:57:03.221 [main-SendThread(test1-sn-platform-zookeeper:2281)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
15:57:03.940 [epollEventLoopGroup-2-1] INFO  org.apache.zookeeper.ClientCnxnSocketNetty - SSL handler added for channel: [id: 0xebde597d]
15:57:03.955 [epollEventLoopGroup-2-1] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.104.0.60:52748, server: test1-sn-platform-zookeeper/10.104.2.46:2281
15:57:03.957 [epollEventLoopGroup-2-1] INFO  org.apache.zookeeper.ClientCnxnSocketNetty - channel is connected: [id: 0xebde597d, L:/10.104.0.60:52748 - R:test1-sn-platform-zookeeper/10.104.2.46:2281]
15:57:04.852 [epollEventLoopGroup-2-1] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server test1-sn-platform-zookeeper/10.104.2.46:2281, session id = 0x100101879700000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zookeeper]
15:57:04.886 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 0

What did you see instead?

2022-09-03T15:46:44,010+0000 [main-SendThread(test1-sn-platform-zookeeper:2281)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server test1-sn-platform-zookeeper/10.104.1.45:2281.
2022-09-03T15:46:44,012+0000 [main-SendThread(test1-sn-platform-zookeeper:2281)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2022-09-03T15:46:44,036+0000 [main-SendThread(test1-sn-platform-zookeeper:2281)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.104.1.44:49514, server: test1-sn-platform-zookeeper/10.104.1.45:2281
2022-09-03T15:46:44,104+0000 [main-SendThread(test1-sn-platform-zookeeper:2281)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server test1-sn-platform-zookeeper/10.104.1.45:2281, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket
    at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[io.streamnative-zookeeper-3.6.3.1.jar:3.6.3.1]
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[io.streamnative-zookeeper-3.6.3.1.jar:3.6.3.1]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1292) [io.streamnative-zookeeper-3.6.3.1.jar:3.6.3.1]
KeeperErrorCode = ConnectionLoss for /
2022-09-03T15:46:44,245+0000 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 1
Connecting to test1-sn-platform-zookeeper:2281
2022-09-03T15:46:52,031+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.6.3--819541497c21cb28b0fa1ed456dee9f2ee6742f1-dirty, built on 01/28/2022 15:35 GMT
2022-09-03T15:46:52,035+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=test1-sn-platform-bookie-bk-init-78kmx
2022-09-03T15:46:52,036+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=11.0.15
2022-09-03T15:46:52,039+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Private Build
2022-09-03T15:46:52,040+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64
2022-09-03T15:46:52,040+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/io.netty-netty-buffer-4.1.77.Final.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.77.Final.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.4.32.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.14.5.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.43.v20210629.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.77.Final.jar:/pulsar/lib/org.apache.avro-avro-1.10.2.jar:/pulsar/lib/io.vertx-vertx-core-3.9.8.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.77.Final-linux-x86_64.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-3.2.5.jar:/pulsar/lib/io.streamnative-zookeeper-3.6.3.1.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.0.1.jar:/pulsar/lib/io.streamnative-pulsar-client-messagecrypto-bc-2.9.2.20.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.14.5.jar:/pulsar/lib/io.streamnative-pulsar-broker-common-2.9.2.20.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.14.5.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.17.1.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/io.vertx-vertx-bridge-common-3.9.8.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/org.apache.logging.log4j-log4j-1.2-api-2.17.1.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.2.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.13.2.1.jar:/pulsar/lib/io.streamnative-pulsar-functions-utils-2.9.2.20.jar:/pulsar/lib/io.kubernetes-client-java-12.0.1.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-windows-x86_64.jar:/pulsar/lib/com.google.http-client-google-http-client-jackson2-1.41.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.17.1.jar:/pulsar/lib/io.streamnative-pulsar-client-admin-api-2.9.2.20.jar:/pulsar/lib/io.vertx-vertx-web-3.9.8.jar:/pulsar/lib/io.streamnative-pulsar-functions-secrets-2.9.2.20.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.43.v20210629.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-osx-x86_64.jar:/pulsar/lib/org.rocksdb-rocksdbjni-6.10.2.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.14.5-tests.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/io.prometheus.jmx-collector-0.14.0.jar:/pulsar/lib/io.streamnative-pulsar-package-core-2.9.2.20.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.6.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.14.5.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.52.Final.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.14.5.jar:/pulsar/lib/io.kubernetes-client-java-proto-12.0.1.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/io.grpc-grpc-services-1.45.1.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.14.5.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.77.Final.jar:/pulsar/lib/io.netty-netty-codec-4.1.77.Final.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.43.v20210629.jar:/pulsar/lib/io.perfmark-perfmark-api-0.19.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.43.v20210629.jar:/pulsar/lib/io.streamnative-pulsar-transaction-common-2.9.2.20.jar:/pulsar/lib/io.netty-netty-transport-4.1.77.Final.jar:/pulsar/lib/com.beust-jcommander-1.78.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.77.Final.jar:/pulsar/lib/org.yaml-snakeyaml-1.30.jar:/pulsar/lib/io.netty-netty-common-4.1.77.Final.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.7.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.5.0.jar:/pulsar/lib/io.vertx-vertx-auth-common-3.9.8.jar:/pulsar/lib/io.streamnative-pulsar-client-auth-sasl-2.9.2.20.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.14.5.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.14.5.jar:/pulsar/lib/io.streamnative-pulsar-functions-runtime-2.9.2.20.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.17.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.13.2.jar:/pulsar/lib/io.streamnative-pulsar-config-validation-2.9.2.20.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.17.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/io.netty-netty-handler-4.1.77.Final.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-3.2.5.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-linux-aarch_64.jar:/pulsar/lib/io.streamnative-pulsar-zookeeper-utils-2.9.2.20.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.77.Final.jar:/pulsar/lib/io.streamnative-pulsar-package-bookkeeper-storage-2.9.2.20.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.13.2.jar:/pulsar/lib/io.streamnative-zookeeper-jute-3.6.3.1.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.43.v20210629.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/commons-collections-commons-collections-3.2.2.jar:/pulsar/lib/io.streamnative-pulsar-functions-worker-2.9.2.20.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.14.5.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.45.1.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.14.5.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.14.5.jar:/pulsar/lib/com.carrotsearch-hppc-0.7.3.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.14.5.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.2.jar:/pulsar/lib/com.squareup.okio-okio-2.8.0.jar:/pulsar/lib/io.streamnative-pulsar-client-admin-original-2.9.2.20.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.14.5.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/io.streamnative-pulsar-client-original-2.9.2.20.jar:/pulsar/lib/io.grpc-grpc-alts-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/io.streamnative-pulsar-broker-auth-sasl-2.9.2.20.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/io.grpc-grpc-all-1.45.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.43.v20210629.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.13.2.jar:/pulsar/lib/io.streamnative-pulsar-io-common-2.9.2.20.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.5.0.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.13.2.jar:/pulsar/lib/io.streamnative-pulsar-common-2.9.2.20.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/io.streamnative-pulsar-transaction-coordinator-2.9.2.20.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.14.5.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.43.v20210629.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.14.5.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/org.slf4j-jul-to-slf4j-1.7.32.jar:/pulsar/lib/io.prometheus-simpleclient-0.5.0.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/io.streamnative-managed-ledger-2.9.2.20.jar:/pulsar/lib/io.streamnative-pulsar-io-core-2.9.2.20.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.6.2-tests.jar:/pulsar/lib/io.streamnative-pulsar-client-api-2.9.2.20.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.77.Final.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.45.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/io.streamnative-pulsar-functions-local-runner-original-2.9.2.20.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.4.jar:/pulsar/lib/io.grpc-grpc-xds-1.45.1.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.13.2.jar:/pulsar/lib/io.streamnative-zookeeper-prometheus-metrics-3.6.3.1.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.77.Final.jar:/pulsar/lib/io.kubernetes-client-java-api-12.0.1.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.77.Final.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/io.streamnative-pulsar-functions-instance-2.9.2.20.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-linux-x86_64.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.5.0.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.43.v20210629.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.4.32.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/io.grpc-grpc-rls-1.45.1.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.4.32.jar:/pulsar/lib/com.google.guava-guava-30.1-jre.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.5.0.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/io.swagger-swagger-core-1.6.2.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.5.0.jar:/pulsar/lib/io.streamnative-pulsar-client-tools-2.9.2.20.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.7.6.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/io.grpc-grpc-core-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.43.v20210629.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.13.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.5.0.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/io.streamnative-pulsar-functions-api-2.9.2.20.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/io.grpc-grpc-api-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.14.5.jar:/pulsar/lib/io.vertx-vertx-web-common-3.9.8.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-osx-aarch_64.jar:/pulsar/lib/io.grpc-grpc-auth-1.45.1.jar:/pulsar/lib/org.checkerframework-checker-qual-3.5.0.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.1.jar:/pulsar/lib/io.streamnative-bouncy-castle-bc-2.9.2.20-pkg.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.14.5.jar:/pulsar/lib/io.swagger-swagger-models-1.6.2.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.43.v20210629.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.14.5.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.10.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.43.v20210629.jar:/pulsar/lib/io.grpc-grpc-context-1.45.1.jar:/pulsar/lib/net.java.dev.jna-jna-4.2.0.jar:/pulsar/lib/io.streamnative-pulsar-broker-2.9.2.20.jar:/pulsar/lib/io.streamnative-pulsar-proxy-2.9.2.20.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.4.32.jar:/pulsar/lib/io.grpc-grpc-stub-1.45.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/io.streamnative-pulsar-testclient-2.9.2.20.jar:/pulsar/lib/com.google.re2j-re2j-1.5.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.45.1.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.14.5.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.43.v20210629.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.43.v20210629.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/io.streamnative-pulsar-functions-proto-2.9.2.20.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.14.5.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-3.2.5.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.77.Final.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/io.streamnative-pulsar-metadata-2.9.2.20.jar:/pulsar/lib/io.streamnative-testmocks-2.9.2.20.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/io.netty-netty-resolver-4.1.77.Final.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.14.5.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.13.2.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.14.5.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.13.2.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.6.2.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.43.v20210629.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/io.grpc-grpc-netty-1.45.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.14.5.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.43.v20210629.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.2.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/io.streamnative-pulsar-websocket-2.9.2.20.jar:/pulsar/lib/pulsar-broker-auth-ldap-shaded-2.9.2.20.jar:/pulsar/lib/pulsar-broker-auth-oauth2-2.9.2.20-sources.jar:/pulsar/lib/pulsar-broker-auth-oauth2-2.9.2.20-javadoc.jar:/pulsar/lib/pulsar-package-storage-cloud-2.9.2.20-jar-with-dependencies.jar:/pulsar/lib/state-storage-server-2.9.2.20.jar:/pulsar/lib/pulsar-broker-auth-oauth2-2.9.2.20.jar:/pulsar/lib/pulsar-broker-auth-vault-shaded-2.9.2.20.jar:/pulsar/lib/oauth-client-2.9.2.20.jar:
2022-09-03T15:46:52,043+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2022-09-03T15:46:52,095+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2022-09-03T15:46:52,095+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2022-09-03T15:46:52,099+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2022-09-03T15:46:52,099+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2022-09-03T15:46:52,100+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.10.109+
2022-09-03T15:46:52,100+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
2022-09-03T15:46:52,101+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
2022-09-03T15:46:52,101+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar
2022-09-03T15:46:52,101+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=12MB
2022-09-03T15:46:52,102+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=494MB
2022-09-03T15:46:52,102+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=46MB
2022-09-03T15:46:52,108+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=test1-sn-platform-zookeeper:2281 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@55342f40
2022-09-03T15:46:52,127+0000 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2022-09-03T15:46:52,138+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
2022-09-03T15:46:52,204+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2022-09-03T15:46:52,227+0000 [main-SendThread(test1-sn-platform-zookeeper:2281)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server test1-sn-platform-zookeeper/10.104.0.61:2281.
2022-09-03T15:46:52,229+0000 [main-SendThread(test1-sn-platform-zookeeper:2281)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2022-09-03T15:46:52,314+0000 [main-SendThread(test1-sn-platform-zookeeper:2281)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.104.1.44:46066, server: test1-sn-platform-zookeeper/10.104.0.61:2281
2022-09-03T15:46:52,324+0000 [main-SendThread(test1-sn-platform-zookeeper:2281)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server test1-sn-platform-zookeeper/10.104.0.61:2281, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket
    at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[io.streamnative-zookeeper-3.6.3.1.jar:3.6.3.1]
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[io.streamnative-zookeeper-3.6.3.1.jar:3.6.3.1]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1292) [io.streamnative-zookeeper-3.6.3.1.jar:3.6.3.1]
KeeperErrorCode = ConnectionLoss for /
2022-09-03T15:46:52,519+0000 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 1

zk server log

2022-09-03T15:45:21,108+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.QuorumPeer - Peer state changed: following - synchronization - snap
2022-09-03T15:45:21,108+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.Learner - Getting a snapshot from leader 0x100000000
2022-09-03T15:45:21,109+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.watch.WatchManagerFactory - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2022-09-03T15:45:21,109+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.watch.WatchManagerFactory - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2022-09-03T15:45:21,155+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.Learner - Learner received NEWLEADER message
2022-09-03T15:45:21,169+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.QuorumPeer - Dynamic reconfig is disabled, we don't store the last seen config.
2022-09-03T15:45:21,174+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.persistence.FileTxnSnapLog - Snapshotting: 0x100000000 to data/zookeeper/version-2/snapshot.100000000
2022-09-03T15:45:21,186+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.ZooKeeperServer - Snapshot taken in 12 ms
2022-09-03T15:45:21,193+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.QuorumPeer - Peer state changed: following - synchronization
2022-09-03T15:45:21,263+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.CommitProcessor - Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1
2022-09-03T15:45:21,268+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.CommitProcessor - Configuring CommitProcessor with 1 worker threads.
2022-09-03T15:45:21,302+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.RequestThrottler - zookeeper.request_throttler.shutdownTimeout = 10000
2022-09-03T15:45:21,503+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.Learner - Learner received UPTODATE message
2022-09-03T15:45:21,504+0000 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=0.0.0.0:2281)] INFO  org.apache.zookeeper.server.quorum.QuorumPeer - Peer state changed: following - broadcast
2022-09-03T15:45:24,310+0000 [epollEventLoopGroup-4-1] INFO  org.apache.zookeeper.server.NettyServerCnxn - Processing ruok command from /127.0.0.1:55378
2022-09-03T15:45:28,671+0000 [epollEventLoopGroup-7-1] ERROR org.apache.zookeeper.server.NettyServerCnxnFactory - Unsuccessful handshake with session 0x0
2022-09-03T15:45:28,852+0000 [epollEventLoopGroup-7-1] WARN  org.apache.zookeeper.server.NettyServerCnxnFactory - Exception caught
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) [io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) [io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) [io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
    at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1214) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[io.netty-netty-handler-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
    ... 15 more

Anything else?

According to zookeeper guide, when turning on secured zk, client will need to use Netty context to communicate with zk server because the default context NIO doesn't support TLS/SSL. 

elif [ $COMMAND == "zookeeper-shell" ]; then
    exec $JAVA $OPTS -Dzookeeper.clientCnxnSocket="org.apache.zookeeper.ClientCnxnSocketNetty" -Dzookeeper.client.secure=true org.apache.zookeeper.ZooKeeperMain $@

Are you willing to submit a PR?

tisonkun commented 2 years ago

@yuweisung please format the output and share your full setup for minimal reproduction:

  1. What's the ZK config exactly?
  2. What're the concrete steps to reproduce the issue?
tisonkun commented 2 years ago

And basically, I think you can use the client-configuration option to force the connection using Netty impl.

Reference: https://zookeeper.apache.org/doc/r3.7.0/zookeeperCLI.html (search client-configuration)

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.

tisonkun commented 1 year ago

Closed as no response.