search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.13k stars 3.57k forks source link

proxy errors using docker-compose #7047

Closed oneum72 closed 1 year ago

oneum72 commented 4 years ago

Within docker-compose, using the pulsar:2.4.1 image everything is ok and proxy is running ok... docker-compose.yml: But using the the same docker-compose.yml structure and only change pulsar image to newer version, any of >2.4.1 i.e. pulsar:2.5.2 the pulsar-proxy log is full of errors:

cut from pulsar-proxy (pulsar:2.5.2) log:

[conf/proxy.conf] Applying config configurationStoreServers = lzk1:2181
[conf/proxy.conf] Applying config zookeeperServers = lzk1:2181
09:34:47.487 [main] INFO  org.apache.pulsar.broker.authentication.AuthenticationService - Authentication is disabled
09:34:47.595 [main] INFO  org.eclipse.jetty.util.log - Logging initialized @1578ms to org.eclipse.jetty.util.log.Slf4jLog
09:34:48.288 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
09:34:48.288 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-proxy
09:34:48.288 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_252
09:34:48.288 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
09:34:48.288 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/local/openjdk-8/jre
09:34:48.288 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/com.beust-jcommander-1.48.jar:/pulsar/lib/com.carrotsearch-hppc-0.7.3.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.10.1.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.10.1.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.10.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.10.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.10.1.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.10.1.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.10.1.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.6.2.jar:/pulsar/lib/com.github.luben-zstd-jni-1.3.7-3.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-1.12.0.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-0.9.0.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/com.google.code.gson-gson-2.8.2.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.1.3.jar:/pulsar/lib/com.google.guava-guava-25.1-jre.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.1.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.5.1.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.5.1.jar:/pulsar/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/pulsar/lib/com.squareup.okhttp-logging-interceptor-2.7.5.jar:/pulsar/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/pulsar/lib/com.squareup.okhttp-okhttp-ws-2.7.5.jar:/pulsar/lib/com.squareup.okio-okio-1.13.0.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.0.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/commons-cli-commons-cli-1.2.jar:/pulsar/lib/commons-codec-commons-codec-1.10.jar:/pulsar/lib/commons-collections-commons-collections-3.2.2.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/commons-io-commons-io-2.5.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-3.1.0.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-3.1.0.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-3.1.0.jar:/pulsar/lib/io.grpc-grpc-all-1.18.0.jar:/pulsar/lib/io.grpc-grpc-auth-1.18.0.jar:/pulsar/lib/io.grpc-grpc-context-1.18.0.jar:/pulsar/lib/io.grpc-grpc-core-1.18.0.jar:/pulsar/lib/io.grpc-grpc-netty-1.18.0.jar:/pulsar/lib/io.grpc-grpc-okhttp-1.18.0.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.18.0.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.18.0.jar:/pulsar/lib/io.grpc-grpc-protobuf-nano-1.18.0.jar:/pulsar/lib/io.grpc-grpc-stub-1.18.0.jar:/pulsar/lib/io.grpc-grpc-testing-1.18.0.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.10.5.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.10.5.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.10.5.jar:/pulsar/lib/io.kubernetes-client-java-2.0.0.jar:/pulsar/lib/io.kubernetes-client-java-api-2.0.0.jar:/pulsar/lib/io.kubernetes-client-java-proto-2.0.0.jar:/pulsar/lib/io.netty-netty-buffer-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-codec-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-common-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-handler-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-resolver-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.48.Final.jar:/pulsar/lib/io.vertx-vertx-web-3.4.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.30.Final.jar:/pulsar/lib/io.netty-netty-transport-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.48.Final.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.48.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.48.Final.jar:/pulsar/lib/io.opencensus-opencensus-api-0.18.0.jar:/pulsar/lib/io.opencensus-opencensus-contrib-grpc-metrics-0.18.0.jar:/pulsar/lib/io.prometheus-simpleclient-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.5.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.5.0.jar:/pulsar/lib/io.prometheus.jmx-collector-0.12.0.jar:/pulsar/lib/io.swagger-swagger-annotations-1.5.21.jar:/pulsar/lib/io.swagger-swagger-core-1.5.21.jar:/pulsar/lib/io.swagger-swagger-models-1.5.21.jar:/pulsar/lib/io.vertx-vertx-auth-common-3.4.1.jar:/pulsar/lib/io.vertx-vertx-core-3.4.1.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.1.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.2.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.2.jar:/pulsar/lib/javax.inject-javax.inject-1.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/joda-time-joda-time-2.10.1.jar:/pulsar/lib/log4j-log4j-1.2.17.jar:/pulsar/lib/net.java.dev.jna-jna-4.2.0.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/org.apache.avro-avro-1.9.1.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.9.1.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.10.0.jar:/pulsar/lib/org.apache.bookkeeper.tests-stream-storage-tests-common-4.10.0.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.1.jar:/pulsar/lib/org.apache.commons-commons-compress-1.19.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.4.jar:/pulsar/lib/org.apache.curator-curator-client-4.0.1.jar:/pulsar/lib/org.apache.curator-curator-framework-4.0.1.jar:/pulsar/lib/org.apache.curator-curator-recipes-4.0.1.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.10.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.10.0-tests.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.10.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.10.0.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.5.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.9.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.10.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.10.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.10.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.10.0.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-shaded-2.5.2.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-2.5.2.jar:/pulsar/lib/org.apache.pulsar-protobuf-shaded-2.1.0-incubating.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-discovery-service-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-examples-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-zookeeper-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.5.2.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.5.7.jar:/pulsar/lib/org.aspectj-aspectjrt-1.9.2.jar:/pulsar/lib/org.aspectj-aspectjweaver-1.9.2.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.7.0.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.7.0.jar:/pulsar/lib/org.checkerframework-checker-compat-qual-2.5.2.jar:/pulsar/lib/org.checkerframework-checker-qual-2.0.0.jar:/pulsar/lib/org.codehaus.mojo-animal-sniffer-annotations-1.14.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty-jetty-xml-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.20.v20190813.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.20.v20190813.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.5.0-b42.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.5.0-b42.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.5.0-b42.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.1.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.5.0-b42.jar:/pulsar/lib/org.glassfish.hk2.external-javax.inject-2.5.0-b42.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.27.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.27.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.27.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.27.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.27.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.27.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-jaxb-2.27.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.27.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.27.jar:/pulsar/lib/org.hamcrest-hamcrest-core-1.3.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.inferred-freebuilder-1.14.9.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.6.jar:/pulsar/lib/org.lz4-lz4-java-1.5.0.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.2.jar:/pulsar/lib/org.rocksdb-rocksdbjni-5.13.3.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.25.jar:/pulsar/lib/org.slf4j-jul-to-slf4j-1.7.25.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.25.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.1.3.jar:/pulsar/lib/org.yaml-snakeyaml-1.24.jar:
09:34:48.294 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
09:34:48.294 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
09:34:48.294 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
09:34:48.294 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
09:34:48.294 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
09:34:48.294 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1062.18.1.el7.x86_64
09:34:48.294 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
09:34:48.295 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
09:34:48.295 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar
09:34:48.295 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=2017MB
09:34:48.295 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=2048MB
09:34:48.295 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=2048MB
09:34:48.299 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=lzk1:2181 sessionTimeout=30000 watcher=org.apache.pulsar.zookeeper.ZookeeperClientFactoryImpl$$Lambda$27/333793193@6b7906b3
09:34:48.304 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
09:34:48.312 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
09:34:48.319 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
09:34:48.332 [main-SendThread(lzk1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server lzk1/172.30.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
09:34:48.338 [main-SendThread(lzk1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.30.0.9:59002, server: lzk1/172.30.0.2:2181
09:34:48.344 [main-SendThread(lzk1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server lzk1/172.30.0.2:2181, sessionid = 0x10000ca04c80020, negotiated timeout = 30000
09:34:48.346 [main-EventThread] INFO  org.apache.pulsar.zookeeper.ZookeeperClientFactoryImpl - ZooKeeper session established: State:CONNECTED Timeout:30000 sessionid:0x10000ca04c80020 local:/172.30.0.9:59002 remoteserver:lzk1/172.30.0.2:2181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
09:34:48.497 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=lzk1:2181 sessionTimeout=30000 watcher=org.apache.pulsar.zookeeper.ZookeeperClientFactoryImpl$$Lambda$27/333793193@5b444398
09:34:48.498 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
09:34:48.498 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
09:34:48.499 [main-SendThread(lzk1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server lzk1/172.30.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
09:34:48.499 [main-SendThread(lzk1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.30.0.9:59004, server: lzk1/172.30.0.2:2181
09:34:48.502 [main-SendThread(lzk1:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server lzk1/172.30.0.2:2181, sessionid = 0x10000ca04c80021, negotiated timeout = 30000
09:34:48.502 [main-EventThread] INFO  org.apache.pulsar.zookeeper.ZookeeperClientFactoryImpl - ZooKeeper session established: State:CONNECTED Timeout:30000 sessionid:0x10000ca04c80021 local:/172.30.0.9:59004 remoteserver:lzk1/172.30.0.2:2181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
09:34:48.511 [main] INFO  org.apache.pulsar.broker.authorization.AuthorizationService - Authorization is disabled
09:34:48.602 [main] INFO  org.apache.pulsar.proxy.server.ProxyService - Started Pulsar Proxy at /0.0.0.0:6650
09:34:48.723 [main] INFO  org.eclipse.jetty.server.Server - jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_252-b09
09:34:48.766 [main] INFO  org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
09:34:48.766 [main] INFO  org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
09:34:48.768 [main] INFO  org.eclipse.jetty.server.session - node0 Scavenging every 600000ms
09:34:48.775 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@1cf2fed4{/metrics,null,AVAILABLE}
09:34:49.465 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@3af4e0bf{/,null,AVAILABLE}
09:34:49.506 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@617fe9e1{/admin,null,AVAILABLE}
09:34:49.508 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@6970140a{/lookup,null,AVAILABLE}
09:34:49.519 [main] INFO  org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@35d08e6c{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
09:34:49.519 [main] INFO  org.eclipse.jetty.server.Server - Started @3504ms
09:34:49.521 [main] INFO  org.apache.pulsar.proxy.server.WebServer - Server started at end point http://172.30.0.9:8080
09:35:19.785 [pulsar-external-web-4-4] INFO  org.eclipse.jetty.util.TypeUtil - JVM Runtime does not support Modules
09:35:19.829 [pulsar-external-web-4-4] WARN  org.eclipse.jetty.server.HttpChannel - /admin/v2/clusters
java.lang.IllegalArgumentException: Negative initial size: -1
        at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:74) ~[?:1.8.0_252]
        at org.apache.pulsar.proxy.server.AdminProxyHandler$ReplayableProxyContentProvider.<init>(AdminProxyHandler.java:168) ~[org.apache.pulsar-pulsar-proxy-2.5.2.jar:2.5.2]
        at org.apache.pulsar.proxy.server.AdminProxyHandler.proxyRequestContent(AdminProxyHandler.java:217) ~[org.apache.pulsar-pulsar-proxy-2.5.2.jar:2.5.2]
        at org.eclipse.jetty.proxy.ProxyServlet.service(ProxyServlet.java:114) ~[org.eclipse.jetty-jetty-proxy-9.4.20.v20190813.jar:9.4.20.v20190813]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-javax.servlet-api-3.1.0.jar:3.1.0]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:544) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.Server.handle(Server.java:494) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.eclipse.jetty-jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [org.eclipse.jetty-jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813]
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) [org.eclipse.jetty-jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
09:35:19.888 [pulsar-external-web-4-3] WARN  org.eclipse.jetty.server.HttpChannel - /admin/v2/clusters
java.lang.IllegalArgumentException: Negative initial size: -1

and that error repeats over and over again, this is not expected behaviour isn't it? Any Hints?

sijie commented 4 years ago

@oneum72 it seems that your problem is similar to apache/pulsar#6141 which is related to apache/pulsar-manager#252. I think pulsar-manager is sending some HTTP requests that are not able to be recognized by the pulsar proxy. Can you first try to disable pulsar-manager in your docker-compose file?

oneum72 commented 4 years ago

@sijie I have tested as you suggested and there is only one error:

15:34:08.471 [main-EventThread] WARN  org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader - Error updating broker info after broker list changed.
java.util.concurrent.TimeoutException: null
    at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) ~[?:1.8.0_252]
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_252]
    at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.2.jar:2.5.2]
    at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.updateBrokerList(ZookeeperCacheLoader.java:118) ~[org.apache.pulsar-pulsar-proxy-2.5.2.jar:2.5.2]
    at org.apache.pulsar.proxy.server.util.ZookeeperCacheLoader.lambda$new$0(ZookeeperCacheLoader.java:82) ~[org.apache.pulsar-pulsar-proxy-2.5.2.jar:2.5.2]
    at org.apache.pulsar.zookeeper.ZooKeeperChildrenCache.lambda$0(ZooKeeperChildrenCache.java:85) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.2.jar:2.5.2]
    at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_252]
    at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) ~[?:1.8.0_252]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_252]
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_252]
    at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$22(ZooKeeperCache.java:428) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.5.2.jar:2.5.2]
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:618) [org.apache.pulsar-pulsar-zookeeper-2.5.2.jar:2.5.2]
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [org.apache.pulsar-pulsar-zookeeper-2.5.2.jar:2.5.2]
15:34:08.533 [main-EventThread] INFO  org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x100001abf490011 local:/172.20.0.8:42954 remoteserver:lzk1/172.20.0.2:2181 lastZxid:111 xid:6 sent:9 recv:12 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/apache-pulsar:8080
15:48:48.507 [main-EventThread] INFO  org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x100001abf490011 local:/172.20.0.8:42954 remoteserver:lzk1/172.20.0.2:2181 lastZxid:117 xid:7 sent:98 recv:101 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/apache-pulsar:8080

Now this behavior raised questions:

  1. How come the pulsar-manager:v0.1.0 interact with pulsar-proxy:v2.5.2 in the first place, even before configuring pulsar-manager:v0.1.0 environment to use pulsar-proxy:v2.5.2?
  2. Why the pulsar-proxy:v2.4.1 does not log errors while used with pulsar-manager:v0.1.0??
  3. How come that every and each successive version of pulsar-proxy:v2.4.1 throws errors if used with pulsar-manager:v0.1.0 resulting the unusable pulsar-manager:v0.1.0 UI???

In other words pulsar-manager:v0.1.0 does not work if used with proxy version >pulsar-proxy:v2.4.1 as environment !!!

sijie commented 4 years ago

What does pulsar-admin interface mean here? Do you mean pulsar-admin CLI tool or pulsar-manager?

oneum72 commented 4 years ago

What does pulsar-admin interface mean here? Do you mean pulsar-admin CLI tool or pulsar-manager?

Edited the previous repy!

sijie commented 4 years ago

@oneum72 I don't think the problem is pulsar-manager.

I have tested as you suggested and there is only one error:

What did you change? I don't have any context about what you have changed to answer your question.

oneum72 commented 4 years ago

@oneum72 Can you first try to disable pulsar-manager in your docker-compose file?

So I did tried the initial docker-compose with pulsar-proxy:v2.5.2 and disabled pulsar-manager as you suggested and found out the following as already described above. I can provide additional info if required.

tisonkun commented 1 year ago

Closed as stale. Please open a new issue if it's still relevant to the maintained versions.