search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.13k stars 3.57k forks source link

[2.10.0] ERROR org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Exception while instantiating ZooKeeper #15647

Closed zfchai closed 2 years ago

zfchai commented 2 years ago

docker image version

25317113938999ae9532727d63bedd2

docker start command

docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf --name pulsar apachepulsar/pulsar:2.10.0 bin/pulsar standalone

log error information

06:17:55.555 [main] INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog - zookeeper.snapshot.trust.empty : true 06:17:55.555 [main] ERROR org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Exception while instantiating ZooKeeper org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.server.ZooKeeperServer.(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:200) [org.apache.pulsar-pulsar-zookeeper-utils-2.10.0.jar:2.10.0] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:443) [org.apache.pulsar-pulsar-zookeeper-utils-2.10.0.jar:2.10.0] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) [org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:139) [org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] 06:17:55.563 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service. java.io.IOException: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:214) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.0.jar:2.10.0] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:443) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.0.jar:2.10.0] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:139) [org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] Caused by: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.server.ZooKeeperServer.(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:200) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.0.jar:2.10.0] ... 3 more

michaeljmarshall commented 2 years ago

@zfchai - thanks for reporting this issue. It should be fixed by #15769 and released in the 2.10.1 release.

michaeljmarshall commented 2 years ago

Fixed by https://github.com/apache/pulsar/pull/15769

jsteggink commented 2 years ago

It seems this is still not working for the standalone version.

docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf a pachepulsar/pulsar:2.10.1 bin/pulsar standalone

WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/pulsar/lib/io.netty-netty-common-4.1.77.Final.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2022-07-21T07:52:45,146+0000 [main] INFO org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Running 1 bookie(s) and advertised them at 127.0.0.1. 2022-07-21T07:52:45,148+0000 [main] INFO org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Starting ZK server 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - ______ _ 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - |___ / | | 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - / / ___ ___ | | __ ___ ___ _ __ ___ _ __ 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - / / / _ \ / _ \ | |/ / / _ \ / _ \ | '_ \ / _ \ | '__| 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - / /__ | (_) | | (_) | | < | __/ | __/ | |_) | | __/ | | 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - /_____| \___/ \___/ |_|\_\ \___| \___| | .__/ \___| |_| 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - | | 2022-07-21T07:52:45,151+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - |_| 2022-07-21T07:52:45,152+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - 2022-07-21T07:52:45,152+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT 2022-07-21T07:52:45,152+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:host.name=22996091af22 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.version=11.0.15 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.vendor=Private Build 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.class.path=/pulsar/conf:::/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.13.2.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.2.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.17.1.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.17.1.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.14.5.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.13.2.jar:/pulsar/lib/commons-collections-commons-collections-3.2.2.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.14.5.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-2.10.1.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.4.32.jar:/pulsar/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-2.10.1.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/io.grpc-grpc-stub-1.45.1.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.14.5-tests.jar:/pulsar/lib/io.vertx-vertx-core-3.9.8.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.6.3.jar:/pulsar/lib/io.swagger-swagger-models-1.6.2.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-2.10.1.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.6.3.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.44.v20210927.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.13.2.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.5.0.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.14.5.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.45.1.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.17.1.jar:/pulsar/lib/io.netty-netty-common-4.1.77.Final.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.77.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-2.10.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-windows-x86_64.jar:/pulsar/lib/io.netty-netty-codec-4.1.77.Final.jar:/pulsar/lib/com.squareup.okio-okio-2.8.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-2.10.1.jar:/pulsar/lib/com.carrotsearch-hppc-0.7.3.jar:/pulsar/lib/io.netty-netty-buffer-4.1.77.Final.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.14.5.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.13.2.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.10.1.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.5.0.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-2.10.1-pkg.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-2.10.1.jar:/pulsar/lib/net.java.dev.jna-jna-4.2.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.44.v20210927.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.2.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.44.v20210927.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.77.Final.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.44.v20210927.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.1.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.5.0.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.77.Final.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.7.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.14.5.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.44.v20210927.jar:/pulsar/lib/io.netty-netty-handler-4.1.77.Final.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.14.5.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.14.5.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.7.6.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.10.2.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.13.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/com.google.re2j-re2j-1.5.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-2.10.1.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.44.v20210927.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.77.Final.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.14.5.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/io.grpc-grpc-alts-1.45.1.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.77.Final.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-osx-aarch_64.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.14.5.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/io.grpc-grpc-context-1.45.1.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.44.v20210927.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.13.2.1.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.14.5.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.14.5.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/org.apache.avro-avro-1.10.2.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.5.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.44.v20210927.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.44.v20210927.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-2.10.1.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/io.kubernetes-client-java-api-12.0.1.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-2.10.1.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-osx-x86_64.jar:/pulsar/lib/org.rocksdb-rocksdbjni-6.10.2.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.14.5.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.2.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-3.2.5.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-2.10.1.jar:/pulsar/lib/io.vertx-vertx-web-3.9.8.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.44.v20210927.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-2.10.1.jar:/pulsar/lib/org.yaml-snakeyaml-1.30.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-2.10.1.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.77.Final.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.14.5.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.77.Final-linux-x86_64.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.14.5.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.14.5.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.77.Final.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-2.10.1.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.13.2.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/io.grpc-grpc-all-1.45.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-2.10.1.jar:/pulsar/lib/io.vertx-vertx-auth-common-3.9.8.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.45.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.13.2.jar:/pulsar/lib/io.prometheus.jmx-collector-0.14.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.10.1.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.5.0.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.14.5.jar:/pulsar/lib/io.perfmark-perfmark-api-0.19.0.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-linux-aarch_64.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.14.5.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.14.5.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.4.32.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.6.3.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/com.beust-jcommander-1.78.jar:/pulsar/lib/io.etcd-jetcd-common-0.5.11.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.77.Final.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-3.2.5.jar:/pulsar/lib/org.apache.pulsar-testmocks-2.10.1.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/io.prometheus-simpleclient-0.5.0.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.45.1.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.44.v20210927.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-2.10.1.jar:/pulsar/lib/io.netty-netty-transport-4.1.77.Final.jar:/pulsar/lib/io.kubernetes-client-java-proto-12.0.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-2.10.1.jar:/pulsar/lib/io.grpc-grpc-netty-1.45.1.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/io.grpc-grpc-xds-1.45.1.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.14.5.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.14.5.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.0.1.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/io.grpc-grpc-rls-1.45.1.jar:/pulsar/lib/io.grpc-grpc-core-1.45.1.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/org.slf4j-jul-to-slf4j-1.7.32.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.52.Final.jar:/pulsar/lib/io.grpc-grpc-services-1.45.1.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.5.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.14.5.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/io.grpc-grpc-auth-1.45.1.jar:/pulsar/lib/io.netty-netty-resolver-4.1.77.Final.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-2.10.1.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-2.10.1.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.45.1.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/io.etcd-jetcd-core-0.5.11.jar:/pulsar/lib/io.vertx-vertx-web-common-3.9.8.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.13.2.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.77.Final.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-2.10.1.jar:/pulsar/lib/io.kubernetes-client-java-12.0.1.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.17.1.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.44.v20210927.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-2.10.1.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/com.google.auto.service-auto-service-annotations-1.0.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.10.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-common-2.10.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-2.10.1.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.14.5.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-2.10.1.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/com.google.http-client-google-http-client-jackson2-1.41.0.jar:/pulsar/lib/io.vertx-vertx-bridge-common-3.9.8.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-2.10.1.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.44.v20210927.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.44.v20210927.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-2.10.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-2.10.1.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.44.v20210927.jar:/pulsar/lib/org.checkerframework-checker-qual-3.12.0.jar:/pulsar/lib/io.swagger-swagger-core-1.6.2.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.4.32.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-2.10.1.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/io.grpc-grpc-api-1.45.1.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.5.0.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-3.2.5.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.44.v20210927.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-2.10.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.52.Final-linux-x86_64.jar:/pulsar/lib/com.google.guava-guava-31.0.1-jre.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-2.10.1.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.44.v20210927.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.44.v20210927.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.44.v20210927.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.14.5.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.4.32.jar: 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.io.tmpdir=/tmp 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.compiler=<NA> 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.name=Linux 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.arch=amd64 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.version=5.10.16.3-microsoft-standard-WSL2 2022-07-21T07:52:45,153+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.name=? 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.home=? 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.dir=/pulsar 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.free=1924MB 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.max=2048MB 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.total=2048MB 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.enableEagerACLCheck = false 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.digest.enabled = true 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.closeSessionTxn.enabled = true 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.flushDelay=0 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.maxWriteQueuePollTime=0 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.maxBatchSize=1000 2022-07-21T07:52:45,154+0000 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.intBufferStartingSizeBytes = 1024 2022-07-21T07:52:45,155+0000 [main] INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog - zookeeper.snapshot.trust.empty : true 2022-07-21T07:52:45,156+0000 [main] ERROR org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Exception while instantiating ZooKeeper org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:200) [org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:443) [org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) [org.apache.pulsar-pulsar-broker-2.10.1.jar:2.10.1] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:139) [org.apache.pulsar-pulsar-broker-2.10.1.jar:2.10.1] 2022-07-21T07:52:45,161+0000 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service. java.io.IOException: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:214) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:443) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) ~[org.apache.pulsar-pulsar-broker-2.10.1.jar:2.10.1] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:139) [org.apache.pulsar-pulsar-broker-2.10.1.jar:2.10.1] Caused by: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:200) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1] ... 3 more

c0090555 commented 2 years ago

@jsteggink @michaeljmarshall Hi, It seems I am having the same issue when running 2.10.1 standalone version on docker:

docker run -it -p 6650:6650  -p 8080:8080 --platform linux/amd64/v8 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.10.1 bin/pulsar standalone

2022-08-10T22:12:18,912+0000 [main] INFO  org.apache.zookeeper.server.persistence.FileTxnSnapLog - zookeeper.snapshot.trust.empty : true
2022-08-10T22:12:18,913+0000 [main] ERROR org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Exception while instantiating ZooKeeper
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
    at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:200) [org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:443) [org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1]
    at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) [org.apache.pulsar-pulsar-broker-2.10.1.jar:2.10.1]
    at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:139) [org.apache.pulsar-pulsar-broker-2.10.1.jar:2.10.1]
2022-08-10T22:12:18,979+0000 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
java.io.IOException: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:214) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:443) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1]
    at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) ~[org.apache.pulsar-pulsar-broker-2.10.1.jar:2.10.1]
    at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:139) [org.apache.pulsar-pulsar-broker-2.10.1.jar:2.10.1]
Caused by: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
    at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:200) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.10.1.jar:2.10.1]
    ... 3 more

Is there any fix on this? Thanks a lot!

michaeljmarshall commented 2 years ago

Thanks for reaching out. Sorry I missed the original comment. I'll take a closer look later tonight.

michaeljmarshall commented 2 years ago

docker run -it -p 6650:6650 -p 8080:8080 --platform linux/amd64/v8 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.10.1 bin/pulsar standalone

This command worked for me on MacOS. Are you running on linux? What are the filesystem permissions of the directories backing the volumes?

c0090555 commented 2 years ago

@michaeljmarshall I am running this in the root directory of a MacBook Pro with M1 chip.

michaeljmarshall commented 2 years ago

The docker image does not yet support running on Arm. Arm support is being tracked here https://github.com/apache/pulsar/issues/12944. We will likely miss the 2.11.0 release since it is already in progress and we don't yet have a PR for the multi-arch build feature. However, I think there is a good chance it will make it into the 2.11.1 release (assuming I or someone else is able to complete that work).

michaeljmarshall commented 2 years ago

@jsteggink - any chance you're on an M1 Mac?

c0090555 commented 2 years ago

@michaeljmarshall thanks for helping with this! So the problem seems to be with the M1(ARM64) chip. I thought add --platform linux/amd64/v8 could help solve this. For me, sometimes I could run this successfully on my M1 machine but it failed most times. May I have a rough timeline on 2.11.1 release?

michaeljmarshall commented 2 years ago

I thought add --platform linux/amd64/v8 could help solve this.

I haven't seen this command argument before, but I see not that it should work based on some quick searches. There must be something different happening with the filesystem permissions for the mounted volume when compared to the Intel arch. The fact that the error is permissions related gives it away: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2. I am not sure what could be causing this, though. What do you see when you run the following?

$ docker run -it -p 6650:6650 -p 8080:8080 --platform linux/amd64/v8 \
--mount source=pulsarsource,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf \
apachepulsar/pulsar:2.10.1 ls -al
total 116
drwxrwxr-x 1 root root  4096 Jun 13 10:17 .
drwxr-xr-x 1 root root  4096 Aug 12 04:55 ..
-rw-r--r-- 1 root root 32726 Jan 22  2020 LICENSE
-rw-r--r-- 1 root root  6612 Jan 22  2020 NOTICE
-rw-r--r-- 1 root root  1269 Jan 22  2020 README
drwxr-xr-x 3 root root  4096 Jun 13 10:14 bin
drwxrwxr-x 5 root root  4096 May 25 05:33 conf
drwxrwxr-x 2 root root  4096 Jun 13 10:14 data
drwxrwxr-x 2 root root  4096 Jun 13 10:14 download
drwxr-xr-x 3 root root  4096 Jun 13 10:13 examples
drwxr-xr-x 4 root root  4096 Jun 13 10:13 instances
drwxr-xr-x 3 root root 28672 Jun 13 10:13 lib
drwxr-xr-x 2 root root  4096 Jan 22  2020 licenses
drwxrwxr-x 2 root root  4096 Jun 13 10:14 logs
drwxr-xr-x 2 root root  4096 Jun 13 10:17 pulsar-client

Additionally, I noticed that the original 2.10.0 worked when using bind volumes. Does that work?

docker run -it -p 6650:6650  -p 8080:8080 \
    --mount type=bind,source=/Users/michaelmarshall/pulsardata,target=/pulsar/data \
    --mount source=pulsarconf,target=/pulsar/conf \
    apachepulsar/pulsar:2.10.0 bin/pulsar standalone
c0090555 commented 2 years ago

@michaeljmarshall Here are what I got:

% docker run -it -p 6650:6650 -p 8080:8080 --platform linux/amd64/v8 \
--mount source=pulsarsource,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf \
apachepulsar/pulsar:2.10.1 ls -al
total 112
drwxrwxr-x 1 root root  4096 Jun 13 10:17 .
drwxr-xr-x 1 root root  4096 Aug 12 20:05 ..
-rw-r--r-- 1 root root 32726 Jan 22  2020 LICENSE
-rw-r--r-- 1 root root  6612 Jan 22  2020 NOTICE
-rw-r--r-- 1 root root  1269 Jan 22  2020 README
drwxr-xr-x 3 root root  4096 Jun 13 10:14 bin
drwxr-xr-x 5 root root  4096 Jul 30 16:17 conf
drwxrwxr-x 2 root root  4096 Jun 13 10:14 data
drwxrwxr-x 2 root root  4096 Jun 13 10:14 download
drwxr-xr-x 3 root root  4096 Jun 13 10:13 examples
drwxr-xr-x 4 root root  4096 Jun 13 10:13 instances
drwxr-xr-x 3 root root 24576 Jun 13 10:13 lib
drwxr-xr-x 2 root root  4096 Jan 22  2020 licenses
drwxrwxr-x 2 root root  4096 Jun 13 10:14 logs
drwxr-xr-x 2 root root  4096 Jun 13 10:17 pulsar-client
docker run -it -p 6650:6650  -p 8080:8080 \
    --mount type=bind,source=/Users/yisongwang/pulsardata,target=/pulsar/data \
    --mount source=pulsarconf,target=/pulsar/conf \
    apachepulsar/pulsar:2.10.0 bin/pulsar standalone
jsteggink commented 2 years ago

@jsteggink - any chance you're on an M1 Mac?

I'm on Windows 10 Pro with WSL2

ericallam commented 1 year ago

I'm on an ARM Mac and I'm seeing the Cannot write to data directory data/standalone/zookeeper/version-2 error only when running pulsar in Docker Compose, but I'm able to run the following docker command and have it work:

docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf --name pulsar apachepulsar/pulsar:latest bin/pulsar standalone

The following docker-compose.yml leads to the error:

version: "3.7"

services:
  pulsar:
    image: apachepulsar/pulsar:latest
    command: bin/pulsar standalone
    ports:
      - 6650:6650
      - 8080:8080
    volumes:
      - pulsardata:/pulsar/data
      - pulsarconf:/pulsar/conf

volumes:
  pulsardata:
  pulsarconf:
ericallam commented 1 year ago

I've gotten passed the error above with the following docker compose file:

version: "3.7"

services:
  pulsar:
    image: apachepulsar/pulsar:latest
    command: bin/pulsar standalone
    ports:
      - 6650:6650
      - 8080:8080
    volumes:
      - pulsardata:/pulsar/data:rw
      - pulsarconf:/pulsar/conf:rw

volumes:
  pulsardata:
    driver: local
    driver_opts:
      type: none
      device: ./pulsar/data
      o: bind
  pulsarconf:
    driver: local
    driver_opts:
      type: none
      device: ./pulsar/conf
      o: bind

As you can see I'm now binding the /pulsar/data and /pulsar/conf directories in the container to my local ./pulsar/data|conf directories.

Unfortunately there is now another error (after ALOT of logs):

pulsar_1  | 2022-12-09T13:28:37,443+0000 [SessionTracker] INFO  org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10000092a760006, timeout of 30000ms exceeded

That repeats about 4 times (with different session IDs) and port 6650 or 8080 are never accessible.

leosanqing commented 1 year ago

Hello, I got this error when I run docker pulsar standalone.(I've try to use 2.8.X,and it occurs another errors)

ENV: MacbookPro(Intel,Not apple's chip), use parallel Desktop to run Ubuntu CMD: Copied from official website docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.10.2 bin/pulsar standalone

ERROR: Caused by: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:201) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] ... 3 more

DETAIL: 14:14:12.797 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.io.tmpdir=/tmp 14:14:12.797 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.compiler=<NA> 14:14:12.797 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.name=Linux 14:14:12.797 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.arch=amd64 14:14:12.797 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.version=5.15.0-40-generic 14:14:12.797 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.name=? 14:14:12.797 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.home=? 14:14:12.797 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.dir=/pulsar 14:14:12.798 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.free=1930MB 14:14:12.798 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.max=2048MB 14:14:12.798 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.total=2048MB 14:14:12.798 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.enableEagerACLCheck = false 14:14:12.798 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.digest.enabled = true 14:14:12.798 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.closeSessionTxn.enabled = true 14:14:12.798 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.flushDelay=0 14:14:12.798 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.maxWriteQueuePollTime=0 14:14:12.799 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.maxBatchSize=1000 14:14:12.799 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - zookeeper.intBufferStartingSizeBytes = 1024 14:14:12.801 [main] INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog - zookeeper.snapshot.trust.empty : true 14:14:12.802 [main] ERROR org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Exception while instantiating ZooKeeper org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:201) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:444) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:143) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] 14:14:12.813 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service. java.io.IOException: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:215) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:444) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:143) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] Caused by: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:201) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2] ... 3 more

michaeljmarshall commented 1 year ago

I think I just figured it out!

When I run docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.10.2 bin/pulsar standalone, it works on my MacOS with an intell chip.

However, that is when I am using a new pulsardata volume. If you use an existing volume that was run with an older version of pulsar, it fails. I reproduced this by first running:

  1. docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardatax,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.8.2 bin/pulsar standalone
  2. docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardatax,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:2.10.2 bin/pulsar standalone

Technically step 1 failed on me for unrelated reasons, but it was able to initialize the zookeeper directories with the old filesystem permissions. Then, when I ran command 2, I reproduced the problematic behavior:

2022-12-16T06:18:37,740+0000 [main] ERROR org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Exception while instantiating ZooKeeper
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
    at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:201) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:444) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:143) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
2022-12-16T06:18:37,755+0000 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
java.io.IOException: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:215) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:444) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:269) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:143) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
Caused by: org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory data/standalone/zookeeper/version-2
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:140) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
    at org.apache.zookeeper.server.ZooKeeperServer.<init>(ZooKeeperServer.java:441) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:201) ~[org.apache.pulsar-pulsar-broker-2.10.2.jar:2.10.2]
    ... 3 more

For anyone following this thread, what happens when you try using a fresh volume?

Hello, I got this error when I run docker pulsar standalone.(I've try to use 2.8.X,and it occurs another errors)

For reference, this comment was extremely helpful. It made me realize that trying different versions together might lead to conflicts. Thanks @leosanqing!

michaeljmarshall commented 1 year ago

If others confirm a new volume solves their observed issues, this might warrant a note in the docs to indicate that a new volume is necessary for anyone that has already run pulsar. Or, a user can manually change the file system permissions on the data directory so that it is writable by the root group.

lhotari commented 1 year ago

Or, a user can manually change the file system permissions on the data directory so that it is writable by the root group.

Here's a way to do that for the example case:

docker run -it --rm --mount source=pulsardatax,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf --user=0 apachepulsar/pulsar:2.10.2 bash -c "chown -R 10000 /pulsar/conf /pulsar/data && chmod g+w /pulsar/conf /pulsar/data"

--user=0 runs as root and bash -c "chown -R 10000 /pulsar/conf /pulsar/data && chmod g+w /pulsar/conf /pulsar/data" fixes the filesystem ownership and permissions.

ericallam commented 1 year ago

I think I was running into a similar issue, and changing to a fresh pulsardata volume fixed the startup issues for me as well. This docker compose file now works:

version: "3.7"

volumes:
  pulsardatav:
    driver: local

services:
  pulsar:
    image: apachepulsar/pulsar:2.10.2
    ports:
      - "6650:6650"
      - "8080:8080"
    volumes:
      - "pulsardatav:/pulsar/data"
    command: bin/pulsar standalone
    container_name: pulsar
    tty: true
    stdin_open: true
    restart: always