search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.23k stars 3.58k forks source link

Pulsar Standalone is shutdown before complete starting #5373

Closed ronaldofreitas closed 4 years ago

ronaldofreitas commented 5 years ago

Describe the bug Before this problem happened, I was using it normally. I had to restart the machine at brute force and after I restarted my computer the pulsar started to catch this problem. I removed and installed the pulsar from scratch and the problem continues.

To Reproduce ./pulsar standalone

23:14:51.138 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.distributedlog.BKLogSegmentWriter - Stream streams_000000000000000001_000000000000000001_000000000000000000::inprogress_000000000000000002 aborted 0 writes 23:14:51.156 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0007 23:14:51.156 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0007 closed 23:14:51.158 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46064 which had sessionid 0x1000044fedc0007 23:14:51.160 [Thread-1] INFO org.apache.distributedlog.BookKeeperClient - BookKeeper Client closed bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared 23:14:51.160 [Thread-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181 23:14:51.164 [DL-io-0] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x9988bc98, L:/127.0.0.1:37070 ! R:localhost/127.0.0.1:3181] 23:14:51.165 [Thread-1] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=8,loopId=3d11891f) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKClientClosedException: BookKeeper client is closed at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:245) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?] at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?] at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1] 23:14:51.165 [Thread-1] ERROR org.apache.distributedlog.BKAbstractLogWriter - Completing Log segments encountered exception java.io.IOException: Failed to close ledger for streams_000000000000000000_000000000000000001_000000000000000000::inprogress_000000000000000002 : BookKeeper client is closed at org.apache.distributedlog.BKLogSegmentWriter$6.closeComplete(BKLogSegmentWriter.java:660) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$0(LedgerHandle.java:550) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$3(LedgerHandle.java:612) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.client.MetadataUpdateLoop.lambda$writeLoop$1(MetadataUpdateLoop.java:146) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.meta.CleanupLedgerManager.lambda$close$1(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[?:1.8.0_202] at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?] at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?] at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1] 23:14:51.167 [bookie-io-1-2] INFO org.apache.bookkeeper.proto.BookieRequestHandler - Channels disconnected: [id: 0x93825262, L:/127.0.0.1:3181 ! R:/127.0.0.1:37070] 23:14:51.167 [DLM-/stream/storage-OrderedScheduler-1-0] INFO org.apache.distributedlog.bk.SimpleLedgerAllocator - Abort ledger allocator without cleaning up on /stream/storage/streams_000000000000000000_000000000000000001_000000000000000000//allocation. 23:14:51.169 [io-read-scheduler-OrderedScheduler-1-0] INFO org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Successfully close the log stream of state store 000000000000000000/000000000000000001/000000000000000000 23:14:51.170 [Thread-1] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9,loopId=69cdb660) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKClientClosedException: BookKeeper client is closed at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:245) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?] at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?] at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1] 23:14:51.171 [Thread-1] ERROR org.apache.distributedlog.BKAbstractLogWriter - Completing Log segments encountered exception java.io.IOException: Failed to close ledger for streams_000000000000000001_000000000000000001_000000000000000000::inprogress_000000000000000002 : BookKeeper client is closed at org.apache.distributedlog.BKLogSegmentWriter$6.closeComplete(BKLogSegmentWriter.java:660) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$0(LedgerHandle.java:550) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$3(LedgerHandle.java:612) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.client.MetadataUpdateLoop.lambda$writeLoop$1(MetadataUpdateLoop.java:146) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.meta.CleanupLedgerManager.lambda$close$1(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[?:1.8.0_202] at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?] at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?] at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1] 23:14:51.173 [DLM-/stream/storage-OrderedScheduler-0-0] INFO org.apache.distributedlog.bk.SimpleLedgerAllocator - Abort ledger allocator without cleaning up on /stream/storage/streams_000000000000000001_000000000000000001_000000000000000000//allocation. 23:14:51.173 [Thread-1] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=6,loopId=6610c33f) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKClientClosedException: BookKeeper client is closed at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:245) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?] at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?] at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1] 23:14:51.174 [io-read-scheduler-OrderedScheduler-1-0] INFO org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Successfully close the log stream of state store 000000000000000001/000000000000000001/000000000000000000 23:14:51.178 [io-read-scheduler-OrderedScheduler-1-0] INFO org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Successfully stopped storage container (1). 23:14:51.174 [Thread-1] ERROR org.apache.distributedlog.BKAbstractLogWriter - Completing Log segments encountered exception java.io.IOException: Failed to close ledger for streams_000000000000000000_000000000000000000_000000000000000000::inprogress_000000000000000002 : BookKeeper client is closed at org.apache.distributedlog.BKLogSegmentWriter$6.closeComplete(BKLogSegmentWriter.java:660) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$0(LedgerHandle.java:550) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$3(LedgerHandle.java:612) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.client.MetadataUpdateLoop.lambda$writeLoop$1(MetadataUpdateLoop.java:146) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_202] at org.apache.bookkeeper.meta.CleanupLedgerManager.lambda$close$1(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[?:1.8.0_202] at org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1410) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.distributedlog.BookKeeperClient.close(BookKeeperClient.java:270) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.doClose(BKNamespaceDriver.java:404) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.impl.BKNamespaceDriver.close(BKNamespaceDriver.java:385) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at com.google.common.io.Closeables.close(Closeables.java:76) ~[com.google.guava-guava-21.0.jar:?] at org.apache.distributedlog.util.Utils.close(Utils.java:544) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.distributedlog.BKDistributedLogNamespace.close(BKDistributedLogNamespace.java:341) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doClose(DLNamespaceProviderService.java:135) ~[org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$4(LifecycleComponentStack.java:123) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) [com.google.guava-guava-21.0.jar:?] at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:123) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doClose(StreamStorageLifecycleComponent.java:61) [org.apache.bookkeeper-stream-storage-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:109) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.stop(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.PulsarStandaloneStarter$1.run(PulsarStandaloneStarter.java:104) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1] 23:14:51.179 [io-read-scheduler-OrderedScheduler-1-0] INFO org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Successfully stopped storage container (1) 23:14:51.179 [io-read-scheduler-OrderedScheduler-1-0] INFO org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Storage container (1) is removed from live set. 23:14:51.179 [DLM-/stream/storage-OrderedScheduler-3-0] INFO org.apache.distributedlog.bk.SimpleLedgerAllocator - Abort ledger allocator without cleaning up on /stream/storage/streams_000000000000000000_000000000000000000_000000000000000000//allocation. 23:14:51.180 [io-read-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Successfully close the log stream of state store 000000000000000000/000000000000000000/000000000000000000 23:14:51.183 [Thread-1] INFO org.apache.distributedlog.ZooKeeperClient - Close zookeeper client bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared:zk. 23:14:51.183 [Thread-1] INFO org.apache.distributedlog.ZooKeeperClient - Closing zookeeper client bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared:zk. 23:14:51.184 [ProcessThread(sid:0 cport:2181):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0008 23:14:51.352 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0008 closed 23:14:51.352 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46070 which had sessionid 0x1000044fedc0008 23:14:51.352 [Thread-1] INFO org.apache.distributedlog.ZooKeeperClient - Closed zookeeper client bk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared:zk. 23:14:51.352 [io-write-scheduler-OrderedScheduler-0-0-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0008 23:14:51.352 [Thread-1] INFO org.apache.distributedlog.ZooKeeperClient - Close zookeeper client dlzk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared. 23:14:51.352 [Thread-1] INFO org.apache.distributedlog.ZooKeeperClient - Closing zookeeper client dlzk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared. 23:14:51.353 [ProcessThread(sid:0 cport:2181):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0006 23:14:51.465 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0006 closed 23:14:51.466 [Thread-1] INFO org.apache.distributedlog.ZooKeeperClient - Closed zookeeper client dlzk:distributedlog://127.0.0.1:2181/stream/storage:factory_writer_shared. 23:14:51.466 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0006 23:14:51.466 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46062 which had sessionid 0x1000044fedc0006 23:14:51.467 [io-read-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Successfully stopped storage container (0). 23:14:51.467 [io-read-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Successfully stopped storage container (0) 23:14:51.467 [io-read-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Storage container (0) is removed from live set. 23:14:51.468 [Thread-1] INFO org.apache.distributedlog.impl.BKNamespaceDriver - Release external resources used by channel factory. 23:14:51.469 [Thread-1] INFO org.apache.distributedlog.impl.BKNamespaceDriver - Stopped request timer 23:14:51.470 [Thread-1] INFO org.apache.distributedlog.BKDistributedLogNamespace - Executor Service Stopped. 23:14:51.471 [Curator-Framework-0] INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting 23:14:51.474 [ProcessThread(sid:0 cport:2181):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0004 23:14:51.794 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0004 closed 23:14:51.794 [Thread-1] INFO org.apache.bookkeeper.proto.BookieServer - Shutting down BookieServer 23:14:51.794 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0004 23:14:51.794 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46058 which had sessionid 0x1000044fedc0004 23:14:51.795 [Thread-1] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 23:14:51.823 [Thread-1] INFO org.apache.bookkeeper.bookie.Bookie - Shutting down Bookie-3181 with exitCode 0 23:14:51.823 [Thread-1] INFO org.apache.bookkeeper.bookie.Bookie - Turning bookie to read only during shut down 23:14:51.823 [Thread-1] INFO org.apache.bookkeeper.bookie.SyncThread - Shutting down SyncThread 23:14:51.827 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file because current active log channel has not initialized yet 23:14:51.836 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file data/standalone/bookkeeper0/current/2.log for logId 2. 23:14:51.842 [pool-5-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file data/standalone/bookkeeper0/current/3.log for logId 3. 23:14:52.351 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.SyncThread - Flush ledger storage at checkpoint CheckpointList{checkpoints=[LogMark: logFileId - 1570846318841 , logFileOffset - 6656]}. 23:14:52.454 [Thread-1] INFO org.apache.bookkeeper.bookie.Journal - Shutting down Journal 23:14:52.455 [ForceWriteThread] INFO org.apache.bookkeeper.bookie.Journal - ForceWrite thread interrupted 23:14:52.456 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.Journal - Journal exits when shutting down 23:14:52.456 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.Journal - Journal exited loop! 23:14:52.457 [Thread-1] INFO org.apache.bookkeeper.bookie.Journal - Finished Shutting down Journal thread 23:14:52.458 [Bookie-3181] INFO org.apache.bookkeeper.bookie.Bookie - Journal thread(s) quit. 23:14:52.843 [Thread-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Shutting down GarbageCollectorThread 23:14:52.844 [Thread-1] INFO org.apache.bookkeeper.bookie.EntryLogger - Stopping EntryLogger 23:14:52.846 [Thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Stopped entry logger preallocator. 23:14:52.850 [Thread-1] INFO org.apache.bookkeeper.bookie.LedgerDirsMonitor - Shutting down LedgerDirsMonitor 23:14:52.853 [ProcessThread(sid:0 cport:2181):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0001 23:14:53.026 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0001 closed 23:14:53.026 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0001 23:14:53.026 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46052 which had sessionid 0x1000044fedc0001 23:14:53.032 [ProcessThread(sid:0 cport:2181):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000044fedc0000 23:14:53.059 [Thread-1] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000044fedc0000 closed 23:14:53.059 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46050 which had sessionid 0x1000044fedc0000 23:14:53.059 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000044fedc0000 23:14:53.059 [Thread-1] INFO org.apache.zookeeper.server.ZooKeeperServer - shutting down 23:14:53.059 [Thread-1] INFO org.apache.zookeeper.server.SessionTrackerImpl - Shutting down 23:14:53.060 [Thread-1] INFO org.apache.zookeeper.server.PrepRequestProcessor - Shutting down 23:14:53.060 [Thread-1] INFO org.apache.zookeeper.server.SyncRequestProcessor - Shutting down 23:14:53.060 [ProcessThread(sid:0 cport:2181):] INFO org.apache.zookeeper.server.PrepRequestProcessor - PrepRequestProcessor exited loop! 23:14:53.060 [SyncThread:0] INFO org.apache.zookeeper.server.SyncRequestProcessor - SyncRequestProcessor exited! 23:14:53.060 [Thread-1] INFO org.apache.zookeeper.server.FinalRequestProcessor - shutdown of request processor complete 23:14:53.063 [Thread-1] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:46078 which had sessionid 0x1000044fedc000b 23:14:53.063 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] INFO org.apache.zookeeper.server.NIOServerCnxnFactory - NIOServerCnxn factory exited run method

sijie commented 5 years ago

@ronaldofreitas Can you try to start with bin/pulsar standalone -nss?

ronaldofreitas commented 5 years ago

I haven't run bin/pulsar standalone -nss yet but even if that solved my problem, I would still be unsure of putting it into production. I am currently testing NATS. Thanks for the answer.

sijie commented 5 years ago

@ronaldofreitas standalone for development purpose. so we enable all the components and new features in standalone. The one you had problems with is from function state. It is irrelative to Pulsar pub/sub. You can use "-nss" to disable this component.

ronaldofreitas commented 5 years ago

Ok, I'll try, it makes sense what you said.

sijie commented 4 years ago

Closed this issue since it is answered.