apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.27k stars 3.59k forks source link

[Bug] brokies ledger data cannot be released #23599

Open wangrenyi opened 1 week ago

wangrenyi commented 1 week ago

Search before asking

Read release policy

Version

openjdk 11.0.18 2023-01-17 LTS apache-zookeeper-3.6.4 pulsar 2.10.6

Minimal reproduce step

broker conf: broker.txt

bookkeeper conf: bookkeeper.txt

zookeeper conf:

根据以上配置:在四台服务器上面部署服务,其中三台服务器上面都分别部署 broker,bookie,zookeeper服务,另外一台服务器上面仅部署了broker,bookie服务。

异常状态: 其中两台服务器存储满了,2T的磁盘仅剩余100GB可用。

What did you expect to see?

磁盘满之前 diskUsageThreshold=0.95 ,满了之后我把所有节点都修改为 diskUsageThreshold=0.9。

四个bookie节点,其中两个节点上面的存储一直无法释放,我重新设置了消息的 set-message-ttl,set-persistence,set-retention,都设置在一个非常小的范围,但是观察两天后资源还是无法释放,中间有进行过多次bookie,broker重启,也没有生效

What did you see instead?

bookkeeper shell listbookies -rw -h 执行结果如下: 2024-11-14T10:33:43,456+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - ReadWrite Bookies : 2024-11-14T10:33:43,679+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:172.2.10.118:3181, IP:172.2.10.118, Port:3181, Hostname:172.2.10.118 2024-11-14T10:33:43,681+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:172.2.10.121:3181, IP:172.2.10.121, Port:3181, Hostname:pulsar4

bookkeeper shell listbookies -ro -h 执行结果如下: 2024-11-14T10:35:51,841+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - Readonly Bookies : 2024-11-14T10:35:51,879+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:172.2.10.119:3181, IP:172.2.10.119, Port:3181, Hostname:172.2.10.119 2024-11-14T10:35:52,053+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:172.2.10.120:3181, IP:172.2.10.120, Port:3181, Hostname:172.2.10.120

bin/bookkeeper shell bookiesanity 在172.2.10.120节点执行如下: 2024-11-14T10:40:54,054+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xae3ccd27, L:/172.2.10.120:38590 - R:172.2.10.120/172.2.10.120:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} 2024-11-14T10:40:54,078+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (9062267, 0) to bookie (0, 172.2.10.120:3181): Attempting to write on ReadOnly bookie 2024-11-14T10:40:54,109+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9062267,loopId=19ebc2d1) Exception updating org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2024-11-14T10:40:54,127+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:9062267, change-id:0000000001)][attempt:1] Exception changing ensemble org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2024-11-14T10:40:54,127+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9062267 due to NotEnoughBookiesException: Not enough non-faulty bookies available 2024-11-14T10:40:54,129+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L9062267 E0 2024-11-14T10:40:54,130+0800 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:431) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1811) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:575) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2024-11-14T10:40:54,143+0800 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=9062267 2024-11-14T10:40:54,143+0800 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9062267,loopId=6d3717ff) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:547) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:613) ~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:538) ~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1] 2024-11-14T10:40:54,144+0800 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server 2024-11-14T10:40:54,144+0800 [main-EventThread] ERROR org.apache.bookkeeper.meta.CleanupLedgerManager - Failed on operating ledger metadata: -25 2024-11-14T10:40:54,145+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 9062267 2024-11-14T10:40:54,145+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.120:3181 2024-11-14T10:40:54,150+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xae3ccd27, L:/172.2.10.120:38590 ! R:172.2.10.120/172.2.10.120:3181] 2024-11-14T10:40:54,151+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.120:3181 2024-11-14T10:40:54,278+0800 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x300000618a40092 2024-11-14T10:40:54,278+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x300000618a40092 closed

zookeeper异常日志如下: 2024-11-14 08:13:53,325 [myid:2] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@139] - Purge task started. 2024-11-14 08:13:53,325 [myid:2] - INFO [PurgeTask:FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false 2024-11-14 08:13:53,326 [myid:2] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@145] - Purge task completed. 2024-11-14 08:19:37,602 [myid:2] - INFO [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global closeSession request for session 0x20000004643007a 2024-11-14 08:19:38,369 [myid:2] - INFO [NIOWorkerThread-2:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.119:60822, session = 0x20000004643007b 2024-11-14 08:19:40,478 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a4007d 2024-11-14 08:19:41,461 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a4007e 2024-11-14 08:19:41,569 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008b 2024-11-14 08:26:16,633 [myid:2] - INFO [NIOWorkerThread-11:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.120:56702, session = 0x20000004643007c 2024-11-14 08:26:16,633 [myid:2] - INFO [NIOWorkerThread-10:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.120:56706, session = 0x20000004643007d 2024-11-14 08:26:18,835 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x20000004643007e 2024-11-14 08:26:20,074 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a4007f 2024-11-14 08:26:20,207 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x20000004643007f 2024-11-14 08:34:54,679 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40080 2024-11-14 08:34:55,786 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40081 2024-11-14 08:34:55,863 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000046430080 2024-11-14 08:41:31,688 [myid:2] - INFO [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global closeSession request for session 0x20000004643007e 2024-11-14 08:41:31,886 [myid:2] - INFO [NIOWorkerThread-7:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.120:56724, session = 0x20000004643007f 2024-11-14 08:41:34,077 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008c 2024-11-14 08:41:35,321 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008d 2024-11-14 08:41:35,430 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008e 2024-11-14 08:50:07,103 [myid:2] - INFO [NIOWorkerThread-14:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.119:60836, session = 0x200000046430080 2024-11-14 08:50:08,898 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008f 2024-11-14 08:50:09,848 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000046430081 2024-11-14 08:50:09,976 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a0090 2024-11-14 08:56:49,294 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000046430082 2024-11-14 08:56:50,504 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40082 2024-11-14 08:56:50,620 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a0091 2024-11-14 09:05:20,614 [myid:2] - INFO [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global closeSession request for session 0x200000046430081 2024-11-14 09:05:23,088 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40083 2024-11-14 09:05:24,087 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40084 2024-11-14 09:05:24,200 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40085 2024-11-14 09:12:02,063 [myid:2] - INFO [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global closeSession request for session 0x200000046430082 2024-11-14 09:12:04,318 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a0092 2024-11-14 09:12:05,479 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40086 2024-11-14 09:12:05,589 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a0093

bookie服务异常如下: journalctl -f | grep ERROR Nov 14 10:44:48 DFPulsar-172-2-10-120 pulsar[7418]: 2024-11-14T10:44:48,842+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 107515342848, Used space fraction: 0.94621605 > threshold 0.9. Nov 14 10:44:58 DFPulsar-172-2-10-120 pulsar[7418]: 2024-11-14T10:44:58,843+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 107515363328, Used space fraction: 0.94621605 > threshold 0.9. Nov 14 10:45:08 DFPulsar-172-2-10-120 pulsar[7418]: 2024-11-14T10:45:08,842+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 107515314176, Used space fraction: 0.9462161 > threshold 0.9.

Anything else?

No response

Are you willing to submit a PR?

wangrenyi commented 1 week ago

zookeeper config: [Uploading zoo.txt…]()

ChenJhua commented 1 week ago

I have the same problem, have you solved it?

wangrenyi commented 1 week ago

I have the same problem, have you solved it?

I tried many ways to solve it, but it didn't work. If you have a good way, please let me know, thank you

dao-jun commented 1 week ago

I don't see you set backlogQuota.

dao-jun commented 1 week ago

Maybe messages are in backlog that the ledgers can't be deleted.

wangrenyi commented 1 week ago

Maybe messages are in backlog that the ledgers can't be deleted.

I do the following,make no difference bin/pulsar-admin namespaces set-backlog-quota --limit 100G --policy producer_request_hold api-open-test/data-push

image

wangrenyi commented 1 week ago

bin/bookkeeper shell bookiesanity ,the results are as follows:

2024-11-15T16:55:44,433+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled 2024-11-15T16:55:44,620+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-15T16:55:44,620+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 10.2.20.72:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-15T16:55:44,623+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-15T16:55:44,623+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-15T16:55:44,718+0800 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [172.2.10.119:3181] for ledger: 9117303 2024-11-15T16:55:44,719+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Create ledger 9117303 2024-11-15T16:55:44,857+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.119:3181 [id: 0xd15f532d, L:/172.2.10.119:38528 - R:172.2.10.119/172.2.10.119:3181] 2024-11-15T16:55:44,868+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xd15f532d, L:/172.2.10.119:38528 - R:172.2.10.119/172.2.10.119:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} 2024-11-15T16:55:44,889+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (9117303, 0) to bookie (0, 172.2.10.119:3181): Attempting to write on ReadOnly bookie 2024-11-15T16:55:44,902+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9117303,loopId=7a0bfa43) Exception updating org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2024-11-15T16:55:44,918+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:9117303, change-id:0000000001)][attempt:1] Exception changing ensemble org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2024-11-15T16:55:44,918+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9117303 due to NotEnoughBookiesException: Not enough non-faulty bookies available 2024-11-15T16:55:44,921+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L9117303 E0 2024-11-15T16:55:44,921+0800 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:431) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1811) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:575) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2024-11-15T16:55:44,930+0800 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=9117303 2024-11-15T16:55:44,930+0800 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9117303,loopId=6a2ca3d5) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:547) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:613) ~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:538) ~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1] 2024-11-15T16:55:44,932+0800 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server 2024-11-15T16:55:44,932+0800 [main-EventThread] ERROR org.apache.bookkeeper.meta.CleanupLedgerManager - Failed on operating ledger metadata: -25 2024-11-15T16:55:44,932+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 9117303 2024-11-15T16:55:44,932+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181 2024-11-15T16:55:44,936+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181 2024-11-15T16:55:44,936+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xd15f532d, L:/172.2.10.119:38528 ! R:172.2.10.119/172.2.10.119:3181] 2024-11-15T16:55:45,060+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1004929423a01c5 closed 2024-11-15T16:55:45,060+0800 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1004929423a01c5

lhotari commented 1 week ago

pulsar 2.10.6

@wangrenyi this is an unsupported version of Pulsar. Please upgrade to a supported version of Pulsar and check if the problem reproduces.

wangrenyi commented 4 days ago

pulsar 2.10.6

@wangrenyi this is an unsupported version of Pulsar. Please upgrade to a supported version of Pulsar and check if the problem reproduces.

thanks,I will try to upgrade version to 3.x LTS

visxu commented 3 days ago

pulsar 2.10.6

@wangrenyi this is an unsupported version of Pulsar. Please upgrade to a supported version of Pulsar and check if the problem reproduces.

thanks,I will try to upgrade version to 3.x LTS

@wangrenyi Does the problem still exist after upgrading to 3.x LTS ?

wangrenyi commented 2 days ago

pulsar 2.10.6

@wangrenyi this is an unsupported version of Pulsar. Please upgrade to a supported version of Pulsar and check if the problem reproduces.

thanks,I will try to upgrade version to 3.x LTS

@wangrenyi Does the problem still exist after upgrading to 3.x LTS ?

yes,yesterday, I upgraded pulsar from version 2.10.6 to 3.0.7, but the above issues still exist

wangrenyi commented 2 days ago

upgraded version: apache-zookeeper-3.6.4 openjdk version "17.0.13" 2024-10-15 apache-pulsar-3.0.7

broker conf: broker.txt

bookie conf: bookkeeper.txt

zk conf: zoo.txt

status of pulsar nodes in the cluster is as follows(16c 16g): 172.2.10.118,172.2.10.119,172.2.10.120 deploy: zk,broker,bookie

172.2.10.121 deploy: broker,bookie

172.2.10.119,172.2.10.120 the disk on the node cannot be released,other nodes are working properly image image

image

bin/bookkeeper shell bookiesanity 2024-11-20T09:28:32,759+0800 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2024-11-20T09:28:32,768+0800 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://172.2.10.118:2181;172.2.10.119:2181;172.2.10.120:2181/ledgers : zkServers = 172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181, ledgersRootPath = /ledgers. 2024-11-20T09:28:32,771+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.9.2-e454e8c7283100c7caec6dcae2bc82aaecb63023, built on 2024-02-12 20:59 UTC 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar2 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.13 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Eclipse Adoptium 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jdk-17 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/data/apache-pulsar-3.0.7/conf:::/data/apache-pulsar-3.0.7/lib/com.auth0-java-jwt-4.3.0.jar:/data/apache-pulsar-3.0.7/lib/com.auth0-jwks-rsa-0.22.0.jar:/data/apache-pulsar-3.0.7/lib/com.beust-jcommander-1.82.jar:/data/apache-pulsar-3.0.7/lib/com.carrotsearch-hppc-0.9.1.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/data/apache-pulsar-3.0.7/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.github.seancfoley-ipaddress-5.5.0.jar:/data/apache-pulsar-3.0.7/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/data/apache-pulsar-3.0.7/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/data/apache-pulsar-3.0.7/lib/com.google.code.gson-gson-2.8.9.jar:/data/apache-pulsar-3.0.7/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-failureaccess-1.0.1.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-guava-32.1.1-jre.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data/apache-pulsar-3.0.7/lib/com.google.http-client-google-http-client-1.41.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/data/apache-pulsar-3.0.7/lib/com.google.protobuf-protobuf-java-3.25.5.jar:/data/apache-pulsar-3.0.7/lib/com.google.protobuf-protobuf-java-util-3.25.5.jar:/data/apache-pulsar-3.0.7/lib/com.google.re2j-re2j-1.6.jar:/data/apache-pulsar-3.0.7/lib/commons-cli-commons-cli-1.5.0.jar:/data/apache-pulsar-3.0.7/lib/commons-codec-commons-codec-1.15.jar:/data/apache-pulsar-3.0.7/lib/commons-configuration-commons-configuration-1.10.jar:/data/apache-pulsar-3.0.7/lib/commons-io-commons-io-2.14.0.jar:/data/apache-pulsar-3.0.7/lib/commons-lang-commons-lang-2.6.jar:/data/apache-pulsar-3.0.7/lib/commons-logging-commons-logging-1.1.1.jar:/data/apache-pulsar-3.0.7/lib/com.spotify-completable-futures-0.3.6.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okio-okio-3.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.sun.activation-javax.activation-1.2.0.jar:/data/apache-pulsar-3.0.7/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/data/apache-pulsar-3.0.7/lib/com.yahoo.datasketches-memory-0.8.3.jar:/data/apache-pulsar-3.0.7/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/data/apache-pulsar-3.0.7/lib/dev.failsafe-failsafe-3.3.2.jar:/data/apache-pulsar-3.0.7/lib/io.airlift-aircompressor-0.27.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-all-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-alts-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-api-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-auth-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-context-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-core-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-grpclb-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-protobuf-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-rls-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-services-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-servlet-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-stub-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-util-1.60.0.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-xds-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.gsonfire-gson-fire-1.8.5.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-api-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-proto-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-buffer-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-dns-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-haproxy-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-http2-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-http-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-socks-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-common-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-handler-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-handler-proxy-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-classes-macos-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-native-macos-4.1.113.Final-osx-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-native-macos-4.1.113.Final-osx-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-osx-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-osx-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-windows-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-classes-2.0.66.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-classes-epoll-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-epoll-4.1.113.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-epoll-4.1.113.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-unix-common-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-api-0.28.0.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-proto-0.2.0.jar:/data/apache-pulsar-3.0.7/lib/io.perfmark-perfmark-api-0.26.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus.jmx-collector-0.16.1.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-annotations-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-core-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-models-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-auth-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-bridge-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-core-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-web-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-web-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/data/apache-pulsar-3.0.7/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/data/apache-pulsar-3.0.7/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/data/apache-pulsar-3.0.7/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/data/apache-pulsar-3.0.7/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/data/apache-pulsar-3.0.7/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/data/apache-pulsar-3.0.7/lib/javax.validation-validation-api-1.1.0.Final.jar:/data/apache-pulsar-3.0.7/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/data/apache-pulsar-3.0.7/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/data/apache-pulsar-3.0.7/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/data/apache-pulsar-3.0.7/lib/jline-jline-2.14.6.jar:/data/apache-pulsar-3.0.7/lib/net.bytebuddy-byte-buddy-1.14.12.jar:/data/apache-pulsar-3.0.7/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/data/apache-pulsar-3.0.7/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/data/apache-pulsar-3.0.7/lib/net.jcip-jcip-annotations-1.0.jar:/data/apache-pulsar-3.0.7/lib/net.jodah-typetools-0.5.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.avro-avro-1.11.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.avro-avro-protobuf-1.11.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-circe-checksum-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-cpu-affinity-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.http-http-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-native-io-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-statelib-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-proto-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-collections4-4.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-compress-1.26.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-lang3-3.11.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-text-1.10.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-client-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-framework-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-recipes-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-core-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-core-4.16.6-tests.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-protocol-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/data/apache-pulsar-3.0.7/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-bouncy-castle-bc-3.0.7-pkg.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-jetcd-core-shaded-3.0.7-shaded.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-managed-ledger-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-tools-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-config-validation-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-instance-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-proto-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-utils-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-worker-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-io-core-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-metadata-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-core-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-proxy-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-testclient-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-transaction-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-websocket-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-testmocks-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.thrift-libthrift-0.14.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-jute-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/data/apache-pulsar-3.0.7/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/data/apache-pulsar-3.0.7/lib/org.bitbucket.b_c-jose4j-0.9.4.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcpkix-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcprov-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcutil-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.checkerframework-checker-qual-3.33.0.jar:/data/apache-pulsar-3.0.7/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-alpn-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-client-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-continuation-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-http-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-io-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-proxy-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-security-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-servlet-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-servlets-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-util-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-util-ajax-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-api-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-client-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-common-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-client-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-common-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-server-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.inject-jersey-hk2-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.media-jersey-media-multipart-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/data/apache-pulsar-3.0.7/lib/org.javassist-javassist-3.25.0-GA.jar:/data/apache-pulsar-3.0.7/lib/org.jctools-jctools-core-2.1.2.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains-annotations-13.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jline-jline-3.21.0.jar:/data/apache-pulsar-3.0.7/lib/org.jvnet.mimepull-mimepull-1.9.15.jar:/data/apache-pulsar-3.0.7/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/data/apache-pulsar-3.0.7/lib/org.roaringbitmap-RoaringBitmap-1.2.0.jar:/data/apache-pulsar-3.0.7/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/data/apache-pulsar-3.0.7/lib/org.slf4j-slf4j-api-1.7.32.jar:/data/apache-pulsar-3.0.7/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/data/apache-pulsar-3.0.7/lib/org.yaml-snakeyaml-2.0.jar:/data/apache-pulsar-3.0.7/lib/org.zeroturnaround-zt-zip-1.17.jar: 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1160.25.1.el7.x86_64 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/data/apache-pulsar-3.0.7 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=1420MB 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1536MB 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=1536MB 2024-11-20T09:28:32,778+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@5db99216 2024-11-20T09:28:32,783+0800 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2024-11-20T09:28:32,923+0800 [main] INFO org.apache.zookeeper.common.X509Util - Default TLS protocol is TLSv1.3, supported TLS protocols are [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello] 2024-11-20T09:28:32,930+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes 2024-11-20T09:28:32,937+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2024-11-20T09:28:32,999+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.2.10.118/172.2.10.118:2181. 2024-11-20T09:28:32,999+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2024-11-20T09:28:33,004+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.2.10.119:53878, server: 172.2.10.118/172.2.10.118:2181 2024-11-20T09:28:33,011+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.2.10.118/172.2.10.118:2181, session id = 0x1004929423a0501, negotiated timeout = 30000 2024-11-20T09:28:33,013+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2024-11-20T09:28:33,139+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled 2024-11-20T09:28:33,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 10.2.20.229:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=10.2.20.229, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,219+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.121:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,221+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,222+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,282+0800 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [172.2.10.119:3181] for ledger: 9247158 2024-11-20T09:28:33,284+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Create ledger 9247158 2024-11-20T09:28:33,371+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.119:3181 [id: 0x1cbd0b25, L:/172.2.10.119:34984 - R:172.2.10.119/172.2.10.119:3181] 2024-11-20T09:28:33,382+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x1cbd0b25, L:/172.2.10.119:34984 - R:172.2.10.119/172.2.10.119:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} 2024-11-20T09:28:33,401+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (9247158, 0) to bookie (0, 172.2.10.119:3181): Attempting to write on ReadOnly bookie 2024-11-20T09:28:33,411+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9247158,loopId=19a2b0ba) Exception updating org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$9(LedgerHandle.java:1940) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1959) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1907) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2252) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2231) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1402) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,417+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:9247158, change-id:0000000001)][attempt:1] Exception changing ensemble org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$9(LedgerHandle.java:1940) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1959) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1907) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2252) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2231) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1402) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,417+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9247158 due to NotEnoughBookiesException: Not enough non-faulty bookies available 2024-11-20T09:28:33,420+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L9247158 E0 2024-11-20T09:28:33,420+0800 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:406) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1802) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$doAsyncCloseInternal$5(LedgerHandle.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,427+0800 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=9247158 2024-11-20T09:28:33,427+0800 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9247158,loopId=156d9542) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:541) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:614) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:539) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] 2024-11-20T09:28:33,427+0800 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server 2024-11-20T09:28:33,428+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 9247158 2024-11-20T09:28:33,428+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181 2024-11-20T09:28:33,432+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181 2024-11-20T09:28:33,432+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x1cbd0b25, L:/172.2.10.119:34984 ! R:172.2.10.119/172.2.10.119:3181] 2024-11-20T09:28:33,433+0800 [main] WARN org.apache.bookkeeper.client.BookKeeper - The mainWorkerPool did not shutdown cleanly 2024-11-20T09:28:33,551+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1004929423a0501 closed 2024-11-20T09:28:33,551+0800 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1004929423a0501

wangrenyi commented 12 hours ago

when bookeeper is started, the log is as follows: there was a problem with this place during init image image

-- Logs begin at Wed 2024-11-20 13:16:45 CST. -- Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: "compactionMaxOutstandingRequests" : "100000", Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: "journalPreAllocSizeMB" : "16", Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: "journalWriteData" : "true" Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: } Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,597+0800 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Binding bookie-rpc endpoint to 0.0.0.0/0.0.0.0:3181 Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,656+0800 [main] INFO org.apache.bookkeeper.server.EmbeddedServer$Builder - Load lifecycle component : bookie-server Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,690+0800 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://172.2.10.118:2181;172.2.10.119:2181;172.2.10.120:2181/ledgers : zkServers = 172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181, ledgersRootPath = /ledgers. Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,691+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@6b3f6585 Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,691+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,692+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,734+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.2.10.118/172.2.10.118:2181. Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,735+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,736+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.2.10.119:39010, server: 172.2.10.118/172.2.10.118:2181 Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,740+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.2.10.118/172.2.10.118:2181, session id = 0x1004929423a0599, negotiated timeout = 30000 Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,741+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,813+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@7a0ef219 Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,813+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,814+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,815+0800 [main-SendThread(172.2.10.119:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar2/172.2.10.119:2181. Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,816+0800 [main-SendThread(172.2.10.119:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,817+0800 [main-SendThread(172.2.10.119:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.2.10.119:50334, server: pulsar2/172.2.10.119:2181 Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,820+0800 [main-SendThread(172.2.10.119:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar2/172.2.10.119:2181, session id = 0x2000000464305e3, negotiated timeout = 30000 Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,821+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,989+0800 [main] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 172.2.10.119, using default rack for it : /default-rack. Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,989+0800 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ @ /default-rack : org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping. Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,989+0800 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,994+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118, protocol=bookie-rpc, auth=[], extensions=[]}]} Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,995+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.121:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121, protocol=bookie-rpc, auth=[], extensions=[]}]} Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,995+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,005+0800 [BookKeeperClientScheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 172.2.10.118, using default rack for it : /default-rack. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,005+0800 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/172.2.10.118:3181 Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,006+0800 [BookKeeperClientScheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 172.2.10.121, using default rack for it : /default-rack. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,006+0800 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/172.2.10.121:3181 Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,011+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120, protocol=bookie-rpc, auth=[], extensions=[]}]} Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,042+0800 [main] INFO org.apache.bookkeeper.server.EmbeddedServer$Builder - Load lifecycle component : autorecovery Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,047+0800 [main] INFO org.apache.bookkeeper.common.component.ComponentStarter - Starting component bookie-server. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,060+0800 [main] INFO org.eclipse.jetty.util.log - Logging initialized @5680ms to org.eclipse.jetty.util.log.Slf4jLog Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,122+0800 [main] INFO org.eclipse.jetty.server.Server - jetty-9.4.54.v20240208; built: 2024-02-08T19:42:39.027Z; git: cef3fbd6d736a21e7d541a5db490381d95a2047d; jvm 17.0.13+11 Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,153+0800 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@6680f714{/,null,AVAILABLE} Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,165+0800 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@7fc420b8{HTTP/1.1, (http/1.1)}{0.0.0.0:8000} Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,166+0800 [main] INFO org.eclipse.jetty.server.Server - Started @5787ms Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,166+0800 [main] INFO org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider - Started Prometheus stats endpoint at 0.0.0.0/:8000 Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,177+0800 [main] INFO org.apache.bookkeeper.bookie.Bookie - Replaying journal 1680073544285 from position 516 Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,179+0800 [main] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data/bookkeeper/journal1/current/1872c2fd25d.txn Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,180+0800 [main] INFO org.apache.bookkeeper.bookie.Bookie - Replaying journal 1680073544286 from position 0 Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,180+0800 [main] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data/bookkeeper/journal1/current/1872c2fd25e.txn Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,180+0800 [main] INFO org.apache.bookkeeper.bookie.Bookie - Finished replaying journal in 5 ms. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,184+0800 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.LedgerDirsManager - No writable ledger dirs below diskUsageThreshold. But Dirs that can accommodate 1073741824 are: [/data/bookkeeper/ledgers/current] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,190+0800 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 1872c2fd258.txn Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,191+0800 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.SyncThread - Flush ledger storage at checkpoint CheckpointList{checkpoints=[LogMark: logFileId - 1680073544286 , logFileOffset - 516]}. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,191+0800 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.LedgerDirsManager - No writable ledger dirs below diskUsageThreshold. But Dirs that can accommodate 1073741824 are: [/data/bookkeeper/ledgers/current] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,193+0800 [main] INFO org.apache.bookkeeper.bookie.Bookie - Finished reading journal, starting bookie Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,195+0800 [BookieStateManagerService-0] INFO org.apache.bookkeeper.bookie.BookieStateManager - Re-registering the bookie Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,195+0800 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.Journal - Starting journal on /data/bookkeeper/journal1/current Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,204+0800 [ForceWriteThread] INFO org.apache.bookkeeper.bookie.Journal - ForceWrite Thread started Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,211+0800 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data/bookkeeper/journal1/current/1872c2fd25f.txn Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119, protocol=bookie-rpc, auth=[], extensions=[]}]} Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,218+0800 [BookieStateManagerService-0] WARN org.apache.bookkeeper.discover.ZKRegistrationManager - No writable bookie registered node /ledgers/available/172.2.10.119:3181 when transitioning to readonly Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /ledgers/available/172.2.10.119:3181 Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.zookeeper.KeeperException.create(KeeperException.java:117) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.zookeeper.KeeperException.create(KeeperException.java:53) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1614) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2101(ZooKeeperClient.java:70) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooKeeperClient$11.call(ZooKeeperClient.java:773) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooKeeperClient$11.call(ZooKeeperClient.java:767) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooKeeperClient.delete(ZooKeeperClient.java:767) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.discover.ZKRegistrationManager.doRegisterReadOnlyBookie(ZKRegistrationManager.java:305) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.discover.ZKRegistrationManager.registerBookie(ZKRegistrationManager.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.bookie.BookieStateManager.doRegisterBookie(BookieStateManager.java:293) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.bookie.BookieStateManager.doRegisterBookie(BookieStateManager.java:281) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.bookie.BookieStateManager$2.call(BookieStateManager.java:244) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.bookie.BookieStateManager$2.call(BookieStateManager.java:239) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,225+0800 [main] INFO org.apache.bookkeeper.bookie.UncleanShutdownDetectionImpl - Created dirty file in ledger dir: /data/bookkeeper/ledgers/current Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,480+0800 [main] INFO org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,497+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1 Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,611+0800 [bookkeeper-io-15-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.121:3181 [id: 0xd9b2f0c6, L:/172.2.10.119:34396 - R:172.2.10.121/172.2.10.121:3181] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,611+0800 [bookkeeper-io-15-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.118:3181 [id: 0x815b954d, L:/172.2.10.119:40516 - R:172.2.10.118/172.2.10.118:3181] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,618+0800 [bookkeeper-io-15-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xd9b2f0c6, L:/172.2.10.119:34396 - R:172.2.10.121/172.2.10.121:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,618+0800 [bookkeeper-io-15-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x815b954d, L:/172.2.10.119:40516 - R:172.2.10.118/172.2.10.118:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,656+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [, ], excluded bookies [, ], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,658+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [,]. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,658+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [, ], fallback to choose bookie randomly from the cluster. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,658+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,658+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,668+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8906044 for 0 number of times, so deferring the ledger lock release by 9375 msecs Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,670+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,680+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1 Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,690+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [, ], excluded bookies [, ], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster. Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,690+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,690+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [, ], fallback to choose bookie randomly from the cluster. Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,691+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,691+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,698+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8906707 for 0 number of times, so deferring the ledger lock release by 9375 msecs Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,698+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,176+0800 [LedgerDirsMonitorThread] WARN org.apache.bookkeeper.bookie.LedgerDirsMonitor - LedgerDirsMonitor check process: All ledger directories are non writable Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,178+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 103906926592, Used space fraction: 0.9480211 > threshold 0.9. Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,708+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1 Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [, ], excluded bookies [, ], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster. Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [,]. Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [, ], fallback to choose bookie randomly from the cluster. Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,777+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 9001906 for 0 number of times, so deferring the ledger lock release by 9375 msecs Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,778+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,796+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [, ], excluded bookies [, ], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster. Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,797+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,797+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [, ], fallback to choose bookie randomly from the cluster. Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,797+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,797+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,801+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8961414 for 0 number of times, so deferring the ledger lock release by 9375 msecs Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,801+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,175+0800 [LedgerDirsMonitorThread] WARN org.apache.bookkeeper.bookie.LedgerDirsMonitor - LedgerDirsMonitor check process: All ledger directories are non writable Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,179+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 103906902016, Used space fraction: 0.9480212 > threshold 0.9. Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,824+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [, ], excluded bookies [, ], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster. Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,824+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,824+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [, ], fallback to choose bookie randomly from the cluster. Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,824+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,825+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,831+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8906044 for 1 number of times, so deferring the ledger lock release by 18750 msecs Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,832+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,838+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1 Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,847+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [, ], excluded bookies [, ], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster. Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,848+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,848+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [, ], fallback to choose bookie randomly from the cluster. Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,848+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,848+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?] Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,856+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8906707 for 1 number of times, so deferring the ledger lock release by 18750 msecs Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,856+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,176+0800 [LedgerDirsMonitorThread] WARN org.apache.bookkeeper.bookie.LedgerDirsMonitor - LedgerDirsMonitor check process: All ledger directories are non writable Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,177+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 103906889728, Used space fraction: 0.9480212 > threshold 0.9. Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,870+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1 Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,914+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [, ], excluded bookies [, ], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster. Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,914+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [, ]. Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,914+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [, ], fallback to choose bookie randomly from the cluster. Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,914+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [, ], allBookies [,]. Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,915+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available