search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.12k stars 3.57k forks source link

broker restart failed #14575

Open JackrayWang opened 2 years ago

JackrayWang commented 2 years ago 
ERROR org.apache.pulsar.functions.worker.FunctionMetaDataManager - Failed to initialize meta data store
org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-fw-192.168.228.59-8080-function-metadata-tailer-reader-55d9f6694a of the topic persistent://public/functions/metadata gets the last message id was failed
{"errorMsg":"Failed to read last entry of the compacted Ledger Error while reading ledger","reqId":240264000667504132, "remote":"192.168.228.107/192.168.228.107:6650", "local":"/192.168.228.59:46798"}
    at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1034) ~[org.apache.pulsar-pulsar-client-api-2.9.1.jar:2.9.1]
    at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1992) ~[org.apache.pulsar-pulsar-client-original-2.9.1.jar:2.9.1]
    at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:181) ~[org.apache.pulsar-pulsar-client-original-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:109) [org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:504) [org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1538) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:750) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
2022-03-07T15:47:11,871+0800 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
java.lang.RuntimeException: Failed to initialize Metadata Manager
    at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:115) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:504) [org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1538) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:750) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-fw-192.168.228.59-8080-function-metadata-tailer-reader-55d9f6694a of the topic persistent://public/functions/metadata gets the last message id was failed
{"errorMsg":"Failed to read last entry of the compacted Ledger Error while reading ledger","reqId":240264000667504132, "remote":"192.168.228.107/192.168.228.107:6650", "local":"/192.168.228.59:46798"}
    at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1034) ~[org.apache.pulsar-pulsar-client-api-2.9.1.jar:2.9.1]
    at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1992) ~[org.apache.pulsar-pulsar-client-original-2.9.1.jar:2.9.1]
    at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:181) ~[org.apache.pulsar-pulsar-client-original-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:109) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    ... 5 more
2022-03-07T15:47:11,872+0800 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: java.lang.RuntimeException: Failed to initialize Metadata Manager
java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:587) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1538) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:750) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
Caused by: java.lang.RuntimeException: Failed to initialize Metadata Manager
    at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:115) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:504) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    ... 4 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-fw-192.168.228.59-8080-function-metadata-tailer-reader-55d9f6694a of the topic persistent://public/functions/metadata gets the last message id was failed
{"errorMsg":"Failed to read last entry of the compacted Ledger Error while reading ledger","reqId":240264000667504132, "remote":"192.168.228.107/192.168.228.107:6650", "local":"/192.168.228.59:46798"}
    at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1034) ~[org.apache.pulsar-pulsar-client-api-2.9.1.jar:2.9.1]
    at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1992) ~[org.apache.pulsar-pulsar-client-original-2.9.1.jar:2.9.1]
    at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:181) ~[org.apache.pulsar-pulsar-client-original-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:109) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:504) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    ... 4 more
2022-03-07T15:47:11,873+0800 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:782) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:587) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1538) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:750) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    ... 2 more
Caused by: java.lang.RuntimeException: Failed to initialize Metadata Manager
    at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:115) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:504) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1538) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:750) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    ... 2 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-fw-192.168.228.59-8080-function-metadata-tailer-reader-55d9f6694a of the topic persistent://public/functions/metadata gets the last message id was failed
{"errorMsg":"Failed to read last entry of the compacted Ledger Error while reading ledger","reqId":240264000667504132, "remote":"192.168.228.107/192.168.228.107:6650", "local":"/192.168.228.59:46798"}
    at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1034) ~[org.apache.pulsar-pulsar-client-api-2.9.1.jar:2.9.1]
    at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1992) ~[org.apache.pulsar-pulsar-client-original-2.9.1.jar:2.9.1]
    at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:181) ~[org.apache.pulsar-pulsar-client-original-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:109) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:504) ~[org.apache.pulsar-pulsar-functions-worker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1538) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:750) ~[org.apache.pulsar-pulsar-broker-2.9.1.jar:2.9.1]
    ... 2 more
codelipenghui commented 2 years ago

@JackrayWang It's a standalone or a cluster? could you please share the complete logs which will help us to see what happens during the broker restart.

JackrayWang commented 2 years ago

cluster @codelipenghui

codelipenghui commented 2 years ago

Ok, is it able to reproduce? or could you please help confirm if the latest branch-2.9 also has the problem? there are some topic compaction fixes in branch-2.9

https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Acomponent%2Fcompaction+is%3Aclosed+label%3Arelease%2F2.9.2

github-actions[bot] commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.

github-actions[bot] commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.