apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.1k stars 3.56k forks source link

[Bug] Producer creation gets blocked “Closed producer before its creation was completed” #22217

Open Xxf9126 opened 6 months ago

Xxf9126 commented 6 months ago

Search before asking

Version

Pulsar 2.10.2

org.apache.pulsar pulsar-client 2.10.2

Minimal reproduce step

Call the Pulsar Client API to create a topic producer.

What did you expect to see?

The client returns normally and the producer is created successfully.

What did you see instead?

broker log: 1.Topic producer creating 2024-03-05T02:35:13,559+0000 [pulsar-io-4-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/42.193.247.132:55368][persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Creating producer. producerId=28

2.It prompts that the creation is successful and then starts blocking. At this time, the client thread is also the same. The blocking process lasts 23 seconds. 2024-03-05T02:35:13,591+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.pulsar.broker.service.BrokerService - Created topic persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report - dedup is disabled 2024-03-05T02:35:36,644+0000 [pulsar-web-36-8] INFO org.eclipse.jetty.server.RequestLog - 192.168.64.2 - - [05/Mar/2024:02:35:36 +0000] "GET /admin/v2/clusters HTTP/1.1" 200 13 "-" "Pulsar-Java-v2.5.2" 1

3.Client reports timeout error broker log printing 2024-03-05T02:35:37,932+0000 [pulsar-io-4-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/42.193.247.132:55368] Closed producer before its creation was completed. producerId=28

4.The client initiates a request again to create the topic producer 2024-03-05T02:35:37,953+0000 [pulsar-io-4-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/42.193.247.132:55368][persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Creating producer. producerId=29

5.The request creation is not blocked but returns an error 2024-03-05T02:36:07,814+0000 [pulsar-io-4-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/42.193.247.132:55368] Closed producer before its creation was completed. producerId=29

6.After continuing to request several times to create a topic, the broker prints an exception log: [persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Global topic inactive for 60 seconds, closed repl producers 2024-03-05T02:37:44,217+0000 [pulsar-inactivity-monitor-24-1] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Topic is already being closed or deleted 2024-03-05T02:37:44,217+0000 [pulsar-inactivity-monitor-24-1] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Inactive topic deletion failed java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException: Topic is already fenced

7.Subsequent requests to the topic kept reporting the error "Topic is already fenced", and finally restarted Pulsar to recover.

The following is the complete log of the broker: 20240307151942 20240307152153 20240307152337 20240307152419

The following is my client-side producer code 20240307152713

Anything else?

How can I improve my code or Pulsar configuration to avoid the blocking scenario, or allow operations to continue on a topic after it gets blocked? Once a topic enters the "fenced" state, the client is unable to perform any further operations on that topic besides restarting the broker.

Would the following information be helpful for my situation? https://github.com/apache/pulsar/issues/20526#issuecomment-1581958101

Are you willing to submit a PR?

lhotari commented 5 months ago

Pulsar 2.10.2

org.apache.pulsar pulsar-client 2.10.2

Pulsar 2.10.x is end-of-support and it won't get bug fixes. Please upgrade to a supported version of Pulsar, https://pulsar.apache.org/contribute/release-policy/#supported-versions . There are important security advisories and all 2.10.x users should upgrade to 2.10.6.