search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.12k stars 3.57k forks source link

Intermittent dropping of messages from Consume Pulsar : Apache NiFi #10925

Closed tausifrazi123 closed 1 year ago

tausifrazi123 commented 3 years ago

Describe the bug We are using Apache Nifi to consume pulsar from a broker. There is an intermittent dropping of messages from the consume pulsar. The broker is sending the message but it is not getting consumed in the process. Surprisingly there is no error in Nifi. Version currently which is used is 2.5.0

image image

Expected behavior It consume pulsar shouldnt drop any messages.

eolivelli commented 3 years ago

Can you please describe more your setup ? How do you configure NiFi ? do you see errors on the logs of the Pulsar client or in the broker ?

It would be also great to upgrade to some recent Pulsar version, 2.5.x is pretty old (we are releasing 2.8)

tausifrazi123 commented 3 years ago

Apologies for the late reply . I think we have figured out the issue. The underlining controller service which is getting called is not closing the thread and trying to open a new thread connection which is not possible in "Exclusive" subscription type mentioned in the attached picture above.

But how do we solve the problem of pulsarclientservice trying to create a new thread of connection when the old one is still alive. This is the configuration of the client service : image

codelipenghui commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.

tisonkun commented 1 year ago

Closed. It seems self-resolved.