search

glazkovalex / Rebus.Kafka

Apache Kafka transport for Rebus
MIT License
16 stars 6 forks source link

Rebus.Kafka in .Net core Web API #5

Closed hotamkushwah closed 3 years ago

hotamkushwah commented 3 years ago

I am trying to use Rebus.Kafka in .Net core Web API. I tried to do by register it in startup.cs but it always throw "The transport initialization procedure is still incomplete. There is no confirmation of completion of the subscription to the input queue".

Code is like this : services.AddRebus(configure => configure .Transport(t => t.UseKafka("localhost:9092", "MainQueue",producerConfig,consumerConfig)) .Routing(r => { r.TypeBased() .MapAssemblyOf(MainQueue) })

zejji commented 3 years ago

@hotamkushwah - Just to note that I had no problems with Rebus.Kafka up to version 1.4.3, but since then I get the same error as you. It may be worth comparing that version to see if it works; if so, it is presumably a bug in the more recent versions...

hotamkushwah commented 3 years ago

Just to note that I had no problems with Rebus.Kafka up to version 1.4.3, but since then I get the same error as you. It may be worth comparing that version to see if it works; if so, it is presumably a bug in the more recent versions...

I have checked with version 1.4.3 but still getting same error. I have to use kafka as transport but due to this error not able to do it so now I am exploring the same with mass transit.

glazkovalex commented 3 years ago

I am trying to use Rebus.Kafka in .Net core Web API. I tried to do by register it in startup.cs but it always throw "The transport initialization procedure is still incomplete. There is no confirmation of completion of the subscription to the input queue".

Code is like this : services.AddRebus(configure => configure .Transport(t => t.UseKafka("localhost:9092", "MainQueue",producerConfig,consumerConfig)) .Routing(r => { r.TypeBased() .MapAssemblyOf(MainQueue) })

I will not be able to verify this error for the next 2 weeks. Try using older versions of the package.

hotamkushwah commented 3 years ago

I have tried with older version up to 1.4.2 but getting same error.

zejji commented 3 years ago

@glazkovalex - Just to add that I had no issues with 1.4.3, but later versions have not worked for me.

hotamkushwah commented 3 years ago

Just to add that I had no issues with 1.4.3, but later versions have not worked for me. @zejji Could you please share your startup configuration or your git hub repo link if possible. Thanks in Advance.

glazkovalex commented 3 years ago

I am trying to use Rebus.Kafka in .Net core Web API. I tried to do by register it in startup.cs but it always throw "The transport initialization procedure is still incomplete. There is no confirmation of completion of the subscription to the input queue".

@hotamkushwah, I wrote a test to reproduce your case, but without the WebAPI. I'm sure it doesn't matter in which application the bus is registered, in the console or WebAPI. This test works correctly for me. You can see for yourself. To run the test, you must have Docker Desktop installed.

You probably have something wrong with the infrastructure or with the Apahe Kafka instance. When sending a message, the transport checks whether the consumer's subscription to the queue has already completed initialization. If this subscription is not completed yet and does not complete in 30 seconds, then this exeption is returned. If in your case the consumer subscription takes longer than 30 seconds, then before the first sending to the bus, you can handle this exception, and wait for the end of the subscription of your consumer to the queue in the loop.

Just in case, in the new version 1.5.4 I increase the wait time for initialization to complete to five minutes to make the transport more patient to wait for the consumer's queue subscription to complete. And also added to the recommendations for waiting for the end of the consumer's subscription to the queue in the message of this exception.

glazkovalex commented 3 years ago

@hotamkushwah, a similar question was discussed in the next topic Usage of Confluent 1.5.3 causes topic creation issues. There, the problem was in a specific type of images with Apache Kafka, in particular in Strimzi, in which, on the strong recommendation of the community, automatic topics creation is DISABLED by default on the server side. Also, the Confluence.Kafka library, which is used in this Rebus.Kffka transport, since version 1.5.3 does not allow you to use the atomic creation of topics on some images with Apache Kafka, for example, Strimzi. At the same time, on the image "spotify/kafka", which is used in the test of this transport Rubus.Kafka, the automatic creation of topics works the same way with the old version of Confluence.Kafka. Therefore, I recommend that you either use a different version of the image for the Apache Kafka server, or use old version of the package 1.4.3 Rebus. Kafka with the old version of Confluence. Kafka.