search

spring-cloud / spring-cloud-stream-binder-kafka

Spring Cloud Stream binders for Apache Kafka and Kafka Streams
Apache License 2.0
331 stars 301 forks source link

Apache Kafka Multiple Processor not working, Error while fetching metadata with correlation id xx #1020

Closed MrMoon closed 3 years ago

MrMoon commented 3 years ago

TL:DR I have multiple functions, processors with an Apache Kafka, that keep giving warning that slows the application down from how much warnings I get, 2021-01-08 22:33:10.535 WARN 29114 --- [-StreamThread-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=apollo-course-9ece1ae1-68da-442b-8a08-b918b7a71d97-StreamThread-1-consumer, groupId=apollo-course] Error while fetching metadata with correlation id 12 : {apollo-course-KSTREAM-AGGREGATE-STATE-STORE-0000000002-repartition=UNKNOWN_TOPIC_OR_PARTITION} Hi there, I was wondering how can I have two processors inside the same application, with an Apache Kafka Clusters when I did it I got these kind of WARN,

that slowed the application down because of how much warning there is, I try to just work around having only one processor by using BiFunction or stuff like that to have multiple input and then process them inside the same function and produce the output inside the function as well and it worked like a charm, but I still think it's better to separate them in different functions, because of the fact that it's better to handle and can be maintained easier, as well as having a high cohesion and low coupling functions. I have try to put them in different classes but no progress, the WARN is still there,

Please tell me if I am doing anything wrong, cause I'm still a beginner in the cloud and stream things, but I looked and could not figure it out here is my code, Apollo-Course, the processor is in the kafka package in a class name CourseProcessors, there is the old single processor commented out as well, and the config is in the application.yml file of course I did not want to put it here so it's not a long confusing comment/issue if this is in the wrong place, other spring-cloud project or Stack Overflow, please let me know THANK YOU IN ADVANCE

MrMoon commented 3 years ago

Hello, Does reactor-kafka affect the stream binder or are they independent from each other because my producer and consumer beans are reactive as shown here KafkaConfiguration.java or should I open an issue in the reactor-kafka

sobychacko commented 3 years ago

@MrMoon Apologies for the delay in getting back to you on this issue. Here are my observations for the two issues you raised above.

  1. That error that you are seeing with fetching metadata and the correlation id is due to the fact that you are using the same Kafka Streams application-id for all the various functions. When you have multiple functions like that, you need to use separate application id's. Here are the docs explaining that.
  2. Kafka Streams binder does not use any reactive components and independent from reactor-kafka. So, you are good to use the reactive components for your non-streams Kafka usage. Just don't mix reactive types in your Kafka Streams functions.

Hope this helps. Please let us know how it goes.

MrMoon commented 3 years ago

No worries, Oh yeah got it!!! my bad, I should have read the documentation in more details, I understand know how the binder works with more than one function ( process ), it worked smoothly after I added the application-id. For the reactive components, yeah I don't mix between them so I think I am safe for now . Thank you so much for your help, I really appreciate it.