search

rob3000 / nestjs-kafka

NestJS integration with KafkaJS
The Unlicense
131 stars 44 forks source link

RegisterAsync: Wait for initial fetch of schemas #24

Open maxgr0 opened 3 years ago

maxgr0 commented 3 years ago

First of all: Sorry for all the issues, just writing down the stuff which goes into my mind while setting up this library together with NestJS for use in a production environment.🤓

When registering KafkaModule asynchronously in NestJS (and using the Avro serializer/deserializer), the application waits for a connection to Kafka before starting and prevents startup when no connection can be established. IMHO, it should act the same way when it comes to schemas. Even when a connection to Kafka is established, it's worth nothing when the events cannot be encoded/decoded.

rob3000 commented 3 years ago

Hey @maxgr0 no problems at all 😄

Just so i understand correctly, this is in relation to the schemas? So if we cannot connect to schema registry or the schemas don't exist then throw an exception and disconnect?

maxgr0 commented 3 years ago

@rob3000 Yeah, as an example: When having no connection to the Kafka brokers, the logs look as following when starting up: Screenshot 2021-04-26 at 3 22 28 PM

This behaviour is absolute right as it prevents e.g. the TerminusModule to be accessible and this results in a not-ready state of the whole application (e.g. readiness probe would fail in k8s).

When the schema registry is not available or when there are some schemas which topics are not registered in the registry, the application still starts (readiness of the container would be okay). It still logs the error correctly but IMHO this type of error should prevent the startup the same way as when there is no connection to Kafka.

Screenshot 2021-04-26 at 3 24 59 PM

Furthermore, the application start should be delayed until the initial fetch of the schemas is done. Imagine a big bunch of requests coming in as soon as the application is ready and the schema fetch from schema registry is still in progress.

rob3000 commented 3 years ago

Perfect thanks @maxgr0 ill look to add this in 😁

maxgr0 commented 3 years ago

@rob3000 nice!✌️ Another point related to this topic it to implement a KafkaHealthIndicator as well as an KafkaSchemaRegistryHealthIndicator to work together with the Terminus module by NestJS. This would have the nice effect that when the connection to one of these components drops while a container is running, it would be easy to see it in the normal Kubernetes management e.g. using kubectl.