Global Kafka and Kafka streams configuration review

syalioune commented 1 year ago

The purpose of this issue is to list checkpoints related to Kafka usage that should be validated before releasing the POC (both on the external application and DT side).

Requirements

[ ] Is the processing semantic clearly defined (ALOS, EOS) and does the implementation match the chosen semantic ?

Topics

[ ] Is the topic partitioning strategy correctly thought of (# of partitions and partition keys) ?
[ ] Is the default replication factor set correctly for each Topic ?
[ ] Is the default minimum ISR set correctly for each Topic ?
[ ] Are the topic retention parameters correctly set (Time VS Capacity based retention) ?
[ ] Is Topic compaction used for the appropriate use case ?

Producers

[ ] Are the producer using the proper acks configuration ?
[ ] Are the producer using the proper batching configuration (linger.ms / batch.size / buffer.memory) ?
[ ] Are the producers providing callbacks to handle retryable errors ?
[ ] Are automatic producer retries configured (retries / retry.backoff.ms / delivery.timeout.ms) ?
[ ] Is idempotence enabled to avoid duplicates (enable.idempotence) ?
[ ] Is the number of in-flight requests per connection correctly setup (max.in.flight.requests.per.connection) ?
[ ] Does max.request.size match the upper cap on broker side ?
[ ] Is the compression algorithm properly thought off ?
[ ] Is client.id properly set ?
[ ] If event publication must be synchronized with DB update, is this correctly designed (i.e. Outbox pattern) ?
[ ] Are event production errors correctly handled ?

Consumers

[ ] Is the consumer auto.commit option correctly setup (enable.auto.commit) ?
[ ] Are the consumer correctly commiting the offsets in the consumer loop (synchronous VS asynchronous) ?
[ ] Are Rebalance operations correctly handled ? Is a rebalance listener defined ?
[ ] Are poison pills correctly handled ?
[ ] Are the consumer polling parameters correctly setup (session.timeout.ms, max.poll.records and max.poll.interval.ms) ?
[ ] Is the consumer heartbeat period correctly setup (heartbeat.interval.ms) ?
[ ] Is the Consumer Group protocol correctly used ? Is the Consumer Group identifier externalized ?
[ ] Is the consumer logic idempotent ?
[ ] Is the partition assignment strategy properly thought of (partition.assignment.strategy) ?
[ ] Are the events fetching parameters matching the required throughput (fetch.min.bytes / max.partition.fetch.bytes / fetch.max.wait.ms) ?
[ ] Is the offset retention matching the consumers SLAs or topic retention (offsets.retention.minutes) ?
[ ] Is the auto reset feature correctly used (auto.offset.reset) ?
[ ] Is client.id properly set ?
[ ] In case of multi DC deployment, is the rack awareness enabled (client.rack) ?
[ ] Are Consumer isolation level properly used (for transactional use cases) ?

Kafka Streams

[ ] Is the application.id option correctly setup (application.id) ?
[ ] Is the threading model properly thought of (# of threads vs # of instances : num.streams.threads) ?
[ ] If the Streams application is Stateful, is the State Directory overriden (state.dir) and made persistent ?
[ ] If the Streams application is Stateful, are hot standby configured for failover (num.standby.replicas) ?
[ ] If the Streams application is Stateful, is the replication factor for interna Topic overidden (replication.factor) ?
[ ] Is the Streams application inner producer configured for resiliency (acks - min.insync.replicas) ?
[ ] Are each processor in the topology explicity named to ensure readibility, prevent data loss and make the topology resilient to updates ?
[ ] Are Serdes explicitly set for each processor in the topology ?
[ ] Is the Streams Application Production Exception Handler configured ?
[ ] Is the Streams Application Deserialization Exception Handler configured ?
[ ] Is the Kafka Streams Topology showing some repartition operations ? If so, are repartition Topic renamed (Indeed repartition Topics should be renamed using the through operator) ?
[ ] Is the stateful cache configuration correct (cache.max.bytes.buffering) ?
[ ] Is the processing guarantee in line with the business requirements (processing.guarantee) ?
[ ] If exactly_once processing guarantee is used, are Kafka Transactional parameters correctly setup (transaction.state.log.replication.factor - transaction.state.log.min.isr) ?
[ ] If exactly_once processing guarantee is used, Does downstream consumers have the correct isolaton level (isolaton.level) ?
[ ] Are the DSL/Processor operations correctly used to implement the Streams application ?
[ ] Is the metric recording level correctly setup (metrics.recording.level) ?
[ ] Do you need to define your own time in your kafka stream application (default.timestamp.extractor) ?
[ ] If KStream/KTable join are used, does it account for KTable data coming after the KStream (max.task.idle.ms) ?
[ ] Are Serdes explicitly set for each operation ?

nscuro commented 1 year ago

Thanks again for yet another detailed and super helpful writeup (and shame on me for the late response ⚰️)!

The processing semantics are definitely something we should consider more sooner than later (although I assume "at least once" will be fine in most cases).

Reading through this list, there are many topics that will depend on the user's expected workload. Meaning we will need to experiment how certain setups behave, and provide proper documentation and example configs. Referring to things like partition count, replication factor etc. here.

syalioune commented 1 year ago

The processing semantics are definitely something we should consider more sooner than later (although I assume "at least once" will be fine in most cases).

As long as the processing logic is idempotent, at least once is sufficient. It should be ok for the analyzers.

Reading through this list, there are many topics that will depend on the user's expected workload. Meaning we will need to experiment how certain setups behave, and provide proper documentation and example configs. Referring to things like partition count, replication factor etc. here.

Yes, that was my goal. List all of the concerns, identify those that should be enforce by DT by default, those related to user's workload and provide sound & safe config/recommendations.

DependencyTrack / hyades