Closed ibukanov closed 1 month ago
[puLL-Merge] - brave-intl/challenge-bypass-server@708
This PR makes several changes to the Kafka package to improve performance, simplify error handling, and make the code more testable.
The main changes are:
Processor
function type and remove the ProcessingResult
structmessageReader
interface for testingprocessMessagesIntoBatchPipeline
to limit concurrency based on available CPU cores and simplify message passingprocessMessagesIntoBatchPipeline
Emit
and processor functionsbatchPipeline
capacity based on CPU cores assumes the message processors are CPU-bound. If they start doing more I/O, the capacity may need to be adjusted.handlePermanentIssuanceError
and handlePermanentRedemptionError
functions no longer return after emitting errors. This should be reviewed to ensure it matches the desired semantics for handling permanent errors.
Refactor processMessagesIntoBatchPipeline to ensure that it closes the message channel to signal that the message processing is ready including the case when the message topic does not match any of configured topics. For that move all processing of the message including the search for topic processors to runMessageProcessor() groutine and use defer to close the channel. That required to change the message channel to be of struct{} type and store the error in the message itself so a simple defer close(msg.done) can be used to cover both normal and error cases.
Provide unit tests for processMessagesIntoBatchPipeline that cover success and error paths.
Bound batchPipeline capacity by the number of CPU cores to avoid running too many CPU-intensive tasks in parallel.
Closes #702 Closes #703 Closes #704