This PR refactors the run loop structure and adds support for specifying explicit starting offsets.
Apologies, there is a lot to look at here, but it needed to be done. The lions share of changes are in lib.rs.
The main changes include:
Made the entry point a function (start_ingest) instead of a struct method and we now execute the run loop directly within it.
Centralized control as much as possible to execute in the run loop.
Merged ProcessingState and IngestProcessor into just IngestProcessor.
Instead of an Arc<Mutex<PartitionAssignment>>, we now have an Arc<RwLock<Option<RebalanceSignal>>> for responding to rebalance events so we no longer have to hold a lock while updating the PartitionAssignment.
In the near future, I think we may look at breaking some files out of lib.rs for another cleanup pass.
This PR refactors the run loop structure and adds support for specifying explicit starting offsets.
Apologies, there is a lot to look at here, but it needed to be done. The lions share of changes are in
lib.rs.The main changes include:
start_ingest) instead of a struct method and we now execute the run loop directly within it.ProcessingStateandIngestProcessorinto justIngestProcessor.Arc<Mutex<PartitionAssignment>>, we now have anArc<RwLock<Option<RebalanceSignal>>>for responding to rebalance events so we no longer have to hold a lock while updating the PartitionAssignment.In the near future, I think we may look at breaking some files out of
lib.rsfor another cleanup pass.