Throttle Kafka consumption to the file writing speed - Githubissues

ess-dmsc / kafka-to-nexus

(Mirror: moved to https://gitlab.esss.lu.se/ecdc/ess-dmsc/kafka-to-nexus) Writes NeXus files from experiment data streamed through Apache Kafka.

BSD 2-Clause "Simplified" License

2 stars 4 forks source link

Throttle Kafka consumption to the file writing speed #686

Closed danesss closed 1 year ago

danesss commented 1 year ago

Issue

ECDC-3253: We currently read from Kakfa at max speed and buffer into memory, which can cause out-of-memory issues if Kafka is faster than the GPFS filesystem.

Description of work

We adapt the consumption rate to the file writing rate:

We add a new configuration setting for the size of write queue --max-queued-writes
We add a method in MessageWriter to be able to check current queue size
We extend StreamController to
- Perform a periodic check of the write queue size
- Pause all consumers for 200ms if queue is larger than configured
Topic and Partition classes have new methods to support pause and resume of the consumers.
Unit tests are added to verify pause/resume behaviour at the Partition level.
- To test the "queue_size -> pause" logic further changes are needed so they will be explored in separate PR: https://github.com/ess-dmsc/kafka-to-nexus/pull/689

Nominate for Group Code Review

[ ] Nominate for code review

Reminder

Changes should be documented in changes.md

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5590 3771 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5590 3778 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5590 3774 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5590 3781 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5590 3774 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5590 3780 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5600 3781 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5603 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5603 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5603 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5603 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5603 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5607 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5607 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5607 3781 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5607 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5607 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5607 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5607 3776 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5610 3775 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5610 3778 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5607 3774 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3777 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3773 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3774 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3773 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3773 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3774 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3774 67% For more detail see Cobertura report in Jenkins interface

cow-bot commented 1 year ago

Code Coverage (Lines Exec Cover) TOTAL 5606 3774 67% For more detail see Cobertura report in Jenkins interface