delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
337 stars 71 forks source link

Rebalance handling: reduce time holding write lock on signal #150

Closed gterzian closed 9 months ago

gterzian commented 10 months ago

This does not fix any open issue, it is rather something that was itching while reading the code.

I'm wondering if you want to hold the write lock on the rebalance signal while handling the rebalance, so I've tried to minimize the time the lock is held.

Current logic appears to be:

With these changes, the base case--when there is no error handling the signal--will only see the write lock acquired once, and held for so long as it takes to call take on an Option<_>.

rtyler commented 9 months ago

:confused: I am really not sure why CI is not running for this pull request :confused:

gterzian commented 9 months ago

I am really not sure why CI is not running for this pull request

I think it's because it's from a fork. See https://docs.github.com/en/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks

cc @rtyler

thanks for the merge!

rtyler commented 9 months ago

@gterzian I believe that there was likely a GitHub Actions outage at the time. The prompt to Approve and Run did show up again over the weekend :shrug: