search

getsentry / arroyo

A library to build streaming applications that consume from and produce to Kafka.
https://getsentry.github.io/arroyo/
Apache License 2.0
39 stars 6 forks source link

ref(rust): Remove strategy.close #361

Closed untitaker closed 1 month ago

untitaker commented 1 month ago

I long felt that nobody ever does anything useful in this method in Rust or in Python. And it makes it a bit harder to think about what "state" the strategy is in, since strategy.join() sometimes calls close() and then can no longer be called multiple times.

untitaker commented 1 month ago

Also strategy.close is incorrectly implemented in rust's RunTaskInThreads. it closes the next strategy before it does its own flushing, which leads to this sequence of events on rebalancing:

  1. StreamProcessor calls self.close
  2. ...calls next_step.close
  3. StreamProcessor calls self.join
  4. ...calls next_step.submit x 100
  5. ...calls next_step.join

according to the API contract, next_step could panic at step 4 because close was already called. luckily nobody ever implemented that in Rust