search

getsentry / arroyo

A library to build streaming applications that consume from and produce to Kafka.
https://getsentry.github.io/arroyo/
Apache License 2.0
41 stars 7 forks source link

fix: Try fixing InvalidStateError #304

Closed lynnagara closed 10 months ago

lynnagara commented 10 months ago

Not confident this is going to do anything but I don't really have other ideas. For some reason the consumer occasionally does not seem to be truly paused even when the self.__is_paused flag is set.

untitaker commented 10 months ago

my current suspicion is that we are applying backpressure while new partitions are being assigned, somehow. so we pause the consumer here:

https://github.com/getsentry/arroyo/blob/ba85564a91f95b0188d626dd3c03bb855f15726c/arroyo/processing/processor.py#L402

but at the next poll, we get a partition assigned, which is not paused

then we fail here; https://github.com/getsentry/arroyo/blob/ba85564a91f95b0188d626dd3c03bb855f15726c/arroyo/processing/processor.py#L352-L354

I wonder if we can just remove the assertion, the StreamProcessor does not really need it i think

lynnagara commented 10 months ago

Agree we don't need the assertion. Removed in https://github.com/getsentry/arroyo/pull/305

lynnagara commented 10 months ago

Superceded by https://github.com/getsentry/arroyo/pull/305