googleapis / python-pubsub

Apache License 2.0
392 stars 205 forks source link

fix: Race condition where future callbacks invoked before client is in paused state #1145

Closed mukund-ananthu closed 5 months ago

mukund-ananthu commented 7 months ago

For non-retriable errors returned from the backend, the client library currently sets future.set_exception() for all outstanding futures:

https://github.com/googleapis/python-pubsub/blob/8c7e2a9e1b158e2b75dec8c5d78be17ce2dd449b/google/cloud/pubsub_v1/publisher/_batch/thread.py#L285-L286

before it tries to change its state to paused by calling _batch_done_callback(): https://github.com/googleapis/python-pubsub/blob/8c7e2a9e1b158e2b75dec8c5d78be17ce2dd449b/google/cloud/pubsub_v1/publisher/_batch/thread.py#L289-L291

which transitions the state of the client library to a paused state: https://github.com/googleapis/python-pubsub/blob/ff229a5fdd4deaff0ac97c74f313d04b62720ff7/google/cloud/pubsub_v1/publisher/_sequencer/ordered_sequencer.py#L204-L206

This results in race conditions where the callback associated with the future is invoked before the client enters paused state. This causes issues if the callback attempts to do a resume_publish() before the client is in the paused state.

This fix ensures that the client is in paused state before future callbacks are invoked as a consequence of future.set_exception

Fixes #1144 🦕

mukund-ananthu commented 5 months ago

@kamalaboulhosn Bumping this up.