search

openshift / origin-server

OpenShift 2 (deprecated)
889 stars 516 forks source link

Fix input/output deadlock condition #6449

Closed ironcladlou closed 7 years ago

ironcladlou commented 7 years ago

If the rate of input and output are very high, the message queue size can trend towards zero. In this case, it's possible that the Input will decide to evict from the queue, but by the time the eviction occurs the output has drained the queue to zero, causing the eviction dequeue to block forever. In this case, Input is now blocked trying to evict, and Output is blocked waiting for Input.

Fix the deadlock by distinguishing between eviction of old messages and dropping of new messages, and making those operations non-blocking.

Add a test which verifies the fix under these conditions.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1464125

thrasher-redhat commented 7 years ago

LGTM. Currently having Miciah take a look as well.

ironcladlou commented 7 years ago

Addressed the great feedback from @Miciah (thanks!)

Miciah commented 7 years ago

openshift-bot, do you remember how to [test][extended:cartridge,gear]?

thrasher-redhat commented 7 years ago

Manual tests passed. @ironcladlou can you squash these commits and then we should be good to go.

ironcladlou commented 7 years ago

Squashed

abhgupta commented 7 years ago

[merge] Reviewed by @thrasher-redhat and @Miciah

thrasher-redhat commented 7 years ago

@abhgupta It looks like openshift-bot's v2 functionality has been removed. Can you merge it manually?

abhgupta commented 7 years ago

Tested by Rory/Tim locally.