We encountered a problem where fiaas-deploy-daemon would hang for an extended period during a control plan rotation.
A traceback indicated it was still listening for events from the previous control plane, that wasn't sending any. It's still uncertain if this was an active connection or one dropped silently by the server.
Lowering the stream_timeout is the chosen solution, but it comes with a few problems, especially that before this, the watcher thread for exit with an exception when the stream times out. That's solved by cathing the timeout and returning normally. Additionally, resource versions are now supported by the watching, including handling the 410 Gone response.
oyvindio
commented
9 months ago
We encountered a problem where fiaas-deploy-daemon would hang for an extended period during a control plan rotation. A traceback indicated it was still listening for events from the previous control plane, that wasn't sending any. It's still uncertain if this was an active connection or one dropped silently by the server.
Lowering the stream_timeout is the chosen solution, but it comes with a few problems, especially that before this, the watcher thread for exit with an exception when the stream times out. That's solved by cathing the timeout and returning normally. Additionally, resource versions are now supported by the watching, including handling the 410 Gone response.