Closed kuyperse closed 2 months ago
Were you able to pinpoint why this makes a difference?
Did some additional testing and discovered the root cause as to why this fixes the issue. We currently throttle on _check_process_conditions
and limit the function to 1 call per 60 seconds. In the hybrid container case the scheduler sub-processes runs first and calls _check_process_conditions, so when the worker sub-process attempts to call the function afterwards it is throttled. To remedy this I will remove calls to _check_process_conditions if the current sub-process does not have conditions and will also restrict throttling to the instance level rather than global.
Issue #, if available: N/A
Description of changes:
Discovered an issue where the worker healthchecks were not being performed in the hybrid container when the worker subprocess was run after the scheduler subprocess. Upon further investigation the root cause was determined to be the throttling conditions for the method
_check_process_conditions
which restricted the container to one call of the function per 60 seconds. This PR removes calls to_check_process_conditions
in the subprocessexecution_loop_iter
loop if the subprocess does not have conditions. Also changes the throttle settings of_check_process_conditions
to be instance level rather than global so that it is throttled per subprocess.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.