search

aws / amazon-mwaa-docker-images

Apache License 2.0
24 stars 11 forks source link

Fix Hybrid Container Healthcheck Throttling #136

Closed kuyperse closed 3 weeks ago

kuyperse commented 3 weeks ago

Issue #, if available: N/A

Description of changes:

Discovered an issue where the worker healthchecks were not being performed in the hybrid container when the worker subprocess was run after the scheduler subprocess. Upon further investigation the root cause was determined to be the throttling conditions for the method _check_process_conditions which restricted the container to one call of the function per 60 seconds. This PR removes calls to _check_process_conditions in the subprocess execution_loop_iter loop if the subprocess does not have conditions. Also changes the throttle settings of _check_process_conditions to be instance level rather than global so that it is throttled per subprocess.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Kytha commented 3 weeks ago

Were you able to pinpoint why this makes a difference?

kuyperse commented 3 weeks ago

Did some additional testing and discovered the root cause as to why this fixes the issue. We currently throttle on _check_process_conditions and limit the function to 1 call per 60 seconds. In the hybrid container case the scheduler sub-processes runs first and calls _check_process_conditions, so when the worker sub-process attempts to call the function afterwards it is throttled. To remedy this I will remove calls to _check_process_conditions if the current sub-process does not have conditions and will also restrict throttling to the instance level rather than global.