Update the AutoScalingCondition such that it returns a non-successful code if the worker is idle, instead of calling the worker shutdown handler. The latter isn't working properly at the moment, as explained in Issue #105. Furthermore, the worker shutdown handler was more suitable in the internal code where we didn't have a standalone class for managing processes, along with conditions that can be employed to terminate processes. With the open source code, however, terminated a process shouldn't be done indirectly like this, and instead a process condition can just return the right response in the check() method to get a process terminated.
Ensure that process is closed (and accordingly all its conditions) at exit. This is achieved by using the atexit module to ensure the close() method is called.
Hide some of the process internal logs regarding process termination due to failing condition, as those are more suitable for the service than the customer.
Add PYTHONPATH to the list of protected enviornment variables so the customer cannot override it in start-up scripts.
Adding Airflow constraints during the bootstrap step of installing Airflow to avoid installing newer versions of the packages that Airflow depends on, as that can break Airflow.
Some refactoring + A couple more logs + fixing some typos.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Issue #, if available: #105
Description of changes:
atexitmodule to ensure theclose()method is called.PYTHONPATHto the list of protected enviornment variables so the customer cannot override it in start-up scripts.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.