We want to start using a stand-alone DAG processor for a better separation. This commit does that. Additionally, I also added the trigger to the scheduler container.
To achieve these two, I changed the Subprocess class such that the execution loop (where we monitor the process and captures its logs) manually, meaning that it is possible to create multiple parallel subprocesses.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Issue #, if available:
Description of changes:
We want to start using a stand-alone DAG processor for a better separation. This commit does that. Additionally, I also added the trigger to the scheduler container.
To achieve these two, I changed the Subprocess class such that the execution loop (where we monitor the process and captures its logs) manually, meaning that it is possible to create multiple parallel subprocesses.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.