We need our source of truth to list all of the ingests, and rather than enumerating all of the ingests in the Jenkins file - we need to wrap the process to execute them in a single python call.
In addition to the bare minimum of executing, it would be good if the ingests can run with some parallelism, and also as a stretch goal, if we can separate out the logs.
I'm going to take a shot at using Prefect for this, but I'll wrap it in a Typer CLI. I also plan to remove the Dagster pipeline code alongside this change.
We need our source of truth to list all of the ingests, and rather than enumerating all of the ingests in the Jenkins file - we need to wrap the process to execute them in a single python call.
In addition to the bare minimum of executing, it would be good if the ingests can run with some parallelism, and also as a stretch goal, if we can separate out the logs.
I'm going to take a shot at using Prefect for this, but I'll wrap it in a Typer CLI. I also plan to remove the Dagster pipeline code alongside this change.