As part of our metrics calculation we need to spin up the Metrics Calculation Service (MCS). We don't want to keep this up at all times due to large costs of the systems required for metrics calculation.
General idea:
We need a python api that can be called from dagster that will spin up the MCS's frontend component. The frontend component will then need be in charge of spinning up the required workers (using dask).
The function used should block until /status returns 200 on the MCS
What is it?
As part of our metrics calculation we need to spin up the Metrics Calculation Service (MCS). We don't want to keep this up at all times due to large costs of the systems required for metrics calculation.
General idea:
/statusreturns 200 on the MCS