astronomer / astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
https://astro-sdk-python.rtfd.io/
Apache License 2.0
329 stars 40 forks source link

Disable schedule astro-deploy job and run the RC test scheduled job only once a day #2136

Closed pankajkoti closed 2 months ago

pankajkoti commented 2 months ago

Since we have a job to test upstream RCs, and additionally our CI runs integrations tests on every PR, we can disable the scheduled astro deploy job for now to reduce our cloud costs. In the future, in case we need to run this on schedule, we can enable it again by uncommenting the schedule.

Additionally, I am changing the RC test scheduled job to run only once a day instead of running it twice. We need to further optimise that job to trigger a deploy only when actual RCs are out. At the moment, we run the deploy and master DAG irrespective of RCs being out.