astronomer / astro-cli

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
https://www.astronomer.io
Other
341 stars 70 forks source link

Allow Parallel dag deploys #1586

Open kushalmalani opened 4 months ago

kushalmalani commented 4 months ago

Currently, if we perform parallel dag deploys, you get the following error

RuntimeError: + astro deploy --dags
Error: ===== RESPONSE ERROR (ErrorCode=InvalidBlockList) =====
Description=The specified block list is invalid.

This happens because we do not assign a unique block id for every dag deploy. We would to update our upload to blob storage flow to have unique blockIds on every push to allow uploading parallel to the same blob.

Slack Conversation - https://astronomer.slack.com/archives/CL44PA4DB/p1709191805125909