Import astronomer registry dags through astro cli

[odaneau-astro]

• [x] I have checked that a similar feature request does not already exist.

✍️ Is your feature request related to a problem? Please describe. No a problem per se, more of an idea/suggestion.

🧩 Describe the solution you'd like

When creating a new project using the astro cli init, two simple examples are included by default. Those examples are ok to understand core concepts of Airflow, but do not help users deliver business value right away.

The Astronomer registry is a fantastic tool to help deliver business value quickly. It offers a wide variety of Dags to jumpstart common use cases such as running astro sdk ETLs, triggering DBT models, loading csv files to Snowflake, executing pipelines in Databricks, etc.

Having the registry Dags accessible through the CLI would allow beginners to start authoring with examples they can relate to and "fill the blanks" to deliver business value faster.

The value-add behind this is to provide a more "batteries included" experience and to accelerate "time-to-business-value" for users.

There are few different ways to go about how this could work:

1. Add an option in the astro dev init command to ask users if they want to add examples from the registry.
2. Add a command line argument to the astro dev init --registry-url=https://registry.astronomer.io/dags/databricks-ml-example
3. Add a new import command such as astro import dag --registry-url=https://registry.astronomer.io/dags/databricks-ml-example -> This way we could also import providers, modules, etc.

🤔 Describe alternatives you've considered

The current alternative is to browse through the Astronomer registry, search the sample that fits your needs, clone the repo locally, then copy-paste the dag code into the dags folder of the astro cli project.

Not only this takes more time and effort, newcomers don't always know the registry even exists.

I'm really open to feedback on this and curious to see if others see any potential value here.

[cmarteepants]

@paolaperaza @sunkickr @virajmparekh -- this is a really cool idea. Do we have this in productboard?

[sunkickr]

I think this is a really cool idea. We could potentially have the CLI query the registry DB directly. With something like astro registry-dags list <search-term> and then select a dag to import

[odaneau-astro]

Would be nice to include Airflow's example DAGs as well. For example. Google's dags here

[josh-fell]

❤️ this idea! The team is currently giving the Registry a glow-up including slick APIs so once those are stable, the CLI could probably use them.

Would be nice to include Airflow's example DAGs as well. For example. Google's dags here

@odaneau-astro These example DAGs are all disappearing from the Airflow repo and being transitioned to system tests via AIP-47 and therefore disappearing from the Registry. There used to be example DAGs for every core provider. Using the system test examples is underwhelming because now that have extra system-test-related functions and format that make them less user-friendly and more confusing.

[fritz-astronomer]

Oh. I forgot this was a requested thing 👀 @odaneau-astro , how's your Go? Wanna help out?

1148

2nd'ing adding Airflow and Google's example DAGs