astronomer / astro-cli

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
https://www.astronomer.io
Other
361 stars 71 forks source link

Add `--from-template` flag on CLI Project Initialization to setup template based project #1728

Closed pritt20 closed 4 days ago

pritt20 commented 1 month ago

Description

This PR includes changes to add --templates flag to astro dev init command. This option will allow users to specify runtime templates name like etl, dbt-on-astro, generative-ai etc and CLI will setup an astro project based on provided template by user.

These changes are extension to the enhancements being made in onboarding-workflow where once user has completed the process of creating the deployment and now wants to make changes to deployed dags or deployment, we want them to explore astro-cli option.

These changes will make it easy for users to modify/update dags as the template based project would setup all the required files which were auto-deployed during deployment create process as part of onboarding-workflow.

🎟 Issue(s)

Related https://github.com/astronomer/astro/issues/24343

πŸ§ͺ Functional Testing

Tested the change on local machine:

Template: generative-ai

astro dev init --from-template=generative-ai
Initializing Astro project
Pulling Airflow development files from Astro Runtime 12.2.0
Initialized empty Astro project in /Users/pritt/Astronomer/plg/astro-cli/airflow-ai
cat requirements.txt                                                                   
# Astro Runtime includes the following pre-installed providers packages: https://docs.astronomer.io/astro/runtime-image-architecture#provider-packages
duckdb==0.10.3
tabulate==0.9.0
sentence-transformers==3.0.1
cd dags && ls                                                                             
example_vector_embeddings.py

Template: etl

astro dev init --from-template=etl          
Initializing Astro project
Pulling Airflow development files from Astro Runtime 12.2.0
Initialized empty Astro project in /Users/pritt/Astronomer/plg/astro-cli/airflow-etl
cat requirements.txt                                                                     
# Astro Runtime includes the following pre-installed providers packages: https://docs.astronomer.io/astro/runtime-image-architecture#provider-packages
duckdb==0.10.3
tabulate==0.9.0% 
cd dags && ls                                                                            
example_etl_galaxies.py

Default Project:

astro dev init
Initializing Astro project
Pulling Airflow development files from Astro Runtime 12.2.0
Initialized empty Astro project in /Users/pritt/Astronomer/plg/astro-cli/test
cat requirements.txt                                                               
# Astro Runtime includes the following pre-installed providers packages: https://www.astronomer.io/docs/astro/runtime-image-architecture#provider-packages
cd dags && ls                                                                         
exampledag.py

Interactive UX test:

Template: dbt-on-astro

astro dev init --from-template
 #     TEMPLATE             
 1     dbt-on-astro         
 2     learning-airflow     
 3     generative-ai        
 4     etl                  

> 1
Initializing Astro project
Pulling Airflow development files from Astro Runtime 12.2.0
Initialized empty Astro project in /Users/pritt/Astronomer/plg/astro-cli/airflow-dbt
cat requirements.txt
astronomer-cosmos==1.6.0
cd dags && ls                               
dbt_cosmos_dag.py

Template: learning-airflow

astro dev init --from-template
 #     TEMPLATE             
 1     dbt-on-astro         
 2     learning-airflow     
 3     generative-ai        
 4     etl                  

> 2
Initializing Astro project
Pulling Airflow development files from Astro Runtime 12.2.0
Initialized empty Astro project in /Users/pritt/Astronomer/plg/astro-cli/airflow-la
cd dags && ls
example_astronauts.py

πŸ“Έ Screenshots

Add screenshots to illustrate the validity of these changes.

πŸ“‹ Checklist

schnie commented 4 days ago

Blocking feedback request was handled and approved async from Github. Administrative merge for that reason.

lzdanski commented 1 day ago

Docs update required: https://www.astronomer.io/docs/astro/cli/astro-dev-init, requires additional content https://www.astronomer.io/docs/astro/cli/develop-project#create-an-astro-project