search

astronomer / dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Apache License 2.0
1.16k stars 176 forks source link

Help Required: XComs in YAML file #109

Open thomasroshin opened 2 years ago

thomasroshin commented 2 years ago

I am trying to create dynamic DAGs using YAML template and dag-factory. With a small example I am able to access MSSQL database and insert data. As a next step I want to perform EL(T)/E(T)L and from airflow side I can perform the extract from DB1 and load to DB(2) with XComs (for small datasets or S3 as XCom backend) . But how would dag-factory handle the XComs or how can I pass the extract data to the load step (keyword in YAML). Apologize if I missed something obvious in the documentation.

Thank you.

julienT-livejourney commented 2 years ago

It seems that something like _arg: "{{task_instance.xcom_pull(task_ids='previous_taskid')}}" works.

vishwa3908 commented 2 years ago

It seems that something like _arg: "{{task_instance.xcom_pull(task_ids='previous_taskid')}}" works.

can you show us the code where you have used xcom