search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.51k stars 14.14k forks source link

Help Needed: Running Airflow DAG with Dependencies in multiple Conda Environments #42616

Open HarikrishnanK9 opened 2 hours ago

HarikrishnanK9 commented 2 hours ago

Apache Airflow version

2.10.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Hi, I’m seeking assistance with my Airflow setup. My Airflow instance runs in an environment named prod_env, but the modules required for a particular DAG only work in a separate Conda environment named (ind_env).

I require guidance on managing dependencies across different environments to work within Airflow

What you think should happen instead?

No response

How to reproduce

install airflow in one environment and create some dags,and create another environment and there you have some python modules related to a particular dag ;you have to call them in ur current environment.Remember the python packages of ur cuurent environment not directly support those packages currently setup in another environment.

Operating System

Linux

Versions of Apache Airflow Providers

apache-airflow==2.10.2 apache-airflow-providers-amazon==8.28.0 apache-airflow-providers-celery==3.8.1 apache-airflow-providers-common-compat==1.2.0 apache-airflow-providers-common-io==1.4.0 apache-airflow-providers-common-sql==1.14.2 apache-airflow-providers-elasticsearch==5.5.0 apache-airflow-providers-fab==1.2.2 apache-airflow-providers-ftp==3.11.0 apache-airflow-providers-google==10.21.1 apache-airflow-providers-http==4.12.0 apache-airflow-providers-imap==3.6.1 apache-airflow-providers-smtp==1.7.1 apache-airflow-providers-sqlite==3.8.1

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 2 hours ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.