search

elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.
https://elyra.readthedocs.io/en/stable/
Apache License 2.0
1.84k stars 344 forks source link

Add support for Airflow 2.x as pipeline runtime #2392

Open ptitzler opened 2 years ago

ptitzler commented 2 years ago

Is your feature request related to a problem? Please describe.

Support for Airflow 1.10 is now limited, as EOL was reached June 2021.

Describe the solution you'd like Add support for Apache Airflow 2.0

jhgoebbert commented 1 year ago

Are there any news on Apache Airflow 2.x support?

I read that it "might" work but it is not tested in the Elyra documentation.

ptitzler commented 1 year ago

@lresende

lresende commented 1 year ago

I was able to get Airflow "runtime specific components" working well, but was not able to try the generic components.

NOT A CONTRIBUTION

jhgoebbert commented 1 year ago

I was able to get Airflow "runtime specific components" working well, but was not able to try the generic components.

That are good news. So there is a high chance that Apache Airflow 2.x actually might work out-of-the-box.
Or did you have to patch Elyra for this?

shalberd commented 1 year ago

Yes, it seems the generic Airflow operators are working (Airflow Package Catalog Connector), while the Airflow Provider Package Catalog Connectors are not, especially with providers written for Airflow 2.x. E.g. in its version for Airflow 2.x, the https://pypi.org/project/apache-airflow-providers-amazon/#files is not working, too.

I was interested in using the DataRobot Provider for Airflow 2.x in the Elyra context, but this, too, is not yet possible.

https://github.com/datarobot/airflow-provider-datarobot/issues/8