search

getindata / kedro-airflow-k8s

Kedro Plugin to support running pipelines on Kubernetes using Airflow.
https://kedro-airflow-k8s.readthedocs.io
Apache License 2.0
29 stars 11 forks source link

Update kedro dependencies #139

Open em-pe opened 2 years ago

em-pe commented 2 years ago

Update plugin dependencies to recent versions:

kedro - 0.18.*

dgajewski1 commented 1 year ago

Support for further versions of kedro, airflow and mlflow

This plugin requires dependency updates, with probable breaking changes. This prevents users from using latest and greatest mlflow, kedro and airflow versions which is not only important feature-wise but also from the security standpoint as there are critical CVEs.

Looking at setup.py - the culprits are:

Kedro locked to an old version

"kedro>=0.16, <0.18",

Dependent plugin which prevents update to MLFlow 2.X

kedro-mlflow==0.4.1"

Outdated Airflow Plugins

"apache-airflow[kubernetes]==2.1.0", "apache-airflow-providers-cncf kubernetes==1.1.0", "apache-airflow-providers-apache-spark==2.0.1"\