search

sul-dlss / dlme-airflow

This is a new repository to capture the work related to the DLME ETL Pipeline and establish airflow
Apache License 2.0
1 stars 0 forks source link

Update dependencies #558

Closed edsu closed 1 week ago

edsu commented 3 weeks ago

I noticed that our dependencies were quite out of date, so I updated them. I tested locally with docker compose and Airflow ran, loaded the DAGS and I watched a harvest run for a bit.

Since apache-airflow-providers-docker was listed as a dependency in our pyproject.toml it was pulling all of the latest airflow into our requirements.txt. We really only need airflow and apache-airflow-providers-docker in requirements-dev.txt for testing since they come with the Airflow Docker image we are using.

Specifying apache-airflow-providers-docker in the pyproject.toml added to quite a bit of install churn when we installed with:

$ uv pip install -r requirements.txt

and then did a:

$ uv run bin/get

which would go and fetch the latest airflow and overwrite the version we wanted because uv was looking to the pyproject.toml file.

I updated the README.md to reflect that we need to run the bin commands with uv run.