search

conda-forge / airflow-feedstock

A conda-smithy repository for airflow.
BSD 3-Clause "New" or "Revised" License
6 stars 17 forks source link

Update to v2.7.0 #117

Closed xylar closed 1 year ago

xylar commented 1 year ago

Checklist

closes #115

conda-forge-webservices[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

xylar commented 1 year ago

@conda-forge-admin, please rerender

xylar commented 1 year ago

@millsks, as promised in https://github.com/conda-forge/airflow-feedstock/issues/115#issuecomment-1684384266, I'm going to explain my process and you can decide if you want to become a maintainer.

A long time ago, I forked this repo. I have a location on my laptop where I have both the main airflow-feedstock repo and my fork available as remotes.

In this location, I first created a branch for 2.7.0 off of main. That was the starting point for this branch.

In another location, I downloaded and untarred the source from GitHub for both the current and the previous versions of Airflow:

wget https://github.com/apache/airflow/archive/refs/tags/2.7.0.tar.gz
wget https://github.com/apache/airflow/archive/refs/tags/2.6.3.tar.gz
tar xvf 2.7.0.tar.gz 
tar xvf 2.6.3.tar.gz

(Normally, I would just look at the changes directly on GitHub but in this case there are ~1,500 files changed so that isn't practical).

I got the sha256 hash with:

sha256sum 2.7.0.tar.gz

The changes to the main package's dependencies are in setup.cfg, so I did:

diff airflow-2.6.3/setup.cfg airflow-2.7.0/setup.cfg

The relevant changes are:

39d38
<     Programming Language :: Python :: 3.7
58c57
< python_requires = ~=3.7
---
> python_requires = ~=3.8
76d74
<     cached_property>=1.5.0;python_version<="3.7"
98c96
<     flask-appbuilder==4.3.1
---
>     flask-appbuilder==4.3.3
114c112
<     jsonschema>=4.0.0
---
>     jsonschema>=4.18.0
122a121,123
>     # Pip can not find a version that satisfies constraints if opentelemetry-api is not pinned.
>     opentelemetry-api==1.15.0
>     opentelemetry-exporter-otlp
124c125
<     pathspec~=0.9.0
---
>     pathspec>=0.9.0
128,129c129,131

I can see that Airflow has dropped python 3.7 support, but this isn't relevant to us because conda-forge dropped 3.7 support a long time ago. I made the other changes corresponding to the dependencies in recipe/meta.yaml in my 2.7.0 branch.

The next part is to handle the "extras" that are not providers. These are the airflow-with-* packages produced here. The dependencies for these are in setup.py:

diff airflow-2.6.3/setup.py airflow-2.7.0/setup.py

It's pretty easy to pick out the parts that relate to dependency changes. Here are what I noticed:

268,270c299,301
<     # Click 8.1.4 breaks our mypy checks. The upper limit can be lifted when the
<     # https://github.com/apache/airflow/issues/32412 issue is resolved
<     "click>=8.0,<8.1.4",
---
>     # click 8.1.4 and 8.1.5 generate mypy errors due to typing issue in the upstream package:
>     # https://github.com/pallets/click/issues/2558
>     "click>=8.0,!=8.1.4,!=8.1.5",
276,278d306
<     # Without this, Sphinx goes in to a _very_ large backtrack on Python 3.7,
<     # even though Sphinx 4.4.0 has this but with python_version<3.10.
<     'importlib-metadata>=4.4; python_version < "3.8"',
296c324
<     "flask-appbuilder[oauth]==4.3.1",
---
>     "flask-appbuilder[oauth]==4.3.3",
320c348
< otel = ["opentelemetry-api==1.15.0", "opentelemetry-exporter-otlp", "opentelemetry-exporter-prometheus"]
---
> otel = ["opentelemetry-exporter-prometheus"]
352c380
<     "mypy==1.0.0",
---
>     "mypy==1.2.0",
376d403
<     "asynctest~=0.13",
401a429
>     "pytest-mock",

Some of these relate to providers, others to documentation and others to testing. We don't need any of those. But I searched for the packages listed here and any that are in an airflow-with-* output I updated following these diffs. In this case, it was just the flask-appbuilder changes that are relevant.

This seems like a pretty easy update. Sometimes, there are new dependencies that aren't on conda-forge and we need to add them. Sometimes, new versions of dependencies required by airflow (or its providers) are not available and we need to help maintain the dependencies, too.

Then, I made this PR. Following the instructions in the checklist, I rerendered.

When the testing failed, I found out that the licenses needed to be updated. I put in the new list of licenses in the licenses directory in the source code.

Fingers crossed that things work now...

github-actions[bot] commented 1 year ago

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

Thus the PR was passing and merged! Have a great day!