Provider checks CI/breeze release-management assume all providers are 3.8-compatible

topherinternational commented 2 months ago

Apache Airflow version

2.10.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

In the Provider checks / Provider packages wheel build and verify CI test task, the command:

breeze release-management verify-provider-packages --use-packages-from-dist --package-format wheel --use-airflow-version wheel --airflow-constraints-reference default --providers-constraints-location /files/constraints-3.8/constraints-source-providers-3.8.txt

installs all of the provider wheels built in the earlier

breeze release-management prepare-provider-packages ... step.

This process all runs on 3.8 and so implicitly assumes that all of the providers are legit for 3.8; neither step looks like it checks excluded-python-versions. In the best case, this results in testing of a provider that wasn't intended to be installed on 3.8. In the case of #41555, a provider can't be installed on 3.8 due to package conflicts with an existing dependency, so CI fails for the very reason we set the provider to exclude 3.8 in the first place.

These build and install commands happen in Provider packages wheel build and verify, and also in Compat 2.8.4:P3.8 provider check and Compat 2.9.1:P3.8 provider check.

So there's a couple of problems:

Any providers that are specifically marked not to be used on 3.8 are installed during CI, which could cause issues with 3.8-compatible providers or their dependencies
Providers never get tested for installation on any newer Python versions

I checked the repo and none of the released providers exclude Python 3.8, I figure maybe this problem hasn't come up yet.

What you think should happen instead?

Maybe some options are?

Make prepare-provider-packages check excluded versions before building a particular provider wheel
Add the "Remove incompatible provider packages" stage to Provider packages wheel build and verify, and have the task reference the provider dependencies json and the excluded python versions.

Another thing to consider is whether CI should run the provider checks on all supported Airflow Python versions (3.8-3.12) - that would add 16 tasks to the CI suite, and validate each provider on every version it claims to be installable to.

How to reproduce

Take any provider and add excluded-python-versions = ['3.8'] to the provider.yaml
Open an airflow project PR with the change to run Github CI
Observe the Provider checks / Provider packages wheel build and verify test task; it will build and install the provider on 3.8 even though the provider metadata excludes it from 3.8

Operating System

CI, but repros on OS X

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

Github CI

Anything else?

No response

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

boring-cyborg[bot] commented 2 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

potiuk commented 2 months ago

Yes. Removing the incompatible providers might be a good idea - feel free to do it. Adding "matrix" of provider check will be too much of an overhead, especially that all the PRs are only buiding "airflow 3.8" image and only selected PRs will run tests for all versions of Python, so this is not a good idea to run all those.

Feel free to work on it.

topherinternational commented 1 month ago

@potiuk I think this issue is done with #41967, #41991 and stuff in #41555.

potiuk commented 1 month ago

Yep

apache / airflow