Closed topherinternational closed 1 month ago
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
Yes. Removing the incompatible providers might be a good idea - feel free to do it. Adding "matrix" of provider check will be too much of an overhead, especially that all the PRs are only buiding "airflow 3.8" image and only selected PRs will run tests for all versions of Python, so this is not a good idea to run all those.
Feel free to work on it.
@potiuk I think this issue is done with #41967, #41991 and stuff in #41555.
Yep
Apache Airflow version
2.10.0
If "Other Airflow 2 version" selected, which one?
No response
What happened?
In the
Provider checks / Provider packages wheel build and verify
CI test task, the command:breeze release-management verify-provider-packages --use-packages-from-dist --package-format wheel --use-airflow-version wheel --airflow-constraints-reference default --providers-constraints-location /files/constraints-3.8/constraints-source-providers-3.8.txt
installs all of the provider wheels built in the earlier
breeze release-management prepare-provider-packages ...
step.This process all runs on 3.8 and so implicitly assumes that all of the providers are legit for 3.8; neither step looks like it checks
excluded-python-versions
. In the best case, this results in testing of a provider that wasn't intended to be installed on 3.8. In the case of #41555, a provider can't be installed on 3.8 due to package conflicts with an existing dependency, so CI fails for the very reason we set the provider to exclude 3.8 in the first place.These build and install commands happen in
Provider packages wheel build and verify
, and also inCompat 2.8.4:P3.8 provider check
andCompat 2.9.1:P3.8 provider check
.So there's a couple of problems:
I checked the repo and none of the released providers exclude Python 3.8, I figure maybe this problem hasn't come up yet.
What you think should happen instead?
Maybe some options are?
prepare-provider-packages
check excluded versions before building a particular provider wheelProvider packages wheel build and verify
, and have the task reference the provider dependencies json and the excluded python versions.Another thing to consider is whether CI should run the provider checks on all supported Airflow Python versions (3.8-3.12) - that would add 16 tasks to the CI suite, and validate each provider on every version it claims to be installable to.
How to reproduce
excluded-python-versions = ['3.8']
to the provider.yamlProvider checks / Provider packages wheel build and verify
test task; it will build and install the provider on 3.8 even though the provider metadata excludes it from 3.8Operating System
CI, but repros on OS X
Versions of Apache Airflow Providers
No response
Deployment
Other
Deployment details
Github CI
Anything else?
No response
Are you willing to submit PR?
Code of Conduct