search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.65k stars 14.18k forks source link

Status of testing Providers that were prepared on April 07, 2022 #22836

Closed potiuk closed 2 years ago

potiuk commented 2 years ago

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 3.3.0rc1

The guidelines on how to test providers can be found in

Verify providers by contributors

Committer

mnojek commented 2 years ago

From my side the testing for Google was done :) #22311

vincbeck commented 2 years ago

All mine are doc and sample dag related so it's all good

SasanAhmadi commented 2 years ago

from my side:

shuhoy commented 2 years ago

From my side, test run of BigQueryToGCSOperator was done. #22506

raphaelauv commented 2 years ago

I can't check the KPO 4.0.0rc1

because it ask for airlfow>=2.3.0rc1

Collecting apache-airflow-providers-cncf-kubernetes==4.0.0rc1
  Downloading apache_airflow_providers_cncf_kubernetes-4.0.0rc1-py3-none-any.whl (51 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.0/51.0 KB 807.9 kB/s eta 0:00:00
Requirement already satisfied: cryptography>=2.0.0 in /usr/local/lib/python3.9/site-packages (from apache-airflow-providers-cncf-kubernetes==4.0.0rc1->-r requirements-custom.txt (line 1)) (36.0.2)
Collecting kubernetes<24,>=21.7.0
  Downloading kubernetes-23.3.0-py2.py3-none-any.whl (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 713.0 kB/s eta 0:00:00
ERROR: Could not find a version that satisfies the requirement apache-airflow>=2.3.0rc1 (from apache-airflow-providers-cncf-kubernetes) (from versions: 1.10.9-bin, 1.8.1, 1.8.2rc1, 1.8.2, 1.9.0, 1.10.0, 1.10.1b1, 1.10.1rc2, 1.10.1, 1.10.2b2, 1.10.2rc1, 1.10.2rc2, 1.10.2rc3, 1.10.2, 1.10.3b1, 1.10.3b2, 1.10.3rc1, 1.10.3rc2, 1.10.3, 1.10.4b2, 1.10.4rc1, 1.10.4rc2, 1.10.4rc3, 1.10.4rc4, 1.10.4rc5, 1.10.4, 1.10.5rc1, 1.10.5, 1.10.6rc1, 1.10.6rc2, 1.10.6, 1.10.7rc1, 1.10.7rc2, 1.10.7rc3, 1.10.7, 1.10.8rc1, 1.10.8, 1.10.9rc1, 1.10.9, 1.10.10rc1, 1.10.10rc2, 1.10.10rc3, 1.10.10rc4, 1.10.10rc5, 1.10.10, 1.10.11rc1, 1.10.11rc2, 1.10.11, 1.10.12rc1, 1.10.12rc2, 1.10.12rc3, 1.10.12rc4, 1.10.12, 1.10.13rc1, 1.10.13, 1.10.14rc1, 1.10.14rc2, 1.10.14rc3, 1.10.14rc4, 1.10.14, 1.10.15rc1, 1.10.15, 2.0.0b1, 2.0.0b2, 2.0.0b3, 2.0.0rc1, 2.0.0rc2, 2.0.0rc3, 2.0.0, 2.0.1rc1, 2.0.1rc2, 2.0.1, 2.0.2rc1, 2.0.2, 2.1.0rc1, 2.1.0rc2, 2.1.0, 2.1.1rc1, 2.1.1, 2.1.2rc1, 2.1.2, 2.1.3rc1, 2.1.3, 2.1.4rc1, 2.1.4rc2, 2.1.4, 2.2.0b1, 2.2.0b2, 2.2.0rc1, 2.2.0, 2.2.1rc1, 2.2.1rc2, 2.2.1, 2.2.2rc1, 2.2.2rc2, 2.2.2, 2.2.3rc1, 2.2.3rc2, 2.2.3, 2.2.4rc1, 2.2.4, 2.2.5rc1, 2.2.5rc2, 2.2.5rc3, 2.2.5)
ERROR: No matching distribution found for apache-airflow>=2.3.0rc1
potiuk commented 2 years ago

@raphaelauv

I can't check the KPO 4.0.0rc1

Actually .. you can :) (see below).

because it ask for airlfow>=2.3.0rc1

That's cool and how it should be (except that it asks for airflow not airlfow that is :) (but typos are my own thing thay make far too ofen.

But to be cool - you actually can test it. You can install the provider in regular "main" breeze. The breeze in main is what soon will become v2-3-test branch and we will build 2.3.0 from it (and it is 2.3.0.dev version so it should be installable there).

Just remove sources of airflow/providers/cncf in Breeze/local sources (after starting a regular version of and do pip install apache-airflow-providers-cncf-kubernetes==4.0.0rc1 --no-deps and it should work.

raphaelauv commented 2 years ago

thanks @potiuk, I was not aware of --no-deps :+1:

since Airflow ( without the kubernetesExectutor ) do not have Kubernetes as a dependency

it's possible to install

apache-airlfow==2.2.5
kubernetes==21.7.0
apache-airflow-providers-cncf-kubernetes==4.0.0rc1 --no-deps

Is there any feature of airflow 2.3 that restrict the usage of the kubernetes provider 4.0.X in airlfow 2.2.5 ?

otherwise https://github.com/apache/airflow/pull/22412 -> WORK

potiuk commented 2 years ago

otherwise https://github.com/apache/airflow/pull/22412 -> WORK

Is there any feature of airflow 2.3 that restrict the usage of the kubernetes provider 4.0.X in airlfow 2.2.5 ?

Yeah. The main problem is that Airlfow 2.2.5 has KubernetesExecutor that both - uses cncf.kubernetes provider to run AND relies on old < 20.0.0 of kubernetes library. A lot of things will work with kubernetes==21.7.0 - but there are quite a few things that won't work unfortunately. An example (but this is just an example) is here: https://github.com/apache/airflow/issues/22560

The new provider depend on > 21 (for good reasons) and rather than risk that there are more things not working in Airflow 2.2.5 - we decided that 2.2. line and before will have to stick with the old (well - not THAT old) kubernetes provider. Hopefully there will not be serious "issues" with the old provider (we can always release a critical fix for the the 3. line of provider). But if you would like to use new features from kubernetes provider 4.0+, you will have to go Airflow 2.3+.

As a bonus, that might actually give people more incentive to move to Airlfow 2.3 at some point in time :).

potiuk commented 2 years ago

BTW. As a learning - in the future we might bundle K8s Executor with the provider, but that will require Airflow 3 and breaking changes I am afraid.

ttomasz commented 2 years ago

tested discord provider (#22570) and it works

o-nikolas commented 2 years ago

My changes are docs/example dags, so all good.

hankehly commented 2 years ago

22710 (google 6.8.0rc1) works as expected on 2022/04/09

pierrejeambrun commented 2 years ago

Tested https://github.com/apache/airflow/pull/22536 and working. I cannot test https://github.com/apache/airflow/pull/22479 but looking good :+1:

pankajastro commented 2 years ago

https://github.com/apache/airflow/pull/22772: Tested (google 6.8.0rc1) working as expected

alexott commented 2 years ago

22541 & #22571 both work. Found bug in #22422, fix will be today. So we'll need to cut rc2

potiuk commented 2 years ago

22541 & #22571 both work. Found bug in #22422, fix will be today. So we'll need to cut rc2

Thanks.

Re #22422 - it depends on the bug - if it's a bug in new operators which can be wokrarounded or does not block things entirely, we can still release the rc1 and follow up in the next release of the provider. We'd only refrain from releasing a provider for sure if there is a serious regression.

alexott commented 2 years ago

@potiuk as result of this, CreateRepository operator doesn't work at all - really strange that I missed it when testing :-( Opened #22885 to fix it.

potiuk commented 2 years ago

Ah.. Then yeah :) rc2 it is. Good that you tested !

josh-fell commented 2 years ago

22519 is good! Just a few docstring changes but confirmed in the package files.

repl-mike-roest commented 2 years ago

Double checked #22667 validated custom headers are delivered in the resulting email delivery

potiuk commented 2 years ago

Thanks for testing everyone!

I proceed with the release of all providers except the Databricks one. RC2 for databricks follows.