search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
35.31k stars 13.79k forks source link

Status of testing Providers that were prepared on June 18, 2021 #16456

Closed potiuk closed 3 years ago

potiuk commented 3 years ago

I have a kind request for all the contributors to the latest provider packages release. Could you help us to test the RC versions of the providers and let us know in the comment, if the issue is addressed there.

Providers that need testing

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 2.0.0rc2

Providers for which previous RC was fully tested (thanks @levyitay !)

Provider qubole: 2.0.0rc2

Providers that do not need testing

Those are providers that were either doc-only or had changes that do not require testing.

potiuk commented 3 years ago

This would be great it we could have confirmation for most of the providers by Friday afternoon, so that I could release them then :)

potiuk commented 3 years ago

@jmelot Also a kind request for you - it's the first release of Asana provider, so if you could test it as well and let us know in the comment, it would be great: https://pypi.org/project/apache-airflow-providers-asana/1.0.0rc1

zhzhang commented 3 years ago

Tested postgres connection changes on a Resdhift cluster, looks good to me.

Goodkat commented 3 years ago

Tested odbc connection, it works as expected with boolean parameters now: https://github.com/apache/airflow/issues/15016#issuecomment-826650629

jmelot commented 3 years ago

Tested Asana provider, all seems to be well!

nlecoy commented 3 years ago

I've been using the fix of #15926 for work and everything looks fine! Also, I tested #15817 and its working too.

pavelhlushchanka commented 3 years ago

The package description says that the provider can be installed on top of existing airflow 2.*. I'm installing amazon provider and it requires airflow 2.1.

potiuk commented 3 years ago

The package description says that the provider can be installed on top of existing airflow 2.*. I'm installing amazon provider and it requires airflow 2.1.

Yep. We have an enty in changelog about it with warning. Do you think it's not enough ?

pavelhlushchanka commented 3 years ago

I tested #15173 and #15848. Works fine.

pavelhlushchanka commented 3 years ago

The package description says that the provider can be installed on top of existing airflow 2.*. I'm installing amazon provider and it requires airflow 2.1.

Yep. We have an enty in changelog about it with warning. Do you think it's not enough ?

@potiuk I believe it's a bit confusing. If such info is added, then it should be precise. But it's up to you.

potiuk commented 3 years ago

@potiuk I believe it's a bit confusing. If such info is added, then it should be precise. But it's up to you.

I will ask others what they think. I am inclined to release new rc for that together with elasticsearch (but let's continue testing).

amatellanes commented 3 years ago

I tested #15680 and it works fine :+1:

levyitay commented 3 years ago

tested Qubole changes, looks good

flolas commented 3 years ago

Tested #16280 in prod env with k8s, it works fine.

potiuk commented 3 years ago

@codenamestif (and others) - yeah it's too misleading indeed. I cancelled the vote and we have updated/fixed a few more things in the meantime. I will keep the test status for those providers that's been tested and will update the issue with the new changes later today when I release new RCs./

potiuk commented 3 years ago

This will be the new README. Note 2.1+ and apache-airflow<=2.1.0 dependency first on the list.

Package apache-airflow-providers-amazon
------------------------------------------------------

Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).

Release: 2.0.0

Provider package
----------------

This is a provider package for ``amazon`` provider. All classes for this provider package
are in ``airflow.providers.amazon`` python package.

Installation
------------

You can install this package on top of an existing airflow 2.1+ installation via
``pip install apache-airflow-providers-amazon``

PIP requirements
----------------

==================  ====================
PIP package         Version required
==================  ====================
``apache-airflow``  ``>=2.1.0``
``boto3``           ``>=1.15.0,<1.18.0``
``watchtower``      ``~=1.0.6``
==================  ====================
fredthomsen commented 3 years ago

Tested SimpleHttpOperator for #15605. All good.

potiuk commented 3 years ago

As new package RCs are released, I encourage those from the list who have not done so, to double-check if the packages are working fine with their changes. The voting time has been prolonged to Tue 22 Jun 21:29:59 CEST 2021.

Looking forward to your thumb-ups :)

TAKEDA-Takashi commented 3 years ago

Tested #16189. Works fine.

dejii commented 3 years ago

Tested #15541. Works as expected.

potiuk commented 3 years ago

Hey everyone. Just a kind ping here.

Due to update in READMEs, the voting on the packages have been prolonged to Tuesday so we have two more days for testing.

Thanks to @dejii @TAKEDA-Takashi @flolas @levyitay @codenamestif @amatellanes @nlecoy @jmelot @Goodkat @zhzhang for testing their changes (you rock!).

It would be great if we hear from others as well before the PMC gets to vote :). @MatthewRBruce @JavierLopezT @mmenarguezpear @monti-python @Junnplus @codenamestif @gabrielsyapse @uranusjr @Dr-Denzy @jpyen @jacobhjkim @olivermeyer @zachliu @thejens @p-kachalov @sunki-hong @sjvanrossum @subkanthi @avocadomaster @jlaneve @sunkickr @malthe @yesemsanthoshkumar @natanweinberger @jedcunningham @Visya @andrewgodwin @dimberman @msumit @grassten @pgillet @tianjianjiang @dstandish @Isaacwhyuenac @sfc-gh-madkins @ngaranko @ashb @tegardp

Isaacwhyuenac commented 3 years ago

Tested #15609. It works fine

MatthewRBruce commented 3 years ago

Tested #15490 - Fix unsuccessful KubernetesPod final_state call when is_delete_operator_pod=True. Works as expected.

olivermeyer commented 3 years ago

16327 works as expected.

monti-python commented 3 years ago

15637 works as expected as well

natanweinberger commented 3 years ago

15730 is just a whitespace linting fix 👍

However, #15728 and #15667 touched the same file around the same time (also for minor changes) and aren't listed here, should they be included?

potiuk commented 3 years ago

However, #15728 and #15667 touched the same file around the same time (also for minor changes) and aren't listed here, should they be included?

Auto-apply is specifically excluded as it was a change that applied to all providers (and main reason for this backwards-incompatible release) but it has been tested before. The other one is a docstring only (and those are excluded from changelog/testing).

jedcunningham commented 3 years ago

Tested elasticsearch 2.0.1rc1 (note: rc1, not 2 as linked in the description), looks good. Do note that some of the fixes/features only work with core changes coming in 2.1.1, but the new provider does still work with earlier 2.x versions.

potiuk commented 3 years ago

Tested elasticsearch 2.0.1rc1 (note: rc1, not 2 as linked in the description), looks good. Do note that some of the fixes/features only work with core changes coming in 2.1.1, but the new provider does still work with earlier 2.x versions.

Thanks @jedcunningham ! corrected the link to keep good track of it.

tianjianjiang commented 3 years ago

@potiuk I've tried to allocate some resource for testing #15026 but unfortunately not happening soon enough. However, at least the situation I've experienced before is coded in https://github.com/apache/airflow/pull/15026/files#diff-ba677556962f02c90a0e6933525d9d3f795e2d44a3d2229de1db9d1b68a067eb and since the datetime-related change is abstract while its upstream has only one change set of docstring lint (#15730), it is highly likely #15026 will work.

potiuk commented 3 years ago

Kind ping again. Last chance to do some testing before we release.

@JavierLopezT @mmenarguezpear @Junnplus @codenamestif @gabrielsyapse @uranusjr @Dr-Denzy @jpyen @jacobhjkim @zachliu @thejens @p-kachalov @sunki-hong @sjvanrossum @subkanthi @avocadomaster @jlaneve @sunkickr @malthe @yesemsanthoshkumar @Visya @andrewgodwin @dimberman @msumit @grassten @pgillet @dstandish @sfc-gh-madkins @ngaranko @ashb @tegardp

JavierLopezT commented 3 years ago

Hello. Regarding https://github.com/apache/airflow/pull/13796, I did make a quick test before merging the PR (the file was there and was not empty) but I can't make a deeper test due to health reasons. I am sorry

ngaranko commented 3 years ago

14701 works as expected.

potiuk commented 3 years ago

Hello. Regarding #13796, I did make a quick test before merging the PR (the file was there and was not empty) but I can't make a deeper test due to health reasons. I am sorry

No worries! thanks for your contributions so far @JavierLopezT !

potiuk commented 3 years ago

Callng for last vote so last chance to test before we release. @mmenarguezpear @Junnplus @codenamestif @gabrielsyapse @uranusjr @Dr-Denzy @jpyen @jacobhjkim @zachliu @thejens @p-kachalov @sunki-hong @sjvanrossum @subkanthi @avocadomaster @jlaneve @sunkickr @malthe @yesemsanthoshkumar @Visya @andrewgodwin @dimberman @msumit @grassten @pgillet @dstandish @sfc-gh-madkins @ashb @tegardp