search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.06k stars 14.29k forks source link

Status of testing Providers that were prepared on December 08, 2023 #36117

Closed eladkal closed 11 months ago

eladkal commented 11 months ago

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.13.0rc1

All users involved in the PRs: @Taragolis @vincbeck @romsharon98 @syedahsn @kacpermuda @raphaelauv @melugoyal @ginolegigot @atrbgithub @flolas @JDarDagran @nathadfield @potiuk @ephraimbuddy @Joffreybvn @pankajkoti @hussein-awala @VladaZakharova @stavdav143 @eladkal

Committer

melugoyal commented 11 months ago

35694 working as expected

atrbgithub commented 11 months ago

An issue has been raised with https://github.com/apache/airflow/pull/34919

Unfortunately it looks like it has broken GCSObjectsWithPrefixExistenceSensor.

Testing myself I can confirm that the following code is able to create the error:

from airflow.providers.google.cloud.hooks.gcs import GCSHook
res = GCSHook().list(
        bucket_name='a-testbucket',
        prefix='a/prefix/in/the/bucket/'
)
print(res)

Please do not include this change.

Edit: I've raised a PR https://github.com/apache/airflow/pull/36130 which appears to fix this issue for me when testing locally.

eladkal commented 11 months ago

Docker provider will be excluded from rc1

hussein-awala commented 11 months ago

I tested my change on the Kubernetes executor, and most of the changes in the Amazon provider, all work as expected.

pankajastro commented 11 months ago

As @atrbgithub has mentioned above https://github.com/apache/airflow/pull/34919 is breaking an existing DAG. so we might need to exclude Google provider

eladkal commented 11 months ago

so we might need to exclude Google provider

Yes. Google provider will also have RC2

potiuk commented 11 months ago

Tested mssql - all looks good.

One more thing I noticed - the removed daskexecutor has not been prepared. We have this in the release process, so it's easy to miss:

NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the provider), package for the provider will not be prepared when you prepare documentation for all providers - you have to specifically use the provider name in a separate command. For example to prepare documentation for removed.provider provider marked for removal you need to run separately this command:

breeze release-management prepare-provider-packages --package-format both removed.provider

But maybye we will change it in the future - we have more removed providers coming. So possibly we shoudl automatically add removed providers to the list of providers prepared (both for documentation and package building - so that we do not have to remember to prepare them. That was probably bad idea to not add them in the first place.

@eladkal -> I guess we should include rc2 (even if there was no rc1) for daskexecutor.

potiuk commented 11 months ago

One small update - as @pankajastro noticed in https://github.com/apache/airflow/pull/32319#discussion_r1421437350 - there is a case where we are not handling properly the fact that handler can return either list of rows or row - in both ODBC and DAtabricks oviders, so IMHO they should also be moved to RC2 (cc: @Joffreybvn )

potiuk commented 11 months ago

Fix to the process to automatically generate removed providers here: https://github.com/apache/airflow/pull/36160

Joffreybvn commented 11 months ago

Fix for https://github.com/apache/airflow/pull/32319#discussion_r1421437350 is here -> #36161

nathadfield commented 11 months ago

36072 has been tested and working as expected.

eladkal commented 11 months ago

Providers: Google, ODBC, Databricks, Docker are excluded from RC1.

eladkal commented 11 months ago

Thank you everyone. Providers are released (excluding Google, ODBC, Databricks, Docker which we will have RC2 soon) I invite everyone to help improve providers for the next release, a list of open issues can be found here.