Status of testing Providers that were prepared on March 03, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 7.3.0rc1

• [ ] add num rows affected to Redshift Data API hook (#29797): @yehoshuadimarsky
• [x] Add wait_for_completion param in RedshiftCreateClusterOperator (#29657): @pankajastro
• [ ] Add Amazon Redshift-data to S3<>RS Transfer Operators (#27947): @yehoshuadimarsky
• [ ] Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters (#29580): @vincbeck
• [x] Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity (#29623): @pgagnon
• [ ] Implement custom boto waiters for some EMR operators (#29822): @ferruzzi
• [x] fix code checking job names in sagemaker (#29245): @vandonr-amz
• [x] Avoid including fallback message for S3TaskHandler if streaming logs (#29708): @pdebelak
• [x] Use waiters in ECS Operators instead of inner sensors (#29761): @Taragolis
• [ ] Impovements for RedshiftDataOperator (#29434): @LMnet
• [x] Standardize AWS lambda naming (#29749): @eladkal
• [x] AWS Glue job hook: Make s3_bucket parameter optional (#29659): @romibuzi
• [x] RedshiftDataOperator replace await_result with wait_for_completion (#29633): @eladkal

  Provider apache.flink: 1.0.1rc1

• [x] Make cncf.kubernetes required for flink provider (#29710): @potiuk

  Provider apache.livy: 3.3.0rc1

• [ ] Add Livy Operator with deferrable mode (#29047): @sunank200

  Provider atlassian.jira: 2.0.1rc1

• [x] Handle jira_method_args in JiraOperator when not provided (#29741): @josh-fell

  Provider cncf.kubernetes: 5.2.2rc1

• [ ] KubernetesPodOperator._render_nested_template_fields improved by changing the conditionals for a map (#29760): @jose-lpa

  Provider common.sql: 1.3.4rc1

• [x] fix do_xcom_push=False bug in SnowflakeOperator (#29599): @fritz-astronomer

  Provider dbt.cloud: 3.1.0rc1

• [ ] Add DbtCloudJobRunAsyncSensor (#29695): @phanikumv

  Provider github: 2.2.1rc1

• [x] Handle github_method_args in GithubOperator when not provided (#29699): @josh-fell

  Provider google: 8.11.0rc1

• [ ] Add deferrable mode to BigQueryTablePartitionExistenceSensor. (#29735): @bkossakowska
• [x] Add a new param for BigQuery operators to support additional actions when resource exists (#29394): @hussein-awala
• [ ] Add deferrable mode to DataprocInstantiateWorkflowTemplateOperator (#28618): @bkossakowska
• [x] Dataproc batches (#29136): @kristopherkane
• [ ] Add CloudSQLCloneInstanceOperator (#29726): @FloMom
• [ ] GCSToGCSOperator: Fix 'NoneType' object is not subscriptable. (#29820): @wiktorn
• [x] Fix and augment check-for-inclusive-language CI check (#29549): @josh-fell
• [x] Don't push secret in XCOM in BigQueryCreateDataTransferOperator (#29348): @pankajastro
• [x] Google Cloud Providers - Introduce GoogleCloudBaseOperator (#29680): @IKholopov
• [ ] Update google cloud dlp package and adjust hook and operators (#29234): @lwyszomi
• [x] Refactor Dataproc Trigger (#29364): @pankajastro
• [ ] Remove <2.0.0 limit on google-cloud-bigtable (#29644): @r-richmond
• [x] Move help message to the google auth code (#29888): @potiuk

  Provider hashicorp: 3.3.0rc1

• [ ] Support multiple mount points in Vault backend secret (#29734): @hussein-awala

  Provider mysql: 4.0.2rc1

• [x] Use MariaDB client binaries in arm64 image for support MySQL backend (#29519): @Taragolis

  Provider sftp: 4.2.4rc1

• [x] Fix SFTPSensor when using newer_than and there are multiple matched files (#29794): @hussein-awala

  Provider ssh: 3.5.0rc1

• [x] SSH Provider: Add cmd_timeout to ssh connection extra (#29347): @Aakcht

  Provider telegram: 4.0.0rc1

• [x] Updated Telegram Provider to ensure compatbility with >=20.0.0 (#28953): @maxnathaniel

  Provider yandex: 3.3.0rc1

• [x] support Yandex SDK feature "endpoint" (#29635): @s0neq

The guidelines on how to test providers can be found in Verify providers by contributors

All users involved in the PRs: @kristopherkane @sunank200 @LMnet @phanikumv @jose-lpa @pdebelak @ferruzzi @vincbeck @r-richmond @vandonr-amz @eladkal @wiktorn @maxnathaniel @pgagnon @hussein-awala @bkossakowska @Taragolis @pankajastro @fritz-astronomer @romibuzi @josh-fell @lwyszomi @potiuk @s0neq @Aakcht @FloMom @IKholopov @yehoshuadimarsky

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[eladkal]

Contributors are also encouraged to cast non-binding vote in the mailing list thread.

[s0neq]

Provider yandex: 3.3.0rc1 has been tested. all is fine 👌

[vandonr-amz]

all good for me as it's covered by a system test that's green

[fritz-astronomer]

Snowflake change has unit test coverage 👍

[hussein-awala]

Provider google: 8.11.0rc1 : I tested #29394 and it works as expected. Provider sftp: 4.2.4rc1: Same for #29794. Provider hashicorp: 3.3.0rc1: I have tested #29734 and it works, but currently we should use the same variables, config and connections paths for all the mount points because there is a bug in the Vault secrets backend, I wonder if we can add #29908 to this release or we add it to the next one.

[eladkal]

@hussein-awala I will cancel RC1 for hashicorp provider and will cut RC2 after the current vote

[Aakcht]

Tested #29347 for SSH provider 3.5.0rc1 - works as expected. Waiting for full release!

[Taragolis]

• https://github.com/apache/airflow/pull/29519, checked if system have pre-installed client binaries for MySQL/MariaDB, than provider might be installed into the ARM. (note: Airflow on ARM is still experimental feature)
• https://github.com/apache/airflow/pull/29761: Test locally, work as expected, and no problem with aws_conn_id which is not set to default one "aws-default". cc: @dominikhei this change fix problem that you described in #29504, if you would like, you also could test this Amazon Provider package

[potiuk]

• Checked flink -> it has the right >= 5.1.0 requirement on cncf.kubernetes 👍
• Checked google -> the constants from auth library are properly inlined 👍

Looks good.

[pdebelak]

I checked apache-airflow-providers-amazon and the change in #29708 is working as expected.

[maxnathaniel]

Provider telegram: 4.0.0rc1 works as expected 👍

[pankajastro]

https://github.com/apache/airflow/pull/29657 https://github.com/apache/airflow/pull/29348 and https://github.com/apache/airflow/pull/29364 Looks good

[romibuzi]

29659 working as expected

[raphaelauv]

for the kubernetes the name of the feature is incorrect

KubernetesPodOperator._render_nested_template_fields improved by changing the conditionals for a map (#29760): @jose-lpa

->

KubernetesPodOperator._render_nested_template_fields improved by changing the conditionals (#29760): @jose-lpa

( there is no map )

[eladkal]

for the kubernetes the name of the feature is incorrect

KubernetesPodOperator._render_nested_template_fields improved by changing the conditionals for a map (#29760): @jose-lpa

->

KubernetesPodOperator._render_nested_template_fields improved by changing the conditionals (#29760): @jose-lpa

( there is no map )

The release mechanism currently does not allow us to change the commit messages. Probably something we need to improve on in the future. Since this is only issue with the commit title and not with the actual functionality I see no reason why we can't continue with release of this package.

[josh-fell]

29741, #29699, and #29549 look good.

Thanks for organizing @eladkal!

[pgagnon]

29623 working as expected.

[IKholopov]

29680 is effectively a noop (for now) and it is working as expected :)

[kristopherkane]

29136 Has been tested with real Google Cloud resources successfully.

[eladkal]

Thank you everyone. Providers are released I invite everyone to help improve providers for the next release, a list of open issues can be found here.

[LMnet]

29434 has a unit test and also I use the same code locally in a while.