Status of testing Providers that were prepared on June 20, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.2.0rc1

• [ ] Add deferrable option to EmrTerminateJobFlowOperator (#31646): @syedahsn
• [ ] Add Deferrable option to EmrCreateJobFlowOperator (#31641): @syedahsn
• [x] Add deferrable mode to BatchSensor (#30279): @phanikumv
• [ ] Add deferrable mode for S3KeySensor (#31018): @sunank200
• [ ] Add Deferrable mode to Emr Add Steps operator (#30928): @syedahsn
• [x] Add deferrable mode in Redshift delete cluster (#30244): @pankajastro
• [x] Add discoverability for triggers in provider.yaml (#31576): @eladkal
• [x] Add deferrable mode to AWS glue operators (Job & Crawl) (#30948): @vandonr-amz
• [x] Add deferrable param in BatchOperator (#30865): @pankajastro
• [ ] Add Deferrable Mode to RedshiftCreateClusterSnapshotOperator (#30856): @syedahsn
• [ ] Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator (#31657): @syedahsn
• [x] allow anonymous AWS access (#31659): @melugoyal
• [ ] Support of wildcard in S3ListOperator and S3ToGCSOperator (#31640): @moiseenkov
• [x] Add deferrable param in EmrContainerSensor (#30945): @pankajastro
• [x] Add realtime container execution logs for BatchOperator (#31837): @killua1zoldyck
• [x] Various fixes on ECS run task operator (#31838): @vandonr-amz
• [x] fix return values on glue operators deferrable mode (#31694): @vandonr-amz
• [ ] Add back missing AsyncIterator import (#31710): @uranusjr
• [x] Use a continuation token to get logs in ecs (#31824): @vandonr-amz
• [ ] Fetch status in while loop so as to not exit too early (#31804): @o-nikolas
• [x] [AWS hook] use provided client to get the official waiter on fallback (#31748): @vandonr-amz
• [x] fix(providers/amazon): handle missing LogUri in emr describe_cluster API response (#31482): @Lee-W
• [x] Add Python 3.11 support (#27264): @potiuk
• [ ] Added config template field to EmrServerlessStartJobOperator (#31746): @mahammi
• [ ] Add null check for host in Amazon Redshift connection (#31567): @sunank200
• [x] add workgroup to templated fields (#31574): @simonprydden
• [ ] Add docstring and signature for _read_remote_logs (#31623): @dstandish
• [x] Deprecate wait_for_completion from EcsRegisterTaskDefinitionOperator and EcsDeregisterTaskDefinitionOperator (#31884): @vandonr-amz
• [ ] Change Deferrable implementation for RedshiftResumeClusterOperator to follow standard (#30864): @syedahsn
• [ ] Change Deferrable implementation for RedshiftPauseClusterOperator to follow standard (#30853): @syedahsn

  Provider apache.hive: 6.1.1rc1

• [x] Sanitize beeline principal parameter (#31983): @potiuk
• [ ] Replace unicodecsv with standard csv library (#31693): @dstandish

  Provider apache.livy: 3.5.1rc1

• [x] Push Spark appId to XCOM for LivyOperator with deferrable mode (#31201): @pankajkoti
• [x] Optimize deferrable mode execution for LivyOperator (#31685): @phanikumv

  Provider apache.spark: 4.1.1rc1

• [x] Update SparkSubmitOperator doc: rename spark_conn_id to conn_id (#31952): @enchant3dmango

  Provider atlassian.jira: 2.1.1rc1

• [x] Fix: JiraOperator support any return response from Jira client (#31672): @Joffreybvn

  Provider cncf.kubernetes: 7.1.0rc1

• [x] KubernetesResourceOperator - KubernetesDeleteResourceOperator & KubernetesCreateResourceOperator (#29930): @raphaelauv
• [x] Add a return statement to enforce a single yield in certain triggers (#31985): @hussein-awala
• [ ] Add possibility to disable logging the pod template in a case when task fails (#31595): @svdimchenko
• [x] Remove return statement after yield from triggers class (#31703): @pankajastro
• [x] Fix Fargate logging for AWS system tests (#31622): @vincbeck

  Provider databricks: 4.3.0rc1

• [x] Add a return statement to enforce a single yield in certain triggers (#31985): @hussein-awala
• [ ] DatabricksSubmitRunOperator : Fix type annotation for libraries param (#31888): @phanikumv
• [x] Fix Databricks SQL operator serialization (#31780): @potiuk
• [ ] Making Databricks run related multi-query string in one session again (#31899): @dimonchik-suvorov
• [x] Remove return statement after yield from triggers class (#31703): @pankajastro

  Provider discord: 3.3.0rc1

• [x] Add DiscordNotifier (#31273): @vargacypher

  Provider google: 10.2.0rc1

• [x] Add a return statement to enforce a single yield in certain triggers (#31985): @hussein-awala
• [ ] Add deferrable mode to PubsubPullSensor (#31284): @bkossakowska
• [x] Add a new param to set parquet row group size in BaseSQLToGCSOperator (#31831): @hussein-awala
• [ ] Add cacheControl field to google cloud storage (#31338): @ying-w
• [ ] Add preserveAsciiControlCharacters to src_fmt_configs for GCStoBQ (#31643): @nathadfield
• [ ] Add credential configuration file support to Google Secrets Manager secrets backend (#31597): @pgagnon
• [ ] Add credential configuration file support to Google Cloud Hook (#31548): @pgagnon
• [x] Add deferrable mode to GCSUploadSessionCompleteSensor (#31081): @phanikumv
• [ ] Add append_job_name parameter in DataflowStartFlexTemplateOperator (#31511): @VladaZakharova
• [x] Introduce configuration to disable md5 used for security (#31171): @longslvr
• [ ] MetastoreHivePartitionSensor (#31016): @moiseenkov
• [ ] Bigquery: fix links for already existing tables and datasets. (#31589): @bkossakowska
• [ ] Provide missing project id and creds for TabularDataset (#31991): @hussein-awala
• [x] Optimize deferrable mode execution for DataprocSubmitJobOperator (#31317): @phanikumv
• [x] Optimize deferrable mode execution for BigQueryInsertJobOperator (#31249): @phanikumv
• [x] Remove return statement after yield from triggers class (#31703): @pankajastro
• [ ] Replace unicodecsv with standard csv library (#31693): @dstandish
• [x] Optimize deferrable mode execution for BigQueryCheckOperator (#31758): @phanikumv

  Provider hashicorp: 3.4.1rc1

• [x] Fix Hashicorp client authentication for userpass and aws_iam methods (#31593): @hussein-awala

  Provider jdbc: 4.0.0rc1

• [x] Restrict direct usage of driver params via extras for JDBC connection (#31849): @pankajkoti
• [x] Improve docstrings in providers (#31681): @uranusjr
• [x] Add D400 pydocstyle check - Providers (#31427): @vincbeck

  Provider microsoft.azure: 6.1.2rc1

• [ ] Replace unicodecsv with standard csv library (#31693): @dstandish
• [x] Removed unused variables in AzureBlobStorageToGCSOperator (#31765): @Adaverse
• [ ] Add docstring and signature for _read_remote_logs (#31623): @dstandish

  Provider microsoft.mssql: 3.4.1rc1

• [x] Check if sqlalchemy_scheme extra contains forbidden characters (#31984): @potiuk

  Provider mysql: 5.1.1rc1

• [ ] Replace unicodecsv with standard csv library (#31693): @dstandish

  Provider odbc: 4.0.0rc1

• [x] Disable setting ODBC driver via extra by default (#31713): @potiuk
• [x] Check if sqlalchemy_scheme extra contains forbidden characters (#31984): @potiuk
• [ ] Control permissibility of driver config in extra from airflow.cfg (#31754): @dstandish
• [ ] Improve docstrings in providers (#31681): @uranusjr
• [x] Add D400 pydocstyle check - Providers (#31427): @vincbeck

  Provider pagerduty: 3.3.0rc1

• [x] Add notifier for pagerduty (#31207): @utkarsharma2
• [ ] Add send_event method in PagerdutyEventsHook (#31290): @utkarsharma2

  Provider smtp: 1.2.0rc1

• [x] Add notifier for Smtp (#31359): @utkarsharma2

  Provider snowflake: 4.2.0rc1

• [x] Add SnowflakeSqlApiOperator operator (#30698): @utkarsharma2
• [ ] SnowflakeSqlApiOperator - Change the base class (#31751): @utkarsharma2
• [ ] Moved sql_api_generate_jwt out of hooks folder (#31586): @utkarsharma2

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs: @o-nikolas @utkarsharma2 @phanikumv @nathadfield @simonprydden @vargacypher @svdimchenko @longslvr @dstandish @uranusjr @dimonchik-suvorov @vincbeck @melugoyal @eladkal @Joffreybvn @sunank200 @moiseenkov @pankajastro @potiuk @killua1zoldyck @hussein-awala @raphaelauv @enchant3dmango @Lee-W @Adaverse @vandonr-amz @syedahsn @ying-w @bkossakowska @VladaZakharova @mahammi @pankajkoti @pgagnon

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[raphaelauv]

apache/airflow:slim-2.6.2-python3.11 with apache-airflow-providers-cncf-kubernetes==7.1.0rc1

on a kubernetes-sigs/kind

KubernetesResourceOperator - KubernetesDeleteResourceOperator & KubernetesCreateResourceOperator (#29930):

:ok_hand:

[pankajkoti]

Verified that my changes in https://github.com/apache/airflow/pull/31201 and https://github.com/apache/airflow/pull/31849 are included in the RCs and that they look good. Tested both with corresponding example DAGs and they yield expected results ✅

Thank you for including this! 🙂

[enchant3dmango]

apache-airflow-providers-apache-spark==4.1.1rc1. ✅

Thank you! 🖖🏼

[simonprydden]

Verified my change in #31574 is included in the RC and working.

Thank you.

[Adaverse]

Tested #31765 works as expected.

Can we include #31783 in this release @eladkal @phanikumv?

[Joffreybvn]

JiraOperator (#31672) works like a charm, thanks !

[vandonr-amz]

good for me 👍

[vargacypher]

Tested apache-airflow-providers-discord==3.3.0rc1 ✅

[killua1zoldyck]

31837 Works as expected

[longslvr]

https://github.com/apache/airflow/pull/31171 is good to go!

[vincbeck]

31622 works as expected. My other changes are style related

[Lee-W]

https://github.com/apache/airflow/pull/31482 works as expected and we have successfully tested other providers using our sample DAGs without any issues.

[eladkal]

Can we include #31783 in this release @eladkal @phanikumv?

This PR is not merged yet. Once it's merged it will be released in a the following wave

[phanikumv]

Tested #30279, #31685, #31081,#31317 , #31249, #31758 as part of below RC and they are working as expected.

apache-airflow-providers-amazon==8.2.0rc1 pip install apache-airflow-providers-apache-livy==3.5.1rc1 pip install apache-airflow-providers-google==10.2.0rc1

[ying-w]

please remove https://github.com/apache/airflow/pull/31338 it is not working as expected, I just pushed up a fix for it

[eladkal]

please remove https://github.com/apache/airflow/pull/31338 it is not working as expected, I just pushed up a fix for it

We can not remove. The choices are cancel RC1 and cut RC2 or release anyway. Assuming this is the only issue I prefer to release anyway since this is a new small enhancement which won't cause any user pain if it's not working as expected. I will cut a followup release for the google provider with the fix as soon as this wave is released.

[utkarsharma2]

Tested #31207, https://github.com/apache/airflow/pull/31207 #31359 https://github.com/apache/airflow/pull/31359 #30698 https://github.com/apache/airflow/pull/30698 and they are working as expected.

Thanks, Utkarsh Sharma

On Thu, Jun 22, 2023 at 11:31 AM eladkal @.***> wrote:

please remove #31338 https://github.com/apache/airflow/pull/31338 it is not working as expected, I just pushed up a fix for it

We can not remove. The choices are cancel RC1 and cut RC2 or release anyway. Assuming this is the only issue I prefer to release anyway since this is a new small enhancement which won't cause any user pain if it's not working as expected. I will cut a followup release for the google provider with the fix as soon as this wave is released.

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/32030#issuecomment-1602054263, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDLAHNDYBA6G5EMZJ56FO3XMPNT5ANCNFSM6AAAAAAZNI7N2Y . You are receiving this because you were mentioned.Message ID: @.***>

[pankajastro]

https://github.com/apache/airflow/pull/30244, https://github.com/apache/airflow/pull/30865 and https://github.com/apache/airflow/pull/30945 looks good. https://github.com/apache/airflow/pull/31703 We reverted in https://github.com/apache/airflow/pull/31985. So all good. thank you!

[hussein-awala]

I tested:

• https://github.com/apache/airflow/pull/31985
• https://github.com/apache/airflow/pull/31831
• https://github.com/apache/airflow/pull/31593

I cannot test https://github.com/apache/airflow/pull/31991 because I don't have a GCP setup, but it's present in the release and it should work according to the GCP documentation

[potiuk]

Checked if #27264 , #31983 , #31780, #31984 are included. All looks good.

[melugoyal]

validated #31659, looks good

[eladkal]

Thank you everyone. Providers are released I invite everyone to help improve providers for the next release, a list of open issues can be found here.