eladkal commented 1 year ago

Body

Issue title: Status of testing Providers that were prepared on September 08, 2023

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 3.3.2rc1

[ ] respect soft_fail argument when exception is raised for airbyte sensors (#34156): @Lee-W
Provider alibaba: 2.5.3rc1
[ ] respect soft_fail argument when exception is raised for alibaba sensors (#34157): @Lee-W
Provider amazon: 8.7.0rc1
[x] Add Amazon SQS Notifier (#33962): @Taragolis
[x] Add Amazon SNS Notifier (#33828): @Taragolis
[ ] Increase waiter_max_attempts default value in EcsRunTaskOperator (#33712): @mjsqu
[x] Fix AWS EmrStepSensor ignoring the specified aws_conn_id in deferred mode (#33952): @yermalov-here
[x] Remove tasks annotation in AppflowHook (#33881): @Taragolis
[x] Make Amazon Chime connection lazy loaded and consistent with docs (#34000): @Taragolis
[ ] respect "soft_fail" argument when running BatchSensor in deferrable mode (#33405): @Lee-W
[x] Combine similar if logics in providers (#33987): @hussein-awala
[x] Replace single quotes by double quotes in tests (#33864): @hussein-awala
[x] Remove useless string join from providers (#33968): @hussein-awala
[x] Make aws.session_factory part of Amazon provider configuration documentation (#33960): @Taragolis
[x] Replace try - except pass by contextlib.suppress in providers (#33980): @hussein-awala
[x] Remove some useless try/except from providers code (#33967): @hussein-awala
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[x] Reorganize devel_only extra in airflow's setup.py (#33907): @potiuk
[x] Remove explicit str concat from Airflow providers package and tests (#33860): @hussein-awala
[x] Improve modules import in AWS probvider by move some of them into a type-checking block (#33780): @hussein-awala
[x] Always use Literal from typing_extensions (#33794): @Taragolis
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
[x] remove unnecessary map and rewrite it using list in providers (#33763): @hussein-awala
Provider apache.beam: 5.2.3rc1
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider apache.drill: 2.4.4rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider apache.flink: 1.1.3rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider apache.hdfs: 4.1.1rc1
[ ] Fix package name in exception message for hdfs provider (#33813): @pierrejeambrun
Provider apache.hive: 6.1.6rc1
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[x] Replace single element slice by next() in hive provider (#33937): @hussein-awala
[x] Use a single statement with multiple contexts instead of nested statements in providers (#33768): @hussein-awala
[x] Use startswith once with a tuple in Hive hook (#33765): @hussein-awala
[x] E731: replace lambda by a def method in Airflow providers (#33757): @hussein-awala
[x] Use f-string instead of format in Airflow providers (#33752): @hussein-awala
Provider apache.impala: 1.1.3rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider apache.livy: 3.5.4rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider apache.pig: 4.1.2rc1
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[x] Use a single statement with multiple contexts instead of nested statements in providers (#33768): @hussein-awala
Provider apache.pinot: 4.1.4rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider apprise: 1.0.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider celery: 3.3.4rc1
[ ] Fix bad "code cleanup" in celery executor (#34192): @dstandish
[x] Combine similar if logics in providers (#33987): @hussein-awala
[x] Limit celery by excluding 5.3.2 and 5.3.3 (#34031): @potiuk
[x] Replace try - except pass by contextlib.suppress in providers (#33980): @hussein-awala
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider cncf.kubernetes: 7.5.1rc1
[ ] respect soft_fail argument when exception is raised for spark-kubernetes sensors (#34167): @Lee-W
[x] Use cached_property for hook in SparkKubernetesSensor (#34106): @josh-fell
[x] Used cached property for hook in SparkKubernetesOperator (#34130): @vijay-jangir
[x] Combine similar if logics in providers (#33987): @hussein-awala
[x] Remove useless string join from providers (#33968): @hussein-awala
[x] replace loop by any when looking for a positive value in providers (#33984): @hussein-awala
[x] Move the try outside the loop when this is possible in kubernetes provider (#33977): @hussein-awala
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[x] Replace dict.items by dict.values when key is not used in Airflow providers (#33939): @hussein-awala
Provider common.sql: 1.7.2rc1
[ ] Fix BigQueryValueCheckOperator deferrable mode optimisation (#34018): @pankajkoti
Provider databricks: 4.5.0rc1
[ ] Add "QUEUED" to RUN_LIFE_CYCLE_STATES following deployement of … (#33886): @GeoffroyDFox
[ ] allow DatabricksSubmitRunOperator to accept a pipeline name for a pipeline_task (#32903): @adam133
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
[x] Use f-string instead of format in Airflow providers (#33752): @hussein-awala
Provider dbt.cloud: 3.3.0rc1
[ ] openlineage, dbt: add OpenLineage support for DBT Cloud (#33959): @JDarDagran
[ ] respect soft_fail argument when exception is raised for dbt sensors (#34164): @Lee-W
[x] Remove some useless try/except from providers code (#33967): @hussein-awala
[x] Use a single statement with multiple contexts instead of nested statements in providers (#33768): @hussein-awala
Provider docker: 3.7.5rc1
[ ] Cleanup Docker operator logging (#33914): @wolfdn
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
[x] Replace type func by isinstance in DockerOperator (#33759): @hussein-awala
Provider elasticsearch: 5.0.2rc1
[x] Make sure that only valid elasticsearch keys are passed to handler (#34119): @potiuk
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
[x] remove unnecessary map and rewrite it using list in providers (#33763): @hussein-awala
[x] Use f-string instead of format in Airflow providers (#33752): @hussein-awala
Provider exasol: 4.2.5rc1
[x] Use a single statement with multiple contexts instead of nested statements in providers (#33768): @hussein-awala
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
Provider facebook: 3.2.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider ftp: 3.5.2rc1
[ ] respect soft_fail argument when exception is raised for ftp sensors (#34161): @Lee-W
Provider google: 10.8.0rc1
[ ] Add deferrable mode to DataplexGetDataQualityScanResultOperator, DataplexRunDataQualityScanOperator (#33954): @bkossakowska
[x] allow impersonation_chain to be set on Google Cloud connection (#33715): @melugoyal
[ ] respect soft_fail argument when exception is raised for google-marketing-platform sensors (#34165): @Lee-W
[ ] fix: docstring in endpoint_service.py (#34135): @fabiogra
[ ] Fix BigQueryValueCheckOperator deferrable mode optimisation (#34018): @pankajkoti
[x] Fixup Datafusion system tests (#34068): @moiseenkov
[ ] Early delete a Dataproc cluster if started in the ERROR state. (#33668): @kristopherkane
[x] Avoid blocking event loop when using DataFusionAsyncHook by replacing sleep by asyncio.sleep (#33756): @hussein-awala
[x] Move the try outside the loop when this is possible in Google provider (#33976): @hussein-awala
[x] Combine similar if logics in providers (#33987): @hussein-awala
[x] Remove useless string join from providers (#33968): @hussein-awala
[ ] Update Azure fileshare hook to use azure-storage-file-share instead of azure-storage-file (#33904): @pankajastro
[x] replace loop by any when looking for a positive value in providers (#33984): @hussein-awala
[x] Replace try - except pass by contextlib.suppress in providers (#33980): @hussein-awala
[x] Remove some useless try/except from providers code (#33967): @hussein-awala
[x] Replace sequence concatination by unpacking in Airflow providers (#33933): @hussein-awala
[ ] Remove a deprecated option from BigQueryHook.get_pandas_df (#33819): @RyuSA
[x] replace unnecessary dict comprehension by dict() in providers (#33857): @hussein-awala
[x] Improve modules import in google provider by move some of them into a type-checking block (#33783): @hussein-awala
[x] Use a single statement with multiple contexts instead of nested statements in providers (#33768): @hussein-awala
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
[x] remove unnecessary map and rewrite it using list in providers (#33763): @hussein-awala
[x] E731: replace lambda by a def method in Airflow providers (#33757): @hussein-awala
[x] Use f-string instead of format in Airflow providers (#33752): @hussein-awala
Provider hashicorp: 3.4.3rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider http: 4.5.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
[ ] Convert hard-coded allowlist error code to be argument of HttpSensor (#33717): @okayhooni
Provider imap: 3.3.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider influxdb: 2.2.3rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider jdbc: 4.0.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider jenkins: 3.3.2rc1
[x] Move the try outside the loop when this is possible in jenkins provider (#33979): @hussein-awala
Provider microsoft.azure: 7.0.0rc1
[ ] Update Azure fileshare hook to use azure-storage-file-share instead of azure-storage-file (#33904): @pankajastro
[x] Remove AzureDataFactoryPipelineRunStatusAsyncSensor class (#34036): @eladkal
[x] Remove LocalToAzureDataLakeStorageOperator class (#34035): @eladkal
[ ] feat(providers/microsoft): add AzureContainerInstancesOperator.volume as a template field (#34070): @Lee-W
[ ] Add DefaultAzureCredential support to AzureContainerRegistryHook (#33825): @Lee-W
[ ] feat(providers/microsoft): add DefaultAzureCredential support to AzureContainerVolumeHook (#33822): @Lee-W
[ ] docs(providers/microsoft): improve documentation for AzureBatchHook DefaultAzureCredential support (#34098): @Lee-W
[ ] Remove azure-storage-common from microsoft azure providers (#34038): @pankajastro
[x] Remove useless string join from providers (#33968): @hussein-awala
Provider microsoft.psrp: 2.3.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider mongo: 3.2.2rc1
[x] Always use Literal from typing_extensions (#33794): @Taragolis
Provider mysql: 5.3.1rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
[x] Use a single statement with multiple contexts instead of nested statements in providers (#33768): @hussein-awala
Provider neo4j: 3.3.3rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider openlineage: 1.1.0rc1
[ ] Allow to disable openlineage at operator level (#33685): @RNHTTR
[ ] openlineage: fix import in get_custom_facets (#34122): @JDarDagran
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
[ ] openlineage, dbt: add OpenLineage support for DBT Cloud (#33959): @JDarDagran
Provider oracle: 3.7.4rc1
[x] Combine similar if logics in providers (#33987): @hussein-awala
[x] remove unnecessary map and rewrite it using list in providers (#33763): @hussein-awala
Provider pagerduty: 3.3.1rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider postgres: 5.6.1rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
[x] Use a single statement with multiple contexts instead of nested statements in providers (#33768): @hussein-awala
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
Provider presto: 5.1.4rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider redis: 3.3.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
[x] Limit Redis dependencies to match celery limits (#33773): @potiuk
Provider salesforce: 5.4.3rc1
[x] Always use Literal from typing_extensions (#33794): @Taragolis
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider samba: 4.2.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider sftp: 4.6.1rc1
[ ] respect soft_fail argument when exception is raised for sftp sensors (#34169): @Lee-W
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider slack: 8.1.0rc1
[x] Add Slack Incoming Webhook Notifier (#33966): @Taragolis
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider smtp: 1.3.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider snowflake: 5.0.1rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
[x] Use a single statement with multiple contexts instead of nested statements in providers (#33768): @hussein-awala
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
Provider ssh: 3.7.3rc1
[x] Use literal dict instead of calling dict() in providers (#33761): @hussein-awala
[x] E731: replace lambda by a def method in Airflow providers (#33757): @hussein-awala
Provider tableau: 4.2.2rc1
[ ] respect soft_fail argument when exception is raised for tableau sensors (#34163): @Lee-W
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider trino: 5.3.1rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala
Provider vertica: 3.5.2rc1
[x] Custom fetch all handler for vertica to not miss errors (#34041): @darkag
Provider zendesk: 4.3.2rc1
[x] Improve modules import in Airflow providers by some of them into a type-checking block (#33754): @hussein-awala

All users involved in the PRs: @Taragolis @fabiogra @vijay-jangir @darkag @okayhooni @moiseenkov @RyuSA @potiuk @yermalov-here @eladkal @bkossakowska @hussein-awala @pankajastro @adam133 @kristopherkane @pankajkoti @melugoyal @pierrejeambrun @JDarDagran @GeoffroyDFox @RNHTTR @mjsqu @dstandish @josh-fell @wolfdn @Lee-W

Committer

[X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

yermalov-here commented 1 year ago

33952 works as expected

Taragolis commented 1 year ago

https://github.com/apache/airflow/pull/33962: works fine
https://github.com/apache/airflow/pull/33828: works fine
https://github.com/apache/airflow/pull/33966: works fine
https://github.com/apache/airflow/pull/33960: docs only
https://github.com/apache/airflow/pull/33794: no side effect after the changes
https://github.com/apache/airflow/pull/33881: ci related/statics checks
https://github.com/apache/airflow/pull/34000: UI fields works fine. Hook doesn't produce requests to DB during initilization (I've put it in top of the dag with invalid connection id)

melugoyal commented 1 year ago

33715 works as expected

JDarDagran commented 1 year ago

34122 works fine

33959 does not work properly, for `DbtCloudRunJobOperator` it raises:

[2023-09-08, 21:36:05 UTC] {base.py:152} ERROR - OpenLineage provider method failed to extract data from provider. 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/openlineage/extractors/base.py", line 137, in _get_openlineage_facets
    facets: OperatorLineage = get_facets_method(*args)
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/dbt/cloud/operators/dbt.py", line 229, in get_openlineage_facets_on_complete
    return generate_openlineage_events_from_dbt_cloud_run(operator=self, task_instance=task_instance)
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/dbt/cloud/utils/openlineage.py", line 60, in generate_openlineage_events_from_dbt_cloud_run
    run_id=operator.run_id, account_id=operator.account_id, include_related=["run_steps,job"]
AttributeError: 'DbtCloudRunJobOperator' object has no attribute 'run_id'

I'll raise an issue and try to solve this.

RNHTTR commented 1 year ago

33685 doesn't work for `@task` decorated operators, but I don't think that's a big deal since I don't think these kinds of tasks by themselves are that useful for lineage data, so I think this one is fine. @mobuchowski what do you think?

mobuchowski commented 1 year ago

@RNHTTR what if you disable _PythonDecoratedOperator?

hussein-awala commented 1 year ago

I checked all my 101 changes, and they are all present in the RC.

Lee-W commented 1 year ago

Tested https://github.com/apache/airflow/pull/33825, https://github.com/apache/airflow/pull/33822, https://github.com/apache/airflow/pull/34098

pankajkoti commented 1 year ago

Tested my change https://github.com/apache/airflow/pull/34018 in the Google RC 10.8.0rc1. It works fine, but, it also has a dependency on common-sql provider 1.7.2.rc1 for the change in same PR. If the common-sql provider is not updated then it fails. How do we handle cross-provider dependency bumps during releases? Does it get handled automatically or we need a manual minimum dependency bump here in Google RC to contain common-sql>=1.7.2?

hussein-awala commented 1 year ago

it also has a dependency on common-sql provider 1.7.2.rc1 for the change in same PR. If the common-sql provider is not updated then it fails.

IMHO if the operator will be broken before upgrading the common-sql version to latest, then we should consider it as breaking change and fix it. The min version of common-sql is 1.3.1 in google provider, bumping it to 1.7.2 could fix the issue, and I think that it's safe as we still use the same major version.

potiuk commented 1 year ago

Yep. We should bump min version of common-sql

pankajkoti commented 1 year ago

Thank you @hussein-awala and @potiuk for your quick inputs and suggestions. I have created a PR now to bump the min version https://github.com/apache/airflow/pull/34257.

cc: @eladkal Sorry din't realise this earlier. What would be the steps for releasing the Google RC now as it may depend on PR https://github.com/apache/airflow/pull/34257?

potiuk commented 1 year ago

Checked all my changes are in (mostly dependencies). All looks good.

eladkal commented 1 year ago

dbt.cloud will be excluded from this wave due to issues found. please keep testing rest of the providers

mobuchowski commented 1 year ago

https://github.com/apache/airflow/pull/33959 bug does not affect OpenLineage provider, just dbt one - we should only exclude this. I will doublecheck OL provider.

EDIT: https://github.com/apache/airflow/pull/34270 should fix this issue, maybe release from RC2?

moiseenkov commented 1 year ago

34068 works fine

darkag commented 1 year ago

34041 works as expected

josh-fell commented 1 year ago

34106 and #34130 look good!

eladkal commented 1 year ago

33959 bug does not affect OpenLineage provider, just dbt one - we should only exclude this. I will doublecheck OL provider.

EDIT: #34270 should fix this issue, maybe release from RC2?

I will cut rc2 for dbt.cloud

eladkal commented 1 year ago

Thank you everyone. dbt.cloud is excluded from this release. The rest of providers are released I invite everyone to help improve providers for the next release, a list of open issues can be found here.

apache / airflow

Status of testing Providers that were prepared on September 08, 2023 #34218

Body

Provider airbyte: 3.3.2rc1

Provider alibaba: 2.5.3rc1

Provider amazon: 8.7.0rc1

Provider apache.beam: 5.2.3rc1

Provider apache.drill: 2.4.4rc1

Provider apache.flink: 1.1.3rc1

Provider apache.hdfs: 4.1.1rc1

Provider apache.hive: 6.1.6rc1

Provider apache.impala: 1.1.3rc1

Provider apache.livy: 3.5.4rc1

Provider apache.pig: 4.1.2rc1

Provider apache.pinot: 4.1.4rc1

Provider apprise: 1.0.2rc1

Provider celery: 3.3.4rc1

Provider cncf.kubernetes: 7.5.1rc1

Provider common.sql: 1.7.2rc1

Provider databricks: 4.5.0rc1

Provider dbt.cloud: 3.3.0rc1

Provider docker: 3.7.5rc1

Provider elasticsearch: 5.0.2rc1

Provider exasol: 4.2.5rc1

Provider facebook: 3.2.2rc1

Provider ftp: 3.5.2rc1

Provider google: 10.8.0rc1

Provider hashicorp: 3.4.3rc1

Provider http: 4.5.2rc1

Provider imap: 3.3.2rc1

Provider influxdb: 2.2.3rc1

Provider jdbc: 4.0.2rc1

Provider jenkins: 3.3.2rc1

Provider microsoft.azure: 7.0.0rc1

Provider microsoft.psrp: 2.3.2rc1

Provider mongo: 3.2.2rc1

Provider mysql: 5.3.1rc1

Provider neo4j: 3.3.3rc1

Provider openlineage: 1.1.0rc1

Provider oracle: 3.7.4rc1

Provider pagerduty: 3.3.1rc1

Provider postgres: 5.6.1rc1

Provider presto: 5.1.4rc1

Provider redis: 3.3.2rc1

Provider salesforce: 5.4.3rc1

Provider samba: 4.2.2rc1

Provider sftp: 4.6.1rc1

Provider slack: 8.1.0rc1

Provider smtp: 1.3.2rc1

Provider snowflake: 5.0.1rc1

Provider ssh: 3.7.3rc1

Provider tableau: 4.2.2rc1

Provider trino: 5.3.1rc1

Provider vertica: 3.5.2rc1

Provider zendesk: 4.3.2rc1

Committer

33952 works as expected

33715 works as expected

34122 works fine

33959 does not work properly, for DbtCloudRunJobOperator it raises:

33685 doesn't work for @task decorated operators, but I don't think that's a big deal since I don't think these kinds of tasks by themselves are that useful for lineage data, so I think this one is fine. @mobuchowski what do you think?

34068 works fine

34041 works as expected

34106 and #34130 look good!

33959 bug does not affect OpenLineage provider, just dbt one - we should only exclude this. I will doublecheck OL provider.

33959 does not work properly, for `DbtCloudRunJobOperator` it raises:

33685 doesn't work for `@task` decorated operators, but I don't think that's a big deal since I don't think these kinds of tasks by themselves are that useful for lineage data, so I think this one is fine. @mobuchowski what do you think?