eladkal commented 1 year ago

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.0.0rc2

[x] Remove deprecated "delegate_to" from GCP operators and hooks (#30748): @shahar1
[x] take advantage of upcoming major version release to remove deprecated things (#30755): @vandonr-amz
[x] add a stop operator to emr serverless (#30720): @vandonr-amz
[ ] SqlToS3Operator - Add feature to partition SQL table (#30460): @utkarsharma2
[x] New AWS sensor — DynamoDBValueSensor (#28338): @mrichman
[x] Add a "force" option to emr serverless stop/delete operator (#30757): @vandonr-amz
[ ] Add support for deferrable operators in AMPP (#30032): @syedahsn
[x] Fixed logging issue in DynamoDBValueSensor (#30703): @mrichman
[ ] DynamoDBHook - waiter_path() to consider resource_type or client_type (#30595): @utkarsharma2
[ ] Add ability to override waiter delay in EcsRunTaskOperator (#30586): @vincbeck
[x] Add support in AWS Batch Operator for multinode jobs (#29522): @vandonr-amz
[ ] AWS logs. Exit fast when 3 consecutive responses are returned from AWS Cloudwatch logs (#30756): @vincbeck
[ ] Remove @poke_mode_only from EmrStepSensor (#30774): @vincbeck
[x] Organize Amazon provider docs index (#30541): @eladkal
[ ] Remove duplicate param docstring in EksPodOperator (#30634): @jlaneve
[ ] Update AWS EMR Cluster Link to use the new dashboard (#30844): @dakshin-k

The guidelines on how to test providers can be found in

Verify providers by contributors

I have marked issues that were tested in RC1 as completed. All users involved in the PRs: @syedahsn @eladkal @utkarsharma2 @vincbeck @vandonr-amz @shahar1 @dakshin-k @mrichman @jlaneve

Committer

[X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

vandonr-amz commented 1 year ago

All good for me

pankajastro commented 1 year ago

Hi @eladkal / @syedahsn https://github.com/apache/airflow/pull/30032 is releasing in amazon: 8.0.0 where we introduced async_conn but I'm getting an error (check https://github.com/apache/airflow/pull/30868) when aws_session_token param is None. so wondering if we should include https://github.com/apache/airflow/pull/30868

mrichman commented 1 year ago

I just got this issue installing in MWAA using Airflow 2.5.1:


Collecting apache-airflow-providers-amazon==8.0.0rc2
--
Downloading apache_airflow_providers_amazon-8.0.0rc2-py3-none-any.whl (360 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 360.8/360.8 kB 27.7 MB/s eta 0:00:00
Requirement already satisfied: mypy-boto3-appflow>=1.24.0 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.26.32)
Requirement already satisfied: watchtower~=2.0.1 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (2.0.1)
Collecting asgiref
Downloading asgiref-3.6.0-py3-none-any.whl (23 kB)
Requirement already satisfied: sqlalchemy-redshift>=0.8.6 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (0.8.12)
Requirement already satisfied: apache-airflow-providers-common-sql>=1.3.1.dev0 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.3.3)
Requirement already satisfied: apache-airflow>=2.3.0.dev0 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (2.5.1)
Requirement already satisfied: jsonpath-ng>=1.5.3 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.5.3)
Collecting aiobotocore[boto3]>=2.2.0
Downloading aiobotocore-2.5.0-py3-none-any.whl (72 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.7/72.7 kB 15.3 MB/s eta 0:00:00
Requirement already satisfied: mypy-boto3-rds>=1.24.0 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.26.47)
Requirement already satisfied: boto3>=1.24.0 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.26.51)
Requirement already satisfied: mypy-boto3-redshift-data>=1.24.0 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.26.30)
Requirement already satisfied: redshift-connector>=2.0.888 in ./.local/lib/python3.10/site-packages (from apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (2.0.909)
Downloading aiobotocore-2.4.2-py3-none-any.whl (66 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.8/66.8 kB 14.7 MB/s eta 0:00:00
Collecting aiohttp>=3.3.1
Downloading aiohttp-3.8.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 56.6 MB/s eta 0:00:00
Requirement already satisfied: wrapt>=1.10.10 in ./.local/lib/python3.10/site-packages (from aiobotocore[boto3]>=2.2.0->apache-airflow-providers-amazon==8.0.0rc2->-r /usr/local/airflow/requirements/requirements.txt (line 3)) (1.14.1)
Collecting aioitertools>=0.5.1
Downloading aioitertools-0.11.0-py3-none-any.whl (23 kB)
Collecting aiobotocore[boto3]>=2.2.0
Downloading aiobotocore-2.4.1-py3-none-any.whl (66 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.8/66.8 kB 13.4 MB/s eta 0:00:00
Downloading aiobotocore-2.4.0-py3-none-any.whl (65 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 kB 12.4 MB/s eta 0:00:00
Downloading aiobotocore-2.3.4-py3-none-any.whl (64 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.7/64.7 kB 13.9 MB/s eta 0:00:00
Downloading aiobotocore-2.3.3.tar.gz (65 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.7/65.7 kB 11.2 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Downloading aiobotocore-2.3.2.tar.gz (104 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.8/104.8 kB 21.0 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Downloading aiobotocore-2.3.1.tar.gz (65 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.3/65.3 kB 11.7 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Downloading aiobotocore-2.3.0.tar.gz (65 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.1/65.1 kB 11.0 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Downloading aiobotocore-2.2.0.tar.gz (59 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.7/59.7 kB 9.4 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of apache-airflow-providers-amazon to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install aiobotocore[boto3]==2.2.0, aiobotocore[boto3]==2.3.0, aiobotocore[boto3]==2.3.1, aiobotocore[boto3]==2.3.2, aiobotocore[boto3]==2.3.3, aiobotocore[boto3]==2.3.4, aiobotocore[boto3]==2.4.0, aiobotocore[boto3]==2.4.1, aiobotocore[boto3]==2.4.2 and aiobotocore[boto3]==2.5.0 because these package versions have conflicting dependencies.
 
The conflict is caused by:
aiobotocore[boto3] 2.5.0 depends on botocore<1.29.77 and >=1.29.76
aiobotocore[boto3] 2.4.2 depends on botocore<1.27.60 and >=1.27.59
aiobotocore[boto3] 2.4.1 depends on botocore<1.27.60 and >=1.27.59
aiobotocore[boto3] 2.4.0 depends on botocore<1.27.60 and >=1.27.59
aiobotocore[boto3] 2.3.4 depends on botocore<1.24.22 and >=1.24.21
aiobotocore[boto3] 2.3.3 depends on botocore<1.24.22 and >=1.24.21
aiobotocore[boto3] 2.3.2 depends on botocore<1.24.22 and >=1.24.21
aiobotocore[boto3] 2.3.1 depends on botocore<1.24.22 and >=1.24.21
aiobotocore[boto3] 2.3.0 depends on botocore<1.24.22 and >=1.24.21
aiobotocore[boto3] 2.2.0 depends on botocore<1.24.22 and >=1.24.21
The user requested (constraint) botocore==1.29.51
 
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
 
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
 
[notice] A new release of pip is available: 23.0.1 -> 23.1.1
[notice] To update, run: pip install --upgrade pip

Here is my requirements.txt:

--constraint "/usr/local/airflow/dags/updated-constraints.txt"

apache-airflow-providers-amazon==8.0.0rc2

My constraints file is basically https://raw.githubusercontent.com/apache/airflow/constraints-2.5.1/constraints-3.10.txt with the one line for our provider changed:

# apache-airflow-providers-amazon==7.1.0
apache-airflow-providers-amazon==8.0.0rc2

And I save this as updated-constraints.txt which you can see in the requirements.txt.

pankajastro commented 1 year ago

yes, I feel it can use cause version conflict issues if we install with Airflow version less than 2.6.0 here the version looks correct https://raw.githubusercontent.com/apache/airflow/constraints-2.6.0rc1/constraints-3.10.txt

mrichman commented 1 year ago

I'll need to talk to the MWAA team about the timeline for 2.6 and possibly issue instructions that 8.0.0 will only work with 2.6+.

shahar1 commented 1 year ago

30748 works as expected

eladkal commented 1 year ago

possibly issue instructions that 8.0.0 will only work with 2.6+.

This is very drastic. Setting provider minimum version to 2.6 will means that all bug fixes and features would be denied from users of earlier versions. Consider many users who don't even use deferable operators...

Lets wait to hear from AWS Airflow team

potiuk commented 1 year ago

Fix is merged in https://github.com/apache/airflow/pull/30874 - this has been caused by turning the optional aiobotocore into required dependency in https://github.com/apache/airflow/pull/30032

eladkal commented 1 year ago

Closed in favor of RC3 issue tracking

apache / airflow

Status of testing Providers that were prepared on April 25, 2023 #30849

Provider amazon: 8.0.0rc2

Committer

30748 works as expected