search

aio-libs / aiobotocore

asyncio support for botocore library using aiohttp
https://aiobotocore.aio-libs.org
Apache License 2.0
1.2k stars 183 forks source link

missing client field endpoint_ruleset_resolver in botocore 1.28.0 #976

Closed abivolmv closed 1 year ago

abivolmv commented 2 years ago

Describe the bug With botocore 1.28.0 (released 14h ago) aiobotocore 2.4.0 will fail when creating client because botocore 1.28.0 has a new required positional field endpoint_ruleset_resolver. Tested locally on Windows and on Lambda (linux py 3.8)

The same code we have works with botocore 1.27.95

  File "C:\Users\xyz\repos\xyz-api\venv2\lib\site-packages\aiobotocore\session.py", line 26, in __aenter__
    self._client = await self._coro
  File "C:\Users\xyz\repos\xyz-api\venv2\lib\site-packages\aiobotocore\session.py", line 193, in _create_client
    client = await client_creator.create_client(
  File "C:\Users\xyz\repos\xyz-api\venv2\lib\site-packages\aiobotocore\client.py", line 70, in create_client
    service_client = cls(**client_args)
TypeError: __init__() missing 1 required positional argument: 'endpoint_ruleset_resolver'

Checklist

pip freeze results

Environment:

Additional context aioboto3==10.1.0 aiobotocore==2.4.0 aiofiles==0.8.0 aiohttp==3.8.1 aioitertools==0.8.0 aiosignal==1.2.0 altgraph==0.17.2 ansicon==1.89.0 async-timeout==4.0.2 asyncmock==0.4.2 atomicwrites==1.4.0 attrs==21.2.0 aws-cdk.assets==1.137.0 aws-cdk.aws-acmpca==1.137.0 aws-cdk.aws-apigateway==1.137.0 aws-cdk.aws-applicationautoscaling==1.137.0 aws-cdk.aws-athena==1.137.0 aws-cdk.aws-autoscaling==1.137.0 aws-cdk.aws-autoscaling-common==1.137.0 aws-cdk.aws-autoscaling-hooktargets==1.137.0 aws-cdk.aws-certificatemanager==1.137.0 aws-cdk.aws-cloudformation==1.137.0 aws-cdk.aws-cloudfront==1.137.0 aws-cdk.aws-cloudfront-origins==1.137.0 aws-cdk.aws-cloudwatch==1.137.0 aws-cdk.aws-cloudwatch-actions==1.137.0 aws-cdk.aws-codebuild==1.137.0 aws-cdk.aws-codecommit==1.137.0 aws-cdk.aws-codeguruprofiler==1.137.0 aws-cdk.aws-codepipeline==1.137.0 aws-cdk.aws-codestarnotifications==1.137.0 aws-cdk.aws-cognito==1.137.0 aws-cdk.aws-dynamodb==1.137.0 aws-cdk.aws-ec2==1.137.0 aws-cdk.aws-ecr==1.137.0 aws-cdk.aws-ecr-assets==1.137.0 aws-cdk.aws-ecs==1.137.0 aws-cdk.aws-efs==1.137.0 aws-cdk.aws-eks==1.137.0 aws-cdk.aws-elasticloadbalancing==1.137.0 aws-cdk.aws-elasticloadbalancingv2==1.137.0 aws-cdk.aws-events==1.137.0 aws-cdk.aws-events-targets==1.137.0 aws-cdk.aws-globalaccelerator==1.137.0 aws-cdk.aws-glue==1.137.0 aws-cdk.aws-iam==1.137.0 aws-cdk.aws-kinesis==1.137.0 aws-cdk.aws-kinesisfirehose==1.137.0 aws-cdk.aws-kms==1.137.0 aws-cdk.aws-lambda==1.137.0 aws-cdk.aws-lambda-event-sources==1.137.0 aws-cdk.aws-logs==1.137.0 aws-cdk.aws-route53==1.137.0 aws-cdk.aws-route53-targets==1.137.0 aws-cdk.aws-s3==1.137.0 aws-cdk.aws-s3-assets==1.137.0 aws-cdk.aws-s3-notifications==1.137.0 aws-cdk.aws-sam==1.137.0 aws-cdk.aws-secretsmanager==1.137.0 aws-cdk.aws-servicediscovery==1.137.0 aws-cdk.aws-signer==1.137.0 aws-cdk.aws-sns==1.137.0 aws-cdk.aws-sns-subscriptions==1.137.0 aws-cdk.aws-sqs==1.137.0 aws-cdk.aws-ssm==1.137.0 aws-cdk.aws-stepfunctions==1.137.0 aws-cdk.aws-stepfunctions-tasks==1.137.0 aws-cdk.cloud-assembly-schema==1.137.0 aws-cdk.core==1.137.0 aws-cdk.custom-resources==1.137.0 aws-cdk.cx-api==1.137.0 aws-cdk.lambda-layer-awscli==1.137.0 aws-cdk.lambda-layer-kubectl==1.137.0 aws-cdk.lambda-layer-node-proxy-agent==1.137.0 aws-cdk.region-info==1.137.0 aws-xray-sdk==2.10.0 blessed==1.19.0 boto==2.49.0 boto3==1.25.0 botocore==1.28.0 cattrs==1.8.0 certifi==2021.10.8 chalice==1.26.3 chardet==3.0.4 charset-normalizer==2.0.9 click==8.0.3 colorama==0.4.4 constructs==3.3.168 coverage==5.5 demjson==2.2.4 dirtyjson==1.0.7 ecdsa==0.17.0 frozenlist==1.2.0 future==0.18.2 idna==2.10 iniconfig==1.1.1 inquirer==2.8.0 jinxed==1.1.0 jmespath==0.10.0 jsii==1.56.0 mock==4.0.3 multidict==5.2.0 mypy-extensions==0.4.3 packaging==21.3 pefile==2021.9.3 Pillow==8.2.0 pluggy==1.0.0 psutil==5.8.0 publication==0.0.3 py==1.11.0 pyasn1==0.4.8 pyinstaller==4.10 pyinstaller-hooks-contrib==2022.2 pyparsing==3.0.7 pytest==6.2.5 pytest-asyncio==0.17.2 pytest-cov==3.0.0 pytest-html==3.1.1 pytest-metadata==2.0.1 python-dateutil==2.8.2 python-editor==1.0.4 python-jose==3.3.0 pywin32-ctypes==0.2.0 PyYAML==6.0 qrcode==7.3.1 readchar==2.0.1 requests==2.25.1 rsa==4.8 s3transfer==0.6.0 six==1.16.0 toml==0.10.2 typing-extensions==3.10.0.2 unittest-xml-reporting==3.0.4 urllib3==1.25.11 watchtower==3.0.0 wcwidth==0.2.5 wrapt==1.13.3 yarl==1.7.2

leandrorebelo commented 2 years ago

I hope that solution will be to update the botocore to the newest version.

ragulpr commented 2 years ago

Thanks for reporting and thanks for all work team! I can verify the same issue with botocore 1.28.1 and aiobotocore 2.4.0

...
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 863, in _ls
    files = await self._lsdir(path, refresh)
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 631, in _lsdir
    await self.set_session()
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 473, in set_session
    self._s3 = await s3creator.__aenter__()
  File "/usr/local/lib/python3.8/dist-packages/aiobotocore/session.py", line 26, in __aenter__
    self._client = await self._coro
  File "/usr/local/lib/python3.8/dist-packages/aiobotocore/session.py", line 193, in _create_client
    client = await client_creator.create_client(
  File "/usr/local/lib/python3.8/dist-packages/aiobotocore/client.py", line 70, in create_client
    service_client = cls(**client_args)
TypeError: __init__() missing 1 required positional argument: 'endpoint_ruleset_resolver'
aiobotocore 2.4.0 requires botocore<1.27.60,>=1.27.59, but you'll have botocore 1.28.1 which is incompatible.
Successfully installed aiobotocore-2.4.0 aiohttp-3.8.3 aioitertools-0.11.0 aiosignal-1.2.0 async-timeout-4.0.2 attrs-22.1.0 botocore-1.28.1 
...

Sorry for not providing more details but that's all I can for now.

But it should be noted that aiobotocore 2.4.0 requires botocore<1.27.60 so this may not really be a aiobotocore issue (did not verify if fixing botocore version is possible/fixes problem). See also https://github.com/boto/botocore/pull/2796 for supposed fix for botocore<1.28

thehesiod commented 2 years ago

you cannot mix and match botocore and aiobotocore versions, that is why we have strict version requirements

abivolmv commented 2 years ago

you cannot mix and match botocore and aiobotocore versions, that is why we have strict version requirements

actually you can. If you shouldn't that's a different question ^^

We fixed this by hardfixing the botocore version. Until now we just had aioboto3==x.x.x and it seems botocore version 1.28.0 was picked up from some other package when doing pip install -r xxx.txt .

But it should be noted that aiobotocore 2.4.0 requires botocore<1.27.60

yes you are right. It is more an issue for future that needs to be taken care of. For the next version of aiobotocore.

thehesiod commented 2 years ago

well, if you obey rules ya cannot per setup.py requirements, if you want to be a rule breaker you have to deal with the consequences :) Each botocore bump is very time consuming and detailed to catch issues like these. In fact in the last attempt to do a minor patch bump it brought forth a new auth codepath that is going to require lots of new async plumbing :(

abivolmv commented 2 years ago

Ah, not the rule breaker by far ^^ We didn't specifically request for botocore upgrade but probably some package from above did it and overrode aiobotocore's requirements. As mentioned we did not have botocore nor aiobotocore in the requirements file until this popped up and we had to do it. I read on contributing guide about botocore version bump , it seems really a tedious task. Anywhere we could vote for botocore to implement aio ?

glarione commented 1 year ago

Absence of this feature from botocore means no use of aiobotocore with minio on-prem service. Any endpoint_url used in config will be overridden by old versioned botocore :crying_cat_face:

thehesiod commented 1 year ago

Ah, not the rule breaker by far ^^ We didn't specifically request for botocore upgrade but probably some package from above did it and overrode aiobotocore's requirements. As mentioned we did not have botocore nor aiobotocore in the requirements file until this popped up and we had to do it. I read on contributing guilde about botocore version bump , it seems really a tedious task. Anywhere we could vote for botocore to implement aio ?

this is why it's critical to use something like pipenv/poetry.

yep! see https://github.com/aio-libs/aiobotocore/issues/36 and https://github.com/boto/botocore/issues/458

thehesiod commented 1 year ago

another option is for this project to swap to the cpp lib...another huge task

thehesiod commented 1 year ago

is there anything else needed for this issue or just a version bump?

abivolmv commented 1 year ago

"Just" a version bump please.

this is why it's critical to use something like pipenv/poetry.

Tried Pipenv and it didn't work well for us. Though it was 2020 and last release then was in 2018 and there was even an issue asking if this project was dead. Now I see the releases are often and project is alive. Maybe it's worth an another try as the concept was nice (npm like). Poetry worked well in other projects I worked so it is in to-do for the project in case too.