thehesiod
commented
2 years ago this sounds like a bug in aiohttp them not aiobotocore?
Closed jmleoni closed 2 years ago
thehesiod
commented
2 years ago this sounds like a bug in aiohttp them not aiobotocore?
thehesiod
commented
2 years ago thanks for report btw, we were just moving to aiohttp 3.8.1
thehesiod
commented
2 years ago I can't repro with this simple testcase on an EC2 machine w/ IAM role based auth:
from aiobotocore.session import AioSession
import asyncio
loop = asyncio.get_event_loop()
session = AioSession()
client = loop.run_until_complete(session.create_client('s3').__aenter__())
loop.run_until_complete(client.list_objects(Bucket=BUCKET, Prefix=PREFIX))I even tried running multiple auth tasks in parallel and couldn't repro
blagojce95
commented
2 years ago I had the same error. Upgrading the aiohttp from 3.7.4 to 3.8.1 fixed the problem. Thanks for the tips!
tests/functional_tests/distributed/test_worker.py:43:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
distributed/worker.py:110: in create_product_simulations
res.fetch_elevation_raster(
flight_planning/raster_elevation.py:212: in fetch_elevation_raster
cache_hits = self._download_and_georeference_tiles(all_tiles, temp_dir_inside)
flight_planning/raster_elevation.py:274: in _download_and_georeference_tiles
cache_hits = asyncio.run(tasks_call())
/usr/lib/python3.8/asyncio/runners.py:44: in run
return loop.run_until_complete(main)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
flight_planning/raster_elevation.py:271: in tasks_call
return await asyncio.gather(*(fetch_file(tile, sema) for tile in tiles))
flight_planning/raster_elevation.py:240: in fetch_file
response = await client.get_object(Bucket=settings.aws_s3_mapbox_cache, Key=file_name)
../../.local/lib/python3.8/site-packages/aiobotocore/client.py:141: in _make_api_call
http, parsed_response = await self._make_request(
../../.local/lib/python3.8/site-packages/aiobotocore/client.py:161: in _make_request
return await self._endpoint.make_request(operation_model, request_dict)
../../.local/lib/python3.8/site-packages/aiobotocore/endpoint.py:77: in _send_request
request = await self.create_request(request_dict, operation_model)
../../.local/lib/python3.8/site-packages/aiobotocore/endpoint.py:70: in create_request
await self._event_emitter.emit(event_name, request=request,
../../.local/lib/python3.8/site-packages/aiobotocore/hooks.py:27: in _emit
response = await handler(**kwargs)
../../.local/lib/python3.8/site-packages/aiobotocore/signers.py:16: in handler
return await self.sign(operation_name, request)
../../.local/lib/python3.8/site-packages/aiobotocore/signers.py:63: in sign
auth.add_auth(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def add_auth(self, request):
if self.credentials is None:
> raise NoCredentialsError()
E botocore.exceptions.NoCredentialsError: Unable to locate credentials
../../.local/lib/python3.8/site-packages/botocore/auth.py:373: NoCredentialsErrorwait, this bug is stating the error happened when upgrading to 3.8.1, @blagojce95 you're saying the issue goes away after upgrading to 3.8.1?
blagojce95
commented
2 years ago I am getting strange behaviour.
botocore.exceptions.NoCredentialsError: Unable to locate credentials.botocore.exceptions.NoCredentialsError: Unable to locate credentials.Also we have botocore in the project, and it works fine. I am getting this error only from aiobotocore.
thehesiod
commented
2 years ago can you try with the latest release? there was a credentials refresh issue that was recently resolved
achimgaedke
commented
2 years ago I have tried it with the latest and tracked down the problem. For some reason the response for the instance metadata token at https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/utils.py#L102
imf = aiobotocore.credentials.AioInstanceMetadataFetcher(
**{'timeout': 1, 'num_attempts': 1, 'user_agent': 'Botocore/1.24.21 Python/3.10.4 Linux/5.4.0-1072-aws',
'config': {'ec2_metadata_service_endpoint': None, 'ec2_metadata_service_endpoint_mode': 'ipv4'}})
await imf._fetch_metadata_token()ends up being interpreted in a random charset, here: utf_16_be - which is not helpful for the subsequent call to fetch the credentials with it.
I'd suggest adding an encoding='utf-8' to any of the .text calls in the util module.
Some relevant version numbers:
aiobotocore 2.2.0
aiohttp 3.8.1
boto3 1.21.21
botocore 1.24.21Here we go - debugging https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client_reqrep.py#L993
ClientResponse.get_encoding gives unhelpful answers in specific cases:
import charset_normalizer as chardet
chardet.detect(b'AC9ktV7QSFrt5lWEWUWX_Vh3Y_5CAL35q1ejpEfjygKn==')returns {'encoding': 'ascii', 'language': 'English', 'confidence': 1.0} for 2.0.12
and {'encoding': 'utf_16_be', 'language': '', 'confidence': 1.0} for 2.0.0
I've manipulated the token, so it is safe to publish. But it also showed me that the result depends on the token. Heisenbug through and through...
achimgaedke
commented
2 years ago thehesiod
commented
2 years ago actually need a more fundamental fix after reviewing code in botocore, will put out a PR shortly
thehesiod
commented
2 years ago ok @achimgaedke started work here: https://github.com/aio-libs/aiobotocore/pull/934 looks like a bunch more new async piping needed :(
Since asynciohttp v3.8.1 was deployed on condaforge 3 days ago while trying to use s3fs with iamRole based credentials in AWS EC2 servers, we are encountering the following errors while opening files on s3 :
Fixing aiohttp in version v3.7.4.post0 solves this issue.
Checklist
pip checkpasses without errorspip freezeresultspip freeze results
Environment:
Additional context
I opened an issue with s3fs on this https://github.com/fsspec/s3fs/issues/558