AttributeError: __aenter__ on 2.5.3

rdbisme commented 1 year ago

Describe the bug 2.5.3 version breaks some code we have with AttributeError: __aenter__. Reverting to 2.5.2 makes the error to vanish. The interesting bit is that we can reproduce this only on docker. Running the same code locally doesn't break.

Checklist

[x] I have reproduced in environment where pip check passes without errors
[x] I have provided pip freeze results
[x] I have provided sample code or detailed way to reproduce
[ ] I have tried the same code in botocore to ensure this is an aiobotocore specific issue
[ ] I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
[x] I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.5.3
aiohttp==3.8.5
aioitertools==0.11.0
aiosignal==1.3.1
async-timeout==4.0.2
attrs==23.1.0
behave==1.2.6
behave-pandas==0.5.0
botocore==1.31.17
Bottleneck==1.3.7
cachetools==5.3.1
certifi==2023.7.22
cffi==1.15.1
charset-normalizer==3.2.0
codetiming==1.4.0
contexttimer==0.3.3
cryptography==38.0.4
decorator==5.1.1
deprecation==2.1.0
docstring-parser==0.15
exchange-calendars==4.2.8
frozenlist==1.4.0
fsspec==2023.6.0
idna==3.4
janus==1.0.0
jmespath==1.0.1
json-logging==1.2.11
korean-lunar-calendar==0.3.1
methodtools==0.4.7
mmh3==3.1.0
msgpack==1.0.5
multidict==6.0.4
mypy-extensions==1.0.0
networkx==2.8.8
numpy==1.25.2
packaging==23.1
pandas==2.0.3
parse==1.19.1
parse-type==0.6.2
pause==0.3
plotly==5.15.0
polars==0.16.18
psutil==5.9.5
py==1.11.0
pyarrow==12.0.1
pycparser==2.21
pydantic==1.10.12
pyluach==2.2.0
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0.1
requests==2.31.0
retry==0.9.2
s3fs==2023.6.0
six==1.16.0
tabulate==0.9.0
tenacity==8.2.2
threadpoolctl==3.2.0
toml==0.10.2
toolz==0.12.0
tqdm==4.65.0
typed-argument-parser==1.8.1
typing-inspect==0.9.0
typing_extensions==4.7.1
tzdata==2023.3
universal-pathlib==0.1.0
urllib3==1.26.16
uvloop==0.17.0
websockets==11.0.3
wirerope==0.4.7
wrapt==1.15.0
yarl==1.9.2

Environment:

Python Version: 3.10
OS name and version: Linux b6fd2ef3b383 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Thu Jan 12 16:32:13 UTC 2023 x86_64 GNU/Linux (docker)

Additional context Traceback

  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/xxx/common.py", line 216, in load_json
    with file as f:
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/core.py", line 102, in __enter__
    f = self.fs.open(self.path, mode=mode)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/spec.py", line 1241, in open
    f = self._open(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 659, in _open
    return S3File(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 2066, in __init__
    super().__init__(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/spec.py", line 1597, in __init__
    self.size = self.details["size"]
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/spec.py", line 1610, in details
    self._details = self.fs.info(self.path)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/asyn.py", line 121, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/asyn.py", line 106, in sync
    raise return_result
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/asyn.py", line 61, in _runner
    result[0] = await coro
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 1271, in _info
    out = await self._call_s3(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 348, in _call_s3
    return await _error_wrapper(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 140, in _error_wrapper
    raise err
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
    return await func(*args, **kwargs)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/client.py", line 366, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/client.py", line 391, in _make_request
    return await self._endpoint.make_request(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 100, in _send_request
    while await self._needs_retry(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 262, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/hooks.py", line 66, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 15, in resolve_awaitable
    return await obj
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/utils.py", line 414, in redirect_from_error
    new_region = await self.get_bucket_region(bucket, response)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/utils.py", line 483, in get_bucket_region
    async with self._client as client:
AttributeError: __aenter__

rdbisme commented 1 year ago

self._client is a weakproxy.

-> async with self._client as client:
(Pdb) p self._client
<weakproxy at 0x7084ddca9530 to S3 at 0x7084de125690>
(Pdb) dir(self._client)
['_PY_TO_OP_NAME', '__aenter__', '__aexit__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_async_getattr', '_cache', '_client_config', '_convert_to_request_dict', '_emit_api_params', '_endpoint', '_exceptions', '_exceptions_factory', '_get_waiter_config', '_load_exceptions', '_loader', '_make_api_call', '_make_request', '_register_handlers', '_request_signer', '_resolve_endpoint_ruleset', '_response_parser', '_ruleset_resolver', '_serializer', '_service_model', '_user_agent_creator', 'abort_multipart_upload', 'can_paginate', 'close', 'complete_multipart_upload', 'copy_object', 'create_bucket', 'create_multipart_upload', 'delete_bucket', 'delete_bucket_analytics_configuration', 'delete_bucket_cors', 'delete_bucket_encryption', 'delete_bucket_intelligent_tiering_configuration', 'delete_bucket_inventory_configuration', 'delete_bucket_lifecycle', 'delete_bucket_metrics_configuration', 'delete_bucket_ownership_controls', 'delete_bucket_policy', 'delete_bucket_replication', 'delete_bucket_tagging', 'delete_bucket_website', 'delete_object', 'delete_object_tagging', 'delete_objects', 'delete_public_access_block', 'exceptions', 'generate_presigned_post', 'generate_presigned_url', 'get_bucket_accelerate_configuration', 'get_bucket_acl', 'get_bucket_analytics_configuration', 'get_bucket_cors', 'get_bucket_encryption', 'get_bucket_intelligent_tiering_configuration', 'get_bucket_inventory_configuration', 'get_bucket_lifecycle', 'get_bucket_lifecycle_configuration', 'get_bucket_location', 'get_bucket_logging', 'get_bucket_metrics_configuration', 'get_bucket_notification', 'get_bucket_notification_configuration', 'get_bucket_ownership_controls', 'get_bucket_policy', 'get_bucket_policy_status', 'get_bucket_replication', 'get_bucket_request_payment', 'get_bucket_tagging', 'get_bucket_versioning', 'get_bucket_website', 'get_object', 'get_object_acl', 'get_object_attributes', 'get_object_legal_hold', 'get_object_lock_configuration', 'get_object_retention', 'get_object_tagging', 'get_object_torrent', 'get_paginator', 'get_public_access_block', 'get_waiter', 'head_bucket', 'head_object', 'list_bucket_analytics_configurations', 'list_bucket_intelligent_tiering_configurations', 'list_bucket_inventory_configurations', 'list_bucket_metrics_configurations', 'list_buckets', 'list_multipart_uploads', 'list_object_versions', 'list_objects', 'list_objects_v2', 'list_parts', 'meta', 'put_bucket_accelerate_configuration', 'put_bucket_acl', 'put_bucket_analytics_configuration', 'put_bucket_cors', 'put_bucket_encryption', 'put_bucket_intelligent_tiering_configuration', 'put_bucket_inventory_configuration', 'put_bucket_lifecycle', 'put_bucket_lifecycle_configuration', 'put_bucket_logging', 'put_bucket_metrics_configuration', 'put_bucket_notification', 'put_bucket_notification_configuration', 'put_bucket_ownership_controls', 'put_bucket_policy', 'put_bucket_replication', 'put_bucket_request_payment', 'put_bucket_tagging', 'put_bucket_versioning', 'put_bucket_website', 'put_object', 'put_object_acl', 'put_object_legal_hold', 'put_object_lock_configuration', 'put_object_retention', 'put_object_tagging', 'put_public_access_block', 'restore_object', 'select_object_content', 'upload_part', 'upload_part_copy', 'waiter_names', 'write_get_object_response']

__aenter__ is well there so I don't understand what's happening here.

rdbisme commented 1 year ago

I can reproduce like this:

import fsspec

_f = fsspec.open("s3://<some file on s3>")

with _f:
    print("a")

Traceback (most recent call last):
  File "/tmp/repro.py", line 5, in <module>
    with _f:
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/core.py", line 102, in __enter__
    f = self.fs.open(self.path, mode=mode)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/spec.py", line 1241, in open
    f = self._open(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 659, in _open
    return S3File(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 2066, in __init__
    super().__init__(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/spec.py", line 1597, in __init__
    self.size = self.details["size"]
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/spec.py", line 1610, in details
    self._details = self.fs.info(self.path)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/asyn.py", line 121, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/asyn.py", line 106, in sync
    raise return_result
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/fsspec/asyn.py", line 61, in _runner
    result[0] = await coro
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 1271, in _info
    out = await self._call_s3(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 348, in _call_s3
    return await _error_wrapper(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 140, in _error_wrapper
    raise err
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
    return await func(*args, **kwargs)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/client.py", line 366, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/client.py", line 391, in _make_request
    return await self._endpoint.make_request(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 100, in _send_request
    while await self._needs_retry(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 262, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/hooks.py", line 66, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 15, in resolve_awaitable
    return await obj
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/utils.py", line 414, in redirect_from_error
    new_region = await self.get_bucket_region(bucket, response)
  File "/root/.local/pipx/venvs/xxx/lib/python3.10/site-packages/aiobotocore/utils.py", line 483, in get_bucket_region
    async with self._client as client:
AttributeError: __aenter__. Did you mean: '_details'?

shiv-planetrics commented 1 year ago

pd.read_parquet throws similar error with s3fs==2023.6.0 and aiobotocore==2.5.3. It works fine when the aiobotocore version is pinned to 2.5.2.

/venv/lib/python3.11/site-packages/pandas/io/parquet.py:509: in read_parquet
    return impl.read(
/venv/lib/python3.11/site-packages/pandas/io/parquet.py:227: in read
    pa_table = self.api.parquet.read_table(
/venv/lib/python3.11/site-packages/pyarrow/parquet/core.py:2939: in read_table
    dataset = _ParquetDatasetV2(
/venv/lib/python3.11/site-packages/pyarrow/parquet/core.py:2465: in __init__
    finfo = filesystem.get_file_info(path_or_paths)
pyarrow/_fs.pyx:571: in pyarrow._fs.FileSystem.get_file_info
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
pyarrow/_fs.pyx:1490: in pyarrow._fs._cb_get_file_info
    ???
/venv/lib/python3.11/site-packages/pyarrow/fs.py:332: in get_file_info
    info = self.fs.info(path)
/venv/lib/python3.11/site-packages/fsspec/asyn.py:121: in wrapper
    return sync(self.loop, func, *args, **kwargs)
/venv/lib/python3.11/site-packages/fsspec/asyn.py:106: in sync
    raise return_result
/venv/lib/python3.11/site-packages/fsspec/asyn.py:61: in _runner
    result[0] = await coro
/venv/lib/python3.11/site-packages/s3fs/core.py:1271: in _info
    out = await self._call_s3(
/venv/lib/python3.11/site-packages/s3fs/core.py:348: in _call_s3
    return await _error_wrapper(
/venv/lib/python3.11/site-packages/s3fs/core.py:140: in _error_wrapper
    raise err
/venv/lib/python3.11/site-packages/s3fs/core.py:113: in _error_wrapper
    return await func(*args, **kwargs)
/venv/lib/python3.11/site-packages/aiobotocore/client.py:366: in _make_api_call
    http, parsed_response = await self._make_request(
/venv/lib/python3.11/site-packages/aiobotocore/client.py:391: in _make_request
    return await self._endpoint.make_request(
/venv/lib/python3.11/site-packages/aiobotocore/endpoint.py:100: in _send_request
    while await self._needs_retry(
/venv/lib/python3.11/site-packages/aiobotocore/endpoint.py:262: in _needs_retry
    responses = await self._event_emitter.emit(
/venv/lib/python3.11/site-packages/aiobotocore/hooks.py:66: in _emit
    response = await resolve_awaitable(handler(**kwargs))
/venv/lib/python3.11/site-packages/aiobotocore/_helpers.py:15: in resolve_awaitable
    return await obj
/venv/lib/python3.11/site-packages/aiobotocore/utils.py:414: in redirect_from_error
    new_region = await self.get_bucket_region(bucket, response)
/venv/lib/python3.11/site-packages/aiobotocore/utils.py:483: in get_bucket_region
    async with self._client as client:
E   TypeError: 'weakref.ProxyType' object does not support the asynchronous context manager protocol

rdbisme commented 1 year ago

@shiv-planetrics does this happen on Docker or you can reproduce locally? Which version of Python (minor included?)

thehesiod commented 1 year ago

thanks will look asap, looks like the change to fix a bug may have introduced one instead

shiv-planetrics commented 1 year ago

@rdbisme It's happening in Docker environment. I didn't test outside Docker though.

martindurant commented 1 year ago

I am watching here to see if we understand the problem. Happy to make any change required in s3fs to make it work with latest aiobotocore in time for our next release. I haven't had a chance to look into it yet myself.

we can reproduce this only on docker

That's really weird! With exactly the same environment?

rdbisme commented 1 year ago

I am watching here to see if we understand the problem. Happy to make any change required in s3fs to make it work with latest aiobotocore in time for our next release. I haven't had a chance to look into it yet myself.

we can reproduce this only on docker

That's really weird! With exactly the same environment?

Yes. Exactly the same. I did a diff of the pip freeze. The docker is running on Amazon Linux, while my local test was on Rocky 8. Kernel version was the same (except the patch version).

thehesiod commented 1 year ago

repro'd, yea this has something to do with weakref, digging

thehesiod commented 1 year ago

ends up that change wasn't needed! reverting and adding cmt. New build should be out shortly

thehesiod commented 1 year ago

sorry about the noise!

rdbisme commented 1 year ago

Thanks for the quick fix and reactivity @thehesiod

aio-libs / aiobotocore

AttributeError: aenter on 2.5.3 #1031

aio-libs / aiobotocore

AttributeError: __aenter__ on 2.5.3 #1031

AttributeError: aenter on 2.5.3 #1031