Empty response when running `list_objects_v2`

aayushgupta899 commented 1 year ago

Describe the bug I am running the list_objects_v2 using aiobotocore, in a docker container on Amazon EMR. The following code snippet returns an empty response:

import asyncio
import aiobotocore
from aiobotocore.session import get_session

BUCKET_NAME = "test-bucket-20230713"

async def list_objects():
    session = get_session()

    async with session.create_client('s3', region_name='us-east-1') as client:
        response = await client.list_objects_v2(Bucket=BUCKET_NAME)
        print(response)

        for obj in response.get('Contents', []):
            print(obj['Key'])

loop = asyncio.run(list_objects())

Output:

{'ResponseMetadata': {'RequestId': 'QJ9FWRY2HJT32QAT', 'HostId': 'iplqiXNK3ot49VwoMyt6xnzH16aQXIUHLlQGA8PNFiiaXY2DcGcucz3cV9GUq3co6Mw6MnCmmA4=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'iplqiXNK3ot49VwoMyt6xnzH16aQXIUHLlQGA8PNFiiaXY2DcGcucz3cV9GUq3co6Mw6MnCmmA4=', 'x-amz-request-id': 'QJ9FWRY2HJT32QAT', 'date': 'Mon, 17 Jul 2023 20:13:04 GMT', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}}

Stack trace: stacktrace_aiobotocore.txt

Same command using botocore:

import botocore.session

BUCKET_NAME = "test-bucket-20230713"

session = botocore.session.get_session()
client = session.create_client('s3')
client.list_objects_v2(Bucket=BUCKET_NAME)

Output:

{'ResponseMetadata': {'RequestId': '4HZWK0QADQVM3T89', 'HostId': 'ogbCYrdCnE0Wse2dEolR/1MuF1lDqAprZ0yn1WyqgNiByFSqhNo8A2nvrHaLI+EgtkG9AzXFIr0=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'ogbCYrdCnE0Wse2dEolR/1MuF1lDqAprZ0yn1WyqgNiByFSqhNo8A2nvrHaLI+EgtkG9AzXFIr0=', 'x-amz-request-id': '4HZWK0QADQVM3T89', 'date': 'Mon, 17 Jul 2023 21:10:08 GMT', 'x-amz-bucket-region': 'us-east-1', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Contents': [{'Key': 'data/test2.bin', 'LastModified': datetime.datetime(2023, 7, 13, 20, 21, 10, tzinfo=tzlocal()), 'ETag': '"2f99a26fd3660ba94188d1b7106eda6a"', 'Size': 1024, 'StorageClass': 'STANDARD'}, {'Key': 'data/test_2.bin', 'LastModified': datetime.datetime(2023, 7, 13, 20, 21, 58, tzinfo=tzlocal()), 'ETag': '"e98a3e683a5fb991a3aecb5f42bb70cd"', 'Size': 1024, 'StorageClass': 'STANDARD'}, {'Key': 'data/test_old.txt', 'LastModified': datetime.datetime(2023, 7, 13, 18, 53, 26, tzinfo=tzlocal()), 'ETag': '"7a1ff55a2596e4d015683d012c9cc7ca"', 'Size': 0, 'StorageClass': 'STANDARD'}], 'Name': 'test-bucket-20230713', 'Prefix': '', 'MaxKeys': 1000, 'EncodingType': 'url', 'KeyCount': 3}

Stack trace: stacktrace_botocore.txt

I tried this with an older version of aiobotocore(==2.1.0), and I get the same issue. However, when I try with older versions of botocore (==1.27.96) and boto3 (1.24.96), it works:

import asyncio
import aiobotocore
from aiobotocore.session import get_session

BUCKET_NAME = "test-bucket-20230713"

async def list_objects():
    session = get_session()

    async with session.create_client('s3', region_name='us-east-1') as client:
        response = await client.list_objects_v2(Bucket=BUCKET_NAME)
        print(response)

        for obj in response.get('Contents', []):
             #print(obj['Key'])

loop = asyncio.run(list_objects())

Output:

{'ResponseMetadata': {'RequestId': 'PRE2VG4B2A3S2392', 'HostId': 'XvXd9+k5PlfKuQpIGVEaWPgK4pNM1kAM9JixYJ1tWLeOKwUvO1tF2ABnPRDu3Gx80igP0Itjv7w=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'XvXd9+k5PlfKuQpIGVEaWPgK4pNM1kAM9JixYJ1tWLeOKwUvO1tF2ABnPRDu3Gx80igP0Itjv7w=', 'x-amz-request-id': 'PRE2VG4B2A3S2392', 'date': 'Mon, 17 Jul 2023 21:41:03 GMT', 'x-amz-bucket-region': 'us-east-1', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Contents': [{'Key': 'data/test2.bin', 'LastModified': datetime.datetime(2023, 7, 13, 20, 21, 10, tzinfo=tzlocal()), 'ETag': '"2f99a26fd3660ba94188d1b7106eda6a"', 'Size': 1024, 'StorageClass': 'STANDARD'}, {'Key': 'data/test_2.bin', 'LastModified': datetime.datetime(2023, 7, 13, 20, 21, 58, tzinfo=tzlocal()), 'ETag': '"e98a3e683a5fb991a3aecb5f42bb70cd"', 'Size': 1024, 'StorageClass': 'STANDARD'}, {'Key': 'data/test_old.txt', 'LastModified': datetime.datetime(2023, 7, 13, 18, 53, 26, tzinfo=tzlocal()), 'ETag': '"7a1ff55a2596e4d015683d012c9cc7ca"', 'Size': 0, 'StorageClass': 'STANDARD'}], 'Name': 'test-bucket-20230713', 'Prefix': '', 'MaxKeys': 1000, 'EncodingType': 'url', 'KeyCount': 3}
data/test2.bin
data/test_2.bin
data/test_old.txt

Checklist

[x] I have reproduced in environment where pip check passes without errors
[x] I have provided pip freeze results
[x] I have provided sample code or detailed way to reproduce
[x] I have tried the same code in botocore to ensure this is an aiobotocore specific issue
[x] I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
[x] I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.4.2
aiohttp==3.7.4
aioitertools==0.5.1
amzn-pyodin==1.2
async-timeout==3.0.1
attrs==23.1.0
boto3==1.28.2
botocore==1.31.2
certifi==2021.5.30
chardet==4.0.0
charset-normalizer==2.0.7
docutils==0.15.2
fsspec==2023.4.0
future==0.18.1
gitdb==0.6.4
GitPython==2.1.8
hnswlib==0.3.4
idna==3.2
idna-ssl==1.0.0
importlib-metadata==6.0.0
jmespath==0.9.5
joblib==1.2.0
multidict==5.1.0
numpy==1.18.4
pandas==1.1.3
patsy==0.5.1
pip==21.0.1
pybind11==2.3.0
PyGreSQL==5.0.7
python-dateutil==2.8.2
pytz==2023.3
PyYAML==5.4.1
requests==2.28.1
s3fs==2023.6.0
s3transfer==0.6.1
scikit-learn==0.22
scipy==1.3.1
setuptools==57.4.0
six==1.16.0
sklearn==0.0
smmap==0.9.0
statsmodels==0.11.1
toml==0.10.2
torch==1.7.1
tqdm==4.64.0
typing-extensions==4.1.1
urllib3==1.26.13
wrapt==1.11.0
yarl==1.7.2
zipp==3.15.0

Environment:

Python Version: 3.8
OS name and version: Amazon Linux 2

Additional context This issue was initially encountered while using s3fs.S3FileSystem().ls() (issue) and was isolated to be an aiobotocore issue.

thehesiod commented 1 year ago

I tried it on one of my buckets and its working fine, is there a way to expose it to my private aws account: 848120881190 ?

aayushgupta899 commented 1 year ago

I don't think I will be able to expose it to your account to it as the code is running on a company AWS account. Anything else I can try to help you reproduce the issue?

thehesiod commented 1 year ago

ya if you could describe how to setup my account to repro this, IAM, bucket config, etc.

thehesiod commented 1 year ago

btw could you fix your log messages? it's spewing a bunch of errors because the socket it's trying to send is already closed

thehesiod commented 1 year ago

another thing would help if you could repro with moto

aayushgupta899 commented 11 months ago

This started working after I upgraded to aiobotocore==2.5.0. Not sure what changed though. Here are my pip freeze results:

aiobotocore==2.5.0
aiohttp==3.8.5
aioitertools==0.5.1
aiosignal==1.2.0
amzn-pyodin==1.2
async-timeout==4.0.2
attrs==23.1.0
boto3==1.28.65
botocore==1.31.65
charset-normalizer==3.0.0
docutils==0.15.2
frozenlist==1.3.3
fsspec==2022.2.0
gitdb==0.6.4
GitPython==2.1.8
hnswlib==0.3.4
idna==3.2
importlib-metadata==4.6.1
jmespath==0.9.5
joblib==1.2.0
multidict==6.0.4
numpy==1.18.4
pandas==1.1.3
patsy==0.5.1
pip==21.0.1
pybind11==2.3.0
PyGreSQL==5.0.7
python-dateutil==2.8.2
pytz==2023.3
s3fs==2022.2.0
s3transfer==0.7.0
scikit-learn==0.22
scipy==1.3.1
setuptools==57.4.0
six==1.16.0
sklearn==0.0
smmap==0.9.0
statsmodels==0.11.1
toml==0.10.2
tqdm==4.64.0
typing-extensions==4.1.1
urllib3==1.26.13
wrapt==1.15.0
yarl==1.8.2
zipp==3.15.0

aayushgupta899 commented 11 months ago

Closing this as this is no longer an issue. Please let me know if there are any more details which are required to be added for this.

aio-libs / aiobotocore

Empty response when running `list_objects_v2` #1024