head_object failure - Githubissues

mzealey commented 1 year ago

Describe the bug

On aiobotocore==2.5.0, ported directly from botocore functionality:

    async def exists(self, filename: str) -> bool:
        async with self.get_client() as client:
            try:
                await client.head_object(Bucket=self.bucket, Key=filename)
                return True
            except ClientError as e:
                if e.response["Error"]["Code"] == "404":
                    return False
                raise

produces the following failure:

app/storage/base.py:137: in exists
    await client.head_object(Bucket=self.bucket, Key=self.get_path(filename))
/usr/local/lib/python3.11/site-packages/aiobotocore/client.py:354: in _make_api_call
    http, parsed_response = await self._make_request(
/usr/local/lib/python3.11/site-packages/aiobotocore/client.py:379: in _make_request
    return await self._endpoint.make_request(
/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py:97: in _send_request
    success_response, exception = await self._get_response(
/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py:139: in _get_response
    success_response, exception = await self._do_get_response(
/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py:191: in _do_get_response
    response_dict = await convert_to_response_dict(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

http_response = <tests.unit.test_storage.mock_http_session_send.<locals>.TestResponse object at 0x7f988e1f3350>, operation_model = OperationModel(name=HeadObject)

    async def convert_to_response_dict(http_response, operation_model):
        """Convert an HTTP response object to a request dict.

        This converts the requests library's HTTP response object to
        a dictionary.

        :type http_response: botocore.vendored.requests.model.Response
        :param http_response: The HTTP response from an AWS service request.

        :rtype: dict
        :return: A response dictionary which will contain the following keys:
            * headers (dict)
            * status_code (int)
            * body (string or file-like object)

        """
        response_dict = {
            # botocore converts keys to str, so make sure that they are in
            # the expected case. See detailed discussion here:
            # https://github.com/aio-libs/aiobotocore/pull/116
            # aiohttp's CIMultiDict camel cases the headers :(
            'headers': HTTPHeaderDict(
                {
                    k.decode('utf-8').lower(): v.decode('utf-8')
                    for k, v in http_response.raw.raw_headers
                }
            ),
            'status_code': http_response.status_code,
            'context': {
                'operation_name': operation_model.name,
            },
        }
        if response_dict['status_code'] >= 300:
            response_dict['body'] = await http_response.content
        elif operation_model.has_event_stream_output:
            response_dict['body'] = http_response.raw
        elif operation_model.has_streaming_output:
            length = response_dict['headers'].get('content-length')
            response_dict['body'] = StreamingBody(http_response.raw, length)
        else:
>           response_dict['body'] = await http_response.content
E           TypeError: object str can't be used in 'await' expression

/usr/local/lib/python3.11/site-packages/aiobotocore/endpoint.py:63: TypeError

Checklist

[x] I have reproduced in environment where pip check passes without errors
[x] I have provided pip freeze results
[x] I have provided sample code or detailed way to reproduce
[x] I have tried the same code in botocore to ensure this is an aiobotocore specific issue
[ ] I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
[ ] I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.5.0
aiohttp==3.8.4
aioitertools==0.11.0
aiosignal==1.3.1
alembic==1.6.5
anyio==3.6.2
asgiref==3.5.2
asn1crypto==1.5.1
asttokens==2.2.1
async-timeout==4.0.2
attrs==22.2.0
backcall==0.2.0
bcrypt==3.2.0
black==23.1.0
blinker==1.4
botocore==1.29.76
botocore-stubs==1.21.38
certifi==2022.12.7
cffi==1.15.1
cfgv==3.3.1
charset-normalizer==3.1.0
click==8.1.3
clickatell==0.1.2
coverage==7.2.2
decorator==5.1.1
dependency-injector==4.41.0
Deprecated==1.2.13
distlib==0.3.6
dnspython==2.3.0
email-validator==1.3.1
executing==1.2.0
factory-boy==3.2.0
Faker==18.3.1
fancycompleter==0.9.1
fastapi==0.95.0
filelock==3.10.6
flake8==3.9.0
flake8-isort==4.0.0
frozenlist==1.3.3
greenlet==2.0.2
gunicorn==20.1.0
h11==0.12.0
httpcore==0.15.0
httpx==0.23.0
identify==2.5.22
idna==3.4
iniconfig==2.0.0
ipython==8.10.0
isort==5.8.0
jedi==0.18.2
jmespath==1.0.1
kafka-python==2.0.2
lz4==3.1.3
Mako==1.2.4
MarkupSafe==2.1.2
matplotlib-inline==0.1.6
mccabe==0.6.1
mock==5.0.1
multidict==6.0.4
mypy==0.982
mypy-boto3-s3==1.18.39
mypy-extensions==1.0.0
nodeenv==1.7.0
packaging==23.0
parso==0.8.3
passlib==1.7.4
pastel==0.2.1
pathspec==0.11.1
pdbpp==0.10.2
pexpect==4.8.0
phonenumberslite==8.12.15
pickleshare==0.7.5
Pillow==9.3.0
platformdirs==3.2.0
pluggy==1.0.0
poethepoet==0.10.0
pre-commit==2.13.0
prompt-toolkit==3.0.38
psycopg2-binary==2.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pycodestyle==2.7.0
pycparser==2.21
pydantic==1.10.4
pyflakes==2.3.1
Pygments==2.14.0
PyJWT==1.7.1
pyrate-limiter==2.5.0
pyrepl==0.9.0
pytest==7.2.1
pytest-asyncio==0.19.0
pytest-cov==2.10.1
pytest-dotenv==0.5.2
pytest-mock==3.3.1
python-dateutil==2.8.2
python-dotenv==0.15.0
python-editor==1.0.4
python-multipart==0.0.6
pytz==2023.2
PyYAML==6.0
redis==4.0.2
requests==2.28.2
rfc3986==1.5.0
scramp==1.4.1
sentry-sdk==1.9.10
six==1.16.0
sniffio==1.3.0
SQLAlchemy==1.4.41
SQLAlchemy-Utils==0.38.3
sqlalchemy2-stubs==0.0.2a27
stack-data==0.6.2
starlette==0.26.1
termcolor==1.1.0
testfixtures==6.18.5
time-machine==2.3.1
toml==0.10.2
tomlkit==0.11.7
traitlets==5.9.0
twilio==6.59.1
types-aiobotocore==2.5.0.post1
types-aiobotocore-s3==2.5.0.post1
types-awscrt==0.16.13.post1
types-certifi==0.1.4
types-mock==0.1.3
types-python-dateutil==2.8.1
types-redis==4.0.2
types-requests==2.28.11.5
types-setuptools==57.0.0
types-six==0.1.7
types-termcolor==1.1.2
types-toml==0.1.3
types-urllib3==1.26.25.4
typing_extensions==4.5.0
urllib3==1.26.15
uvicorn==0.21.1
virtualenv==20.21.0
wcwidth==0.2.6
Werkzeug==2.2.3
wmctrl==0.4
wrapt==1.15.0
yarl==1.8.2

Environment:

Python Version: 3.11
OS name and version: docker python:3.11-buster

Additional context Add any other context about the problem here.

thehesiod commented 1 year ago

seems like your test framework is not mocking the response correctly. You can't use the same mocks from botocore as aiobotocore is async and uses aiohttp

thehesiod commented 11 months ago

going to close, your mock for content needs to match the AioAWSRequest: https://github.com/aio-libs/aiobotocore/blob/88a43098b1194bf759581acc710ad5bdbdd99a96/aiobotocore/awsrequest.py#L18

aio-libs / aiobotocore

head_object failure #1002