select_object_content not working in latest boto3

JayFields commented 3 hours ago

Describe the bug

s3_client.select_object_content() has stopped working in the latest boto3 release (possibly more). It fails with TypeError: a bytes-like object is required, not 'HTTPResponse'

Regression Issue

[X] Select this option if this issue appears to be a regression.

Expected Behavior

A successful response that points to bucket data when issuing a query such as:

result = s3_client.select_object_content(Bucket='some-bucket',
    Key='some/path/somefile.json.gz',
    ExpressionType='SQL',
    Expression="SELECT s.id FROM s3object s WHERE s.criteria != 'some_criteria'",
    InputSerialization={'CompressionType': 'GZIP', 'JSON': {'Type': 'Lines'}},
    OutputSerialization={'JSON': {}})

Response with boto3 1.34.148 (from about 3 months ago):

{'ResponseMetadata': {
    'RequestId': '<redacted>',
    'HostId': '<redacted>',
    'HTTPStatusCode': 200,
    'HTTPHeaders': {
        'x-amz-id-2': '<redacted>',
        'x-amz-request-id': '<redacted>',
        'date': 'Tue, 22 Oct 2024 17:00:37 GMT',
        'transfer-encoding': 'chunked', 'server': 'AmazonS3'},
     'RetryAttempts': 1}, 'Payload': <botocore.eventstream.EventStream object at 0x109130fb0>

Current Behavior

In the latest boto3 1.35.45 I get a stack trace:

Traceback (most recent call last):
  File "/Users/me/my_project/lambdas/my_lambda/app.py", line 242, in <module>
    pgUtils.main_handler(lambda_handler)
  File "/Users/me/src/my_layer/utils.py", line 23, in main_handler
    result = callback(event_json, {})
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/lambdas/my_lambda/app.py", line 193, in lambda_handler
    review_ids = run_the_query()
                 ^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/lambdas/my_lambda/app.py", line 54, in run_the_query
    result = s3_client.select_object_content(Bucket='some-bucket',
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/client.py", line 1005, in _make_api_call
    http, parsed_response = self._make_request(
                            ^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/client.py", line 1029, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/endpoint.py", line 216, in _send_request
    success_response, exception = self._get_response(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/endpoint.py", line 239, in _get_response
    success_response, exception = self._do_get_response(
                                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/endpoint.py", line 306, in _do_get_response
    self._event_emitter.emit(
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
               ^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/handlers.py", line 1252, in _handle_200_error
    if _looks_like_special_case_error(
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/my_project/env/lib/python3.12/site-packages/botocore/handlers.py", line 174, in _looks_like_special_case_error
    parser.feed(body)

Reproduction Steps

Perform an S3 query with boto3 1.35.45 similar to that described in Expected Behavior

Possible Solution

Revert the code change that introduced the issue

Additional Information/Context

No response

SDK version used

1.35.45

Environment details (OS name and version, etc.)

MacOS (ARM) and Lambda (x86)

tim-finnigan commented 3 hours ago

Thanks for reporting the issue. This appears to be same issue reported in https://github.com/boto/botocore/issues/3284. As mentioned there:

...we were able to reproduce this in 1.35.45. A fix is pending release here: #3285

https://github.com/boto/botocore/pull/3285 was just merged so should the fix should be released soon. We'll update these issues when the new release including the fix is out, and you can also check the CHANGELOG for updates.

tim-finnigan commented 28 minutes ago

This should now be fixed in version 1.35.46. Please try updating your version of boto3 and let us know if you're still running into any issues. I'll close this for consolidation and follow up on the original issue (https://github.com/boto/botocore/issues/3284).

boto / boto3