boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.99k stars 1.86k forks source link

S3: Expose both part size and full size in GET/HEAD requests #4169

Closed vincer closed 3 months ago

vincer commented 3 months ago

Describe the feature

I'd like to inspect an object and get its part size as well as its total size in a single request. The API does expose this information. If a HEAD request (or GET) is made with partNumber=1, then the response will have a content-length header that has the size of the part as well as a content-range header that also has the size of the whole object. e.g.

'content-range': 'bytes 0-400/5000'

In this example the full size of the object is 5000.

Currently boto3 doesn't have a way to get to this information easily with a single request. It only exposes the part size and not the whole object size. My workaround looks like this:


def get_full_size(parsed: dict, **kwargs):
    content_range = parsed["ResponseMetadata"]["HTTPHeaders"]["content-range"]
    full_size = int(content_range.split("/")[1])
    print(f"full size: {full_size}")

# The handler needs to be registered/unregistered every time
# to be able to access the full_size from the correct context each time.
client.meta.events.register(
    "after-call.s3.HeadObject",
    get_full_size,
    unique_id="get_full_size",
)
info = client.head_object(
    Bucket=bucket, Key=key, PartNumber=1
)
client.meta.events.unregister(
    "after-call.s3.HeadObject",
    unique_id="get_full_size",
)```

### Use Case

Getting both pieces of info in a single request.

### Proposed Solution

Expose part size as well as full object size when making `head_object` calls as well as in the `Object` resource so that only a single HEAD request is made for that information.

### Other Information

_No response_

### Acknowledgements

- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### SDK version used

1.34.69

### Environment details (OS name and version, etc.)

MacOS 14.5
tim-finnigan commented 3 months ago

Thanks for the feature request. There is a feature freeze on resources so your request involving the Object resource would not be considered. Changes involving the HeadObject API would need to be implemented by the S3 team. Please consider opening an issue in our cross-SDK repository if you'd like someone to reach out to S3 on your behalf. The content-length and content-range is already returned in the HeadObject response so if you're request is for something beyond that, then please explain more about what that involves and what use case you are trying to address.

github-actions[bot] commented 3 months ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.