ListObjectVersionsOutputTypeDef CommonPrefixes key can be missing

[guilhem-dvr]

Describe the bug

ListObjectVersionsOutputTypeDef has the key CommonPrefixes typed as a List[CommonPrefixTypeDef], but the key can be missing in the response to list_object_versions.

To Reproduce Steps to reproduce the behavior:

1. Install boto3-stubs[s3]
2. Run mypy/pyright on the following code sample

import boto3

s3_client = boto3.client("s3")
response = s3_client.list_object_versions(
    Bucket="some-bucket", Delimiter="/", Prefix="a/prefix/with/no/common/prefixes/"
)

common_prefixes = response["CommonPrefixes"]

3. The type checker is fine with this piece of code.

Actual output

The CommonPrefixes is missing from the response and the snippet fails when running on a real-life example.

KeyError: 'CommonPrefixes'

Expected output

There is no expected output, I would have liked my type checker to warn me about a potentially missing key.

Additional context I'm using macOS 14.2, python 3.9.5, poetry 1.8.1 to manage dependencies, boto3 & boto3-stubs 1.34.31

An example of a response I got from AWS where the key CommonPrefixes is missing.

{
  "DeleteMarkers": [
    {
      "IsLatest": "true",
      "Key": "some/key/to/a/file",
      "LastModified": "",
      "Owner": {
        "ID": "**********"
      },
      "VersionId": "**********"
    }
  ],
  "Delimiter": "/",
  "EncodingType": "url",
  "IsTruncated": "false",
  "KeyMarker": "",
  "MaxKeys": 1000,
  "Name": "bucket-name",
  "Prefix": "some/prefix/",
  "ResponseMetadata": {
    "HTTPHeaders": {
      "content-type": "application/xml",
      "date": "**********",
      "server": "AmazonS3",
      "transfer-encoding": "chunked",
      "x-amz-id-2": "**********",
      "x-amz-request-id": "**********"
    },
    "HTTPStatusCode": 200,
    "HostId": "**********",
    "RequestId": "**********",
    "RetryAttempts": 0
  },
  "VersionIdMarker": ""
}

[vemel]

Unfortunately, right now there is no way to tell if the key in output shape is optional or not. I am still investigating it.

By the way, you are getting key error because method name is incorrect: list_object_versionslist_object_versions

[guilhem-dvr]

By the way, you are getting key error because method name is incorrect: list_object_versionslist_object_versions

corrected, but this doesn't change my issue: the key is optional from the S3 response but the type checker misses that.

[vemel]

I will try to fix it. If you have time you can also take a look at botocore shapes and help me figure out how optional/required keys are marked for output shapes.

One solution would be to use the same key totality as for input shapes. But it is also not an ideal solution, because almost all keys are optional in input shapes.

Another solution would be to somehow figure out which keys are required in output shapes, but I still have no idea how to do it.

[vemel]

If I use the same rules as for input shapes, this is what I get :

ListObjectVersionsOutputTypeDef = TypedDict(
    "ListObjectVersionsOutputTypeDef",
    {
        "ResponseMetadata": ResponseMetadataTypeDef,
        "IsTruncated": NotRequired[bool],
        "KeyMarker": NotRequired[str],
        "VersionIdMarker": NotRequired[str],
        "NextKeyMarker": NotRequired[str],
        "NextVersionIdMarker": NotRequired[str],
        "Versions": NotRequired[List[ObjectVersionTypeDef]],
        "DeleteMarkers": NotRequired[List[DeleteMarkerEntryTypeDef]],
        "Name": NotRequired[str],
        "Prefix": NotRequired[str],
        "Delimiter": NotRequired[str],
        "MaxKeys": NotRequired[int],
        "CommonPrefixes": NotRequired[List[CommonPrefixTypeDef]],
        "EncodingType": NotRequired[Literal["url"]],
        "RequestCharged": NotRequired[Literal["requester"]],
    },
)

As you see, all keys are marked as not required. I do not think that this is a proper solution.

[vemel]

The issue has finally been fixed in mypy_boto3_builder 7.24.0 release. I have also released mypy-boto3-s3 1.34.91 with the fix included. Please update and let me know if it works as it should.