When calling DynamoDB client `list_exports` with NextToken receive a ValidationException

shomerj commented 1 month ago

Describe the bug

I am trying to fetch all of the DynamoDB exports for a given table. When I try and fetched the next page of exports using the NextToken returned in the response I get the following exception:

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the ListExports operation: Invalid Request: 1 validation error detected: Value *** at 'nextToken' failed to satisfy constraint: Member must have length less than or equal to 656

Fetching the next page (set of exports) works until the exports count is above 10. For example, if I set the MaxResults to 5, grabbing the next set of exports using the NextToken works. Grabbing the 10 through 15 exports will fail giving me the exception stated above.

Expected Behavior

The NextToken returned from the list_exports function should not raise a ValidationExceptions. Instead it should return all exports until all have been returned.

Current Behavior

boto3 version: '1.33.8'

The following is the stack trace returned:

Running test list Dynamo list exports script
Length of next_token: 848
Traceback (most recent call last):
  File "/var/code/test_exports.py", line 15, in <module>
    res = db.list_exports(TableArn=table_arn, MaxResults=25, NextToken=next_token)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.11/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the ListExports operation: Invalid Request: 1 validation error detected: Value 'c3eb5df0d20ff8322d64606edb6be7b4db24746959595b9be7a5a412834cad0e536de1041f86ba79d8e6854b6715a174c16517b7829a917bbcf0c6edbc32226d5413d0d44595176bb093e58754b56fa85bf18a8f0ab68f5cd66d7d184c596c4d1c3b0799240c98b5843e1117710d32750429a0ca2f55930ce3586f91e1f41c09474d305193e0bd0559db77786dfdca81ab21bab0f32a318200dcaff6d33e29338384bc9b70587a340381bc60f16aebda45ae9a5c6ed720505b6262b25b52104910ce20842050235aec7c0564c277f64596d5af29acc24a99d74c47b89dc716f5e0e716223882adf682376bc9b66b35442ed938f6060900781db5571e9282681e6b63f8366ef4c6ddb1b522d14ce81570b6cde4ae7334ee88afea9b1639fb642bff01fb2bd24aba31dc0c1aa75da70f1c13d84dcc7a30fe1889e1d11e9f08daa8130ce9c34fca060c3cd531d19a01186d2081a78baf209887bac44b8146114d1138ce22869c3901195d1f321fc52a8e347ab57880fa8a4d88b51d3ea4ff9f6cebc56cf10d9c90d073c18f98c6025b345b7a5ba0c1b567d229f4ddfc8a47482ff70000000000000000' at 'nextToken' failed to satisfy constraint: Member must have length less than or equal to 656

Reproduction Steps

The following is the test script I used to demonstrate the error. Private information is starred out:

import boto3

table_arn = ***
db_name = ***
catalog = "***
session = boto3.Session(profile_name=***)

if __name__ == "__main__":
    print("Running test list Dynamo list exports script")
    db = session.client("dynamodb", region_name="us-east-2")
    response = db.list_exports(TableArn=table_arn, MaxResults=25)
    next_token = response.get("NextToken")
    while next_token:
        print(f"Length of next_token: {len(next_token)}")
        res = db.list_exports(TableArn=table_arn, MaxResults=25, NextToken=next_token)
        next_token = res.get("NextToken")

    print("It worked!")

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.33.8

Environment details (OS name and version, etc.)

Amazon Linux 6.6.32-linuxkit

tim-finnigan commented 1 month ago

Thanks for reaching out. The Boto3 list_exports command involves a call to the underlying DynamoDB ListExports API and the NextToken is failing validation here. I tried to reproduce this issue but was not able to. First I created 11 exports:

import boto3

dynamodb = boto3.client('dynamodb')
s3 = boto3.client('s3')

NUM_EXPORTS = 11

for i in range(1, NUM_EXPORTS + 1):
    response = dynamodb.export_table_to_point_in_time(
        TableArn='arn:aws:dynamodb:us-west-2:account-id:table/table',
        S3Bucket='my-bucket',
        S3Prefix='my-prefix',
        ExportFormat='DYNAMODB_JSON'
    )

Then I ran list_exports:

import boto3

dynamodb = boto3.client('dynamodb')

response = dynamodb.list_exports()
print(len(response['ExportSummaries']))

while 'NextToken' in response:
    response = dynamodb.list_exports(NextToken=response['NextToken'])
    print(len(response['ExportSummaries']))

And as expected that outputs:

10
1

And if I specify MaxResults=25 in my initial call, then it will output 11 as expected.

So I'm no sure what is causing the issue you're experiencing. I recommend that you first update your version of Boto3 by uninstalling/reinstalling it. And if still seeing an issue, please share your full debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('') to your script.

github-actions[bot] commented 1 month ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

boto / boto3