Closed samuelwithey closed 4 months ago
Thanks for reporting this issue. Here is where the paginator is modeled in Botocore (which Boto3 uses).
Note that it is missing this line: "limit_key": "MaxResults",
which other paginator models have. I believe this is causing the behavior to not be aligned with what is documented here: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html.
I created a new issue (https://github.com/aws/aws-sdk/issues/751) to track this in our cross-SDK repository going forward, since paginators are used across SDKs. I also reached out to the Redshift Data team regarding this and will share any updates in the new issue I created. Thanks again for bringing this to our attention.
This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.
Describe the bug
Identified an issue with AWS Paginators, specifically using the GetStatementResult paginator from the AWS SDK for Python (Boto3) RedshiftDataAPIService. The primary concern is related to the expected behaviour of pagination when setting the
MaxItems
parameter in thePaginationConfig
dictionary. Contrary to the AWS documentation expectations, providing a value forMaxItems
inPaginationConfig
does not result in multiple items representing pages in thePageIterator
object when the total number of items is less than the available items and aNextToken
is not returned to resume pagination.Related issue:
Expected Behavior
In the GetStatementResult documentation, it states a
PaginationConfig
dict can be provide to control pagination, whereMaxItems
limits the total number of items to return, and if the total number of items available is more than the value specified in max-items then aNextToken
will be provided in the output that you can use to resume pagination.Current Behavior
The description of
MaxItems
andStartingToken
does not match the observed behaviour. When setting theMaxItems
to 1000, the expected behaviour is for the returned number of records to be the same as the setMaxItems
, and if the returned number of items is less than the total number of items available, aNextToken
will be provided in the output. From the code snippet below, aNextToken
is not returned and is not present in the response syntax in RedshiftDataAPIService.Paginator.GetStatementResult. We can verify this by comparingTotalNumRows
to the length ofresult['Records']
and viewing the response dictionary keys.In the linked issue, it specifies using
PageSize
in the pagination config and using aposition
token, howeverPageSize
is not a valid parameter andposition
ormarker
are not returned.Reproduction Steps
Possible Solution
When applying the
dir()
function to the iterator or looking at the code for PageIterator, propertyresume_token
is present. Using theresume_token
forStartingToken
inPaginationConfig
forPageIterator
will resume pagination. The newPageIterator
will include another value for theresume_token
property if there are more items available.Additional Information/Context
No response
SDK version used
Python 3.10, boto3 1.28.78
Environment details (OS name and version, etc.)
MacOS Sonoma 14.4.1