mock_dynamodb: response from boto3 and moto is inconsistent when scanning a table with a limit

Hi, I recently implemented a new function that gets distinct primary keys in a DynamoDB table. I used the script (Scan with a modified exclusive start key) from this tutorial on AWS Blogs.

When writing tests for this function, I used moto to mock DynamoDB service. However, the response from DynamoDB and moto is inconsistent when scanning a table with a limit.

The script relies on scanning only 1 entry and skipping to the next entry with unique pk. Here is the actual response from the table scan call with limit 1:

{'Items': [{'pk': '123456_2'}], 'Count': 1, 'ScannedCount': 1, 'LastEvaluatedKey': {'message_id': '6789', 'pk': '123456_2'}, 'ResponseMetadata': {'RequestId': 'XXXOKIKHL63TH6E93DQ4PTEMO3VV4KQNSO5AEMVJF66Q9ASUADSC', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'Server', 'date': 'Wed, 29 Nov 2023 18:27:01 GMT', 'content-type': 'application/x-amz-json-1.0', 'content-length': '173', 'connection': 'keep-alive', 'x-amzn-requestid': 'XXXOKIKHL63TH6E93DQ4PTEMO3VV4KQNSO5AEMVJF66QXXXXXXJG', 'x-amz-crc32': '3747171547'}, 'RetryAttempts': 0}}

This is what the table scan with limit 1 returns with moto. Moto scans the whole table and then returns 1 item. However in the Boto3 documentation it says "If you did not use a FilterExpression in the scan request, then Count is the same as ScannedCount." So I expect to see the Count and ScannedCount values to be 1.

{'Items': [{'pk': '123456_2'}], 'Count': 1, 'ScannedCount': 10, 'LastEvaluatedKey': {'pk': '123456_2', 'message_id': '6789'}, 'ResponseMetadata': {'RequestId': 'XXXg6wCorZLSSnkfqVABaSsAPj2CE2QKQAjCnJDM3qbfbcRElxxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com', 'date': 'Wed, 29 Nov 2023 13:46:25 GMT', 'x-amzn-requestid': 'XXXg6wCorZLSSnkfqVABaSsAPj2CE2QKQAjCnJDM3qbfbcRElxxx', 'x-amz-crc32': '3259505435'}, 'RetryAttempts': 0}}

Here is my test case:

@mock_dynamodb
def test_printing_pks():
    db = boto3.resource('dynamodb', region_name='eu-central-1', aws_access_key_id='testing', aws_secret_access_key='testing', aws_session_token='testing')

    table = db.create_table(
        TableName="temp-test-writing",
        KeySchema=[
           {'AttributeName': 'pk','KeyType': 'HASH'},
            { 'AttributeName': 'message_id','KeyType': 'RANGE'},
        ],
        AttributeDefinitions=[
            {'AttributeName': 'pk', 'AttributeType': 'S' },
            { 'AttributeName': 'message_id',  'AttributeType': 'S'},
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 10,
            'WriteCapacityUnits': 10,
        },
    )
    notifications = notifications = [
        {'message_id': '6789', 'pk': '123456_2', 'title': 'Hey', 'status':'NEW'},
        {'message_id': '345654', 'pk': '15990465_25_de', 'title': 'Hey', 'status':'READ'},
        {'message_id': '12345679', 'pk': '15990465_4', 'title': 'Hey', 'status':'UNREAD'},
        {'message_id': '12345680', 'pk': '15990465_4', 'title': 'Message', 'status':'NEW'},
        {'message_id': '0', 'pk': '15990465_1_metadata', 'last_actionqueue_id': 765313, 'last_fetch_timestamp':1700230648},
        {'message_id': '345654', 'pk': '123456799_1', 'title': 'Hey', 'status':'NEW'},
        {'message_id': '345654', 'pk': '15990465_25_fr', 'title': 'Hey', 'status':'NEW'},
        {'message_id': '12345678', 'pk': '15990465_1', 'title': 'Hey', 'status':'READ'},
        {'message_id': '12345679', 'pk': '15990465_1', 'title': 'Hey', 'status':'NEW'},
        {'message_id': '345654', 'pk': '15990465_1', 'title': 'Hey', 'status':'UNREAD'}
    ]
    for notification in notifications:
        table.put_item(Item=notification)

    print_distinct_pks(table)

test_printing_pks()

Library versions that I'm using are:

boto3                     1.33.2
botocore                  1.33.2
moto                      4.2.10

Thank you. Pinar

getmoto / moto

mock_dynamodb: response from boto3 and moto is inconsistent when scanning a table with a limit #7080