localstack / localstack

💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline
https://localstack.cloud
Other
53.98k stars 3.91k forks source link

bug: Kinesis list-shards does not handle shard filter correctly when using AT_TRIM_HORIZON #10939

Open z3d1k opened 3 months ago

z3d1k commented 3 months ago

Is there an existing issue for this?

Current Behavior

When listing shards for Kinesis stream and specifying shard filter type as AT_TRIM_HORIZON, the response is the same as when shard filter type is set to FROM_TRIM_HORIZON - contains all currently available shards, existing or created after TRIM_HORIZON.

➜  ~ awslocal kinesis create-stream --stream-name testing --shard-count 2
➜  ~ awslocal kinesis list-shards --stream-name testing --no-cli-pager --query 'Shards[].ShardId' --output table
--------------------------
|       ListShards       |
+------------------------+
|  shardId-000000000000  |
|  shardId-000000000001  |
+------------------------+
➜  ~ awslocal kinesis update-shard-count --stream-name testing --target-shard-count 4 --scaling-type UNIFORM_SCALING
{
    "StreamName": "testing",
    "CurrentShardCount": 2,
    "TargetShardCount": 4
}
➜  ~ awslocal kinesis list-shards --stream-name testing --shard-filter Type=AT_TRIM_HORIZON --no-cli-pager --query 'Shards[].ShardId' --output table
--------------------------
|       ListShards       |
+------------------------+
|  shardId-000000000000  |
|  shardId-000000000001  |
|  shardId-000000000002  |
|  shardId-000000000003  |
|  shardId-000000000004  |
|  shardId-000000000005  |
+------------------------+
➜  ~ awslocal kinesis list-shards --stream-name testing --shard-filter Type=FROM_TRIM_HORIZON --no-cli-pager --query 'Shards[].ShardId' --output table
--------------------------
|       ListShards       |
+------------------------+
|  shardId-000000000000  |
|  shardId-000000000001  |
|  shardId-000000000002  |
|  shardId-000000000003  |
|  shardId-000000000004  |
|  shardId-000000000005  |
+------------------------+

Expected Behavior

Request with shard filter type set to AT_TRIM_HORIZON should return only shards that were open at TRIM_HORIZON as per documentation:

AT_TRIM_HORIZON - the response includes all the shards open at TRIM_HORIZON. https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ShardFilter.html

Same sequence of commands when calling AWS produces the desired result:

➜  ~ aws kinesis create-stream --stream-name testing --shard-count 2
➜  ~ aws kinesis list-shards --stream-name testing --query 'Shards[].ShardId' --no-cli-pager --output table
--------------------------
|       ListShards       |
+------------------------+
|  shardId-000000000000  |
|  shardId-000000000001  |
+------------------------+
➜  ~ aws kinesis update-shard-count --stream-name testing --target-shard-count 4 --scaling-type UNIFORM_SCALING --no-cli-pager
{
    "StreamName": "testing",
    "CurrentShardCount": 2,
    "TargetShardCount": 4,
    "StreamARN": "arn:aws:kinesis:us-east-1:xxxxxxxxxxxx:stream/testing"
}
➜  ~ aws kinesis list-shards --stream-name testing --shard-filter Type=AT_TRIM_HORIZON --query 'Shards[].ShardId' --no-cli-pager --output table
--------------------------
|       ListShards       |
+------------------------+
|  shardId-000000000000  |
|  shardId-000000000001  |
+------------------------+
➜  ~ aws kinesis list-shards --stream-name testing --shard-filter Type=FROM_TRIM_HORIZON --query 'Shards[].ShardId' --no-cli-pager --output table
--------------------------
|       ListShards       |
+------------------------+
|  shardId-000000000000  |
|  shardId-000000000001  |
|  shardId-000000000002  |
|  shardId-000000000003  |
|  shardId-000000000004  |
|  shardId-000000000005  |
+------------------------+

How are you starting LocalStack?

With the localstack script

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

localstack start -d

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

# Create new stream
awslocal kinesis create-stream --stream-name testing --shard-count 2
# Stream should have 2 initial shards
awslocal kinesis list-shards --stream-name testing --no-cli-pager --query 'Shards[].ShardId' --output table
# Scale stream up to 4 shards
awslocal kinesis update-shard-count --stream-name testing --target-shard-count 4 --scaling-type UNIFORM_SCALING
# Get shards open at TRIM_HORIZON
awslocal kinesis list-shards --stream-name testing --shard-filter Type=AT_TRIM_HORIZON --no-cli-pager --query 'Shards[].ShardId' --output table
# Get all shards from TRIM_HORIZON
awslocal kinesis list-shards --stream-name testing --shard-filter Type=FROM_TRIM_HORIZON --no-cli-pager --query 'Shards[].ShardId' --output table

Environment

- OS: Ubuntu 24.04, also tried on MacOS
- LocalStack:
  LocalStack version: 3.4.1.dev
  LocalStack Docker image sha: sha256:0f085d73a2f3a3e762ab74daa89d42c654ee5bbe2a9c863966160fdf8d5604a4
  LocalStack build date: 2024-06-01
  LocalStack build git hash: 7295eb303

Anything else?

No response

localstack-bot commented 3 months ago

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.