Add support for UpdateShardCount

[pjrt]

This PR is mostly just an updated copy of https://github.com/mhart/kinesalite/pull/48

That PR's author mentioned encountering some issue with discovery, however I did not understand what that meant.

Please review, it is possible I missed whatever issue he encountered.

[semenodm]

Is there any updates on this PR? I miss this feature in Localstack, because it looks like it allows to call this API but does nothing in response with underlying Kinesis stream.

[hampsterx]

because it looks like it allows to call this API but does nothing in response with underlying Kinesis stream.

@semenodm huh?

Are you saying this PR is incomplete?

I tested this just yesterday and it seems to work well (working on sharding behaviour in a python kinesis library) but didn't actually read all the data off the shards to see if it was completely functional. I presume it is just glancing at the code?

[semenodm]

i don't think this pr is merged to the mainline. I use latest localstack. if i call UpdateShardCount for the stream with 1 shard and try to update it to 2, it remains 1 shard.

[semenodm]

here is what i have

> docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED       STATUS       PORTS                                        NAMES
82a15e89f31d   localstack/localstack   "docker-entrypoint.sh"   2 hours ago   Up 2 hours   4571/tcp, 0.0.0.0:4566->4566/tcp, 8080/tcp   fs2-aws_localstack_1
>docker run --rm -it -e AWS_SECRET_ACCESS_KEY=dummy -e AWS_ACCESS_KEY_ID=dummy -e AWS_REGION=us-east-1 amazon/aws-cli --endpoint-url=http://0.0.0.0:4566 kinesis list-streams
{
    "StreamNames": [
        "test"
    ]
}
>docker run --rm -it -e AWS_SECRET_ACCESS_KEY=dummy -e AWS_ACCESS_KEY_ID=dummy -e AWS_REGION=us-east-1 amazon/aws-cli --endpoint-url=http://0.0.0.0:4566 kinesis describe-stream --stream-name test
{
    "StreamDescription": {
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "StartingHashKey": "0",
                    "EndingHashKey": "340282366920938463463374607431768211455"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49615611602771301094567112768827423688679043031870996482"
                }
            }
        ],
        "StreamARN": "arn:aws:kinesis:us-east-1:000000000000:stream/test",
        "StreamName": "test",
        "StreamStatus": "ACTIVE",
        "RetentionPeriodHours": 24,
        "EnhancedMonitoring": [
            {
                "ShardLevelMetrics": []
            }
        ],
        "EncryptionType": "NONE",
        "KeyId": null,
        "StreamCreationTimestamp": "2021-02-18T00:13:39+00:00"
    }
}

at this point i have stream test with one shard, let's try to update shard count to 2

>docker run --rm -it -e AWS_SECRET_ACCESS_KEY=dummy -e AWS_ACCESS_KEY_ID=dummy -e AWS_REGION=us-east-1 amazon/aws-cli --endpoint-url=http://0.0.0.0:4566 kinesis update-shard-count --stream-name test --target-shard-count 2 --scaling-type UNIFORM_SCALING
{
    "StreamName": "test",
    "CurrentShardCount": 1,
    "TargetShardCount": 2
}

this cli call accepted, and i expect 2 shards are created, the old one becomes parent for new 2 shards, lets check it

> docker run --rm -it -e AWS_SECRET_ACCESS_KEY=dummy -e AWS_ACCESS_KEY_ID=dummy -e AWS_REGION=us-east-1 amazon/aws-cli --endpoint-url=http://0.0.0.0:4566 kinesis describe-stream --stream-name test
{
    "StreamDescription": {
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "StartingHashKey": "0",
                    "EndingHashKey": "340282366920938463463374607431768211455"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49615611602771301094567112768827423688679043031870996482"
                }
            }
        ],
        "StreamARN": "arn:aws:kinesis:us-east-1:000000000000:stream/test",
        "StreamName": "test",
        "StreamStatus": "ACTIVE",
        "RetentionPeriodHours": 24,
        "EnhancedMonitoring": [
            {
                "ShardLevelMetrics": []
            }
        ],
        "EncryptionType": "NONE",
        "KeyId": null,
        "StreamCreationTimestamp": "2021-02-18T00:13:39+00:00"
    }
}

Nothing has changed, that's my problem

[hampsterx]

https://github.com/localstack/localstack/blob/master/localstack/services/kinesis/kinesis_starter.py

interesting, python project that runs nodejs services, what could go wrong :P

using latest..

npm install -g kinesalite
+ kinesalite@3.3.3

when I call this from python I get

botocore.exceptions.ClientError: An error occurred (UnknownOperationException) when calling the UpdateShardCount operation: 

so kinda surprised/confused its accepting the command?

[semenodm]

it is, but i believe there is a stub implemented in Localstack https://github.com/localstack/localstack/pull/1229

[jmcgrath207]

There is a issue with this pull request, while describe_stream does reflect the updated shards and returns them well, I can not fetch the updated shard ID in get_shard_iterator . I am assuming something in actions/getShardIterator.js needs to be changed.

[jmcgrath207]

Update shard count is also seeing duplicates. For example there are two shard ID that are the same here.

It should perform like this https://brandur.org/kinesis-by-example

[pjrt]

Closing as I don't know enough JS to fix this issue. Anyone else is free to pick it up.