mhart / kinesalite

An implementation of Amazon's Kinesis built on LevelDB
MIT License
808 stars 86 forks source link

Amount of shards are always 0 #86

Closed yooouuri closed 5 years ago

yooouuri commented 5 years ago
export const createKinesis = (port, log) => {
  const server = new Kinesalite();

  return new Promise((resolve, reject) => {
    server.listen(port, (error) => {
      if (error) {
        reject(error);
      }

      log(`🚀 Local kinesis is running at ${port}`);

      resolve();
    });
  });
};
export const createStream = (streamName, shards, log) => {
  return new Promise((resolve, reject) => {
    // create stream
    const child = exec(`aws kinesis create-stream --stream-name ${streamName} --shard-count ${shards} --endpoint-url http://localhost:4567`);

    child.on('error', (err) => {
      reject();

      throw err;
    });

    child.on('exit', async (code) => {
      if (code === 0) {
        const child2 = exec('aws kinesis describe-stream --stream-name event-stream-dev --endpoint-url http://localhost:4567');

        child2.stdout.on('data', (data) => {
          log(data);
        });

        resolve();
      }
    });
  });
};

Calling the both methods

(async () => {
  try {
    await createKinesis(4567, console.log);

    await createStream('event-stream-dev', 1, console.log);
  } catch (e) {
    console.error(e);
  }
})();

I get the following response:

🚀 Local kinesis is running at 4567
{
    "StreamDescription": {
        "Shards": [],
        "StreamARN": "arn:aws:kinesis:us-east-1:000000000000:stream/event-stream-dev",
        "StreamName": "event-stream-dev",
        "StreamStatus": "CREATING",
        "RetentionPeriodHours": 24,
        "EnhancedMonitoring": [
            {
                "ShardLevelMetrics": []
            }
        ],
        "EncryptionType": "NONE",
        "KeyId": null,
        "StreamCreationTimestamp": 1550425245
    }
}

The Shards array is always empty...

Running

$ aws kinesis create-stream --stream-name stream --shard-count 1 --endpoint-url http://localhost:4567

And

$ aws kinesis describe-stream --stream-name stream --endpoint-url http://localhost:4567

In the terminal gives me a stream with shards...

{
    "StreamDescription": {
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "StartingHashKey": "0",
                    "EndingHashKey": "340282366920938463463374607431768211455"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49593067858305541509637617739695903267775486461961306114"
                }
            }
        ],
        "StreamARN": "arn:aws:kinesis:us-east-1:000000000000:stream/stream",
        "StreamName": "stream",
        "StreamStatus": "ACTIVE",
        "RetentionPeriodHours": 24,
        "EnhancedMonitoring": [
            {
                "ShardLevelMetrics": []
            }
        ],
        "EncryptionType": "NONE",
        "KeyId": null,
        "StreamCreationTimestamp": 1550426190
    }
}
mhart commented 5 years ago

You can see the StreamStatus is still CREATING – this is to match production so that your code doesn't assume that the shards will be available immediately – you can adjust the amount of time the stream stays in this state using --createStreamMs as in the README

yooouuri commented 5 years ago

@mhart Ah now I see it, thanks for you reply!