The KinesisSinkSpec is often failing with message....
List(ReceivedEvents(List(test-payload),Some(2023-11-23T13:39:37.852Z)), ReceivedEvents(List(test-payload),Some(2023-11-23T13:39:37.852Z)))' doesn't have size 1 but size 2 (KinesisSourceSpec.scala:67)
I believe this is happening because localstack lazily initializes the kinesis stream when it receives the first http request. And during that lazy initialization, our KinesisSink assumes that the first http request failed, so it re-sends the batch of events.
The correct fix is to explicitly create the kinesis stream via the API, instead of relying on the localstack config param KINESIS_INITIALIZE_STREAMS (which is deprecated anyway).
But as a short-term workaround, it seems we can avoid the problem just by bumping localstack to a slightly newer version. It is unclear why this fix works.
The KinesisSinkSpec is often failing with message....
I believe this is happening because localstack lazily initializes the kinesis stream when it receives the first http request. And during that lazy initialization, our KinesisSink assumes that the first http request failed, so it re-sends the batch of events.
The correct fix is to explicitly create the kinesis stream via the API, instead of relying on the localstack config param
KINESIS_INITIALIZE_STREAMS(which is deprecated anyway).But as a short-term workaround, it seems we can avoid the problem just by bumping localstack to a slightly newer version. It is unclear why this fix works.