search

awslabs / amazon-kinesis-client

Client library for Amazon Kinesis
Apache License 2.0
644 stars 467 forks source link

migrating from 1.x to 2.x increases API usage / costs significantly #664

Open eyusim opened 4 years ago

eyusim commented 4 years ago

The defaults in the 2.x configuration to use Fan-out and detailed Cloudwatch metrics can be very surprising when migrating from 1.x, which does not do either of those things. We noticed a significant rise in Kinesis and Cloudwatch costs (we use datadog to pull custom metrics and didn't expect to all of a sudden have 100s of thousands more metrics)

These things should be spelled out as warnings in the docs (https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html)

eyusim commented 4 years ago

in addition 2.x increased the load on our servers significantly, which would have required us to use much more compute resources. we ended up rolling back to the latest 1.x version

mrdziuban commented 2 years ago

I was surprised by this as well. Do I understand correctly that passing an instance of PollingConfig to the Scheduler should disable the new fan-out behavior and eliminate additional Kinesis costs?