search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.9k stars 2.27k forks source link

awskinesisexporter: Add support for partitioning records by traceId #32027

Open preetsarb opened 5 months ago

preetsarb commented 5 months ago

Component(s)

exporter/awskinesis

Is your feature request related to a problem? Please describe.

Currently spans of a single trace can be spread over multiple shards, so its not possible (or require peer forwarding of spans) to perform tail sampling when consuming data from kinesis.

Describe the solution you'd like

Kinesis supports partitioning of records. traceId can be used as the partitionKey so all spans are routed to the same shard.

Describe alternatives you've considered

No response

Additional context

Similar feature was added for kafka exporter: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/12318

github-actions[bot] commented 5 months ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

jamesmoessis commented 5 months ago

@preetsarb have you considered using layered collectors connected with load balancing exporter to shard by trace ID? Could be an option in your interim.

Another problem I foresee with this is the Kinesis limit of 1MB per second per shard. Traces can easily be over 1MB so it begs the question how can that limit be avoided at scale.

preetsarb commented 3 months ago

@jamesmoessis we have been using something similar to load balancing exporter in our trace processing stack, which is hosted in AWS. The problem we have been facing is that current solution results in a lot of data transfer b/w instances across different AWS availability-zones which contributes to a big chunk of cost.

As for the issue with shard data rate limit, I don't think this change would have any impact. Mapping of traceIds to specific shard should be random enough to still have even distribution of records across shards. But will keep an eye out for any issues with hot partitioning.

Created a pull request to add this feature #33563

github-actions[bot] commented 1 month ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.