hypertrace / hypertrace-collector

OpenTelemetry collector distribution for Hypertrace platform
6 stars 5 forks source link

fix: kafkaexporter:- log span info early when producer maxMessageBytes is exceeded #69

Closed tim-mwangi closed 2 years ago

tim-mwangi commented 2 years ago

Description

We want to diagnose why the kafkaexporter is failing to export some messages because they are too large.

2022-01-12T14:10:33.425Z  info  exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.  {"kind": "exporter", "name": "kafka", "error": "Failed to deliver 1 messages due to kafka server: Message was too large, server rejected it to avoid allocation error.", "interval": "15.687771456s"}
2022-01-12T14:10:45.128Z  info  exporterhelper/queued_retry.go:215  Exporting failed. Will retry the request after interval.  {"kind": "exporter", "name": "kafka", "error": "Failed to deliver 1 messages due to kafka server: Message was too large, server rejected it to avoid allocation error.", "interval": "4.033118042s"}

Since for this exporter, each span is converted to a message, the implication is that some spans when encoded using "jaeger_proto" are larger that the Producer.MaxMessageBytes which by default is 1M. So some spans are 1MB or more in size.

Testing

Added some tests for the new debug mashaler.

Checklist: