If user forget to add attribute service.name while creating resource object using resource.New and otlpmetrichttp with a collector setup of loadbalancing exporter (which uses service for routing purpose), error msg that user receives in scenario doesn't say anything about the actual problem, this is the actual error msg that user gets
failed to upload metrics: context deadline exceeded: retry-able request failure
this doesn't happen with otlpmetricgrpc exporter, it sends the actual error message that comes from collector, this is the error msg.
failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = unable to get service name
Issue: it is difficult to figure out the actual reason behind the problem and it is happening because underlying collector sends the actual reason in response body, which otlpmetrichttp discards in case of retryable code and actual issue behind the failure gets missed.
Description
If user forget to add attribute service.name while creating resource object using
resource.New
andotlpmetrichttp
with a collector setup of loadbalancing exporter (which uses service for routing purpose), error msg that user receives in scenario doesn't say anything about the actual problem, this is the actual error msg that user getsthis doesn't happen with
otlpmetricgrpc
exporter, it sends the actual error message that comes from collector, this is the error msg.Issue: it is difficult to figure out the actual reason behind the problem and it is happening because underlying collector sends the actual reason in response body, which otlpmetrichttp discards in case of
retryable code
and actual issue behind the failure gets missed.Part1: otlpmetrichttp -----------api call------------> otel-collector (loadbalancer exporter)
Part2: otel-collector --- sends error via---> otlpmetric UploadMetrics func
Part3: otlpmetric UploadMetrics func --- discards body ---> sends error
retry-able request failure
instead of actual errorSetup Details
Environment
Steps To Reproduce
Export env variable
export ENDPOINT_DOMAIN='otel-gateway:4317' export REMOTE_WRITE_URL='http://victoriametrics:8429/api/v1/write'
Create golang app Dockerfile and build it under myapp in docker compose
docker-compose.yaml
otel-gateway-config.yaml
otel-collector-config.yaml
docker compose up
Make an api call to the myapp on
http://localhost:8081
Expected behavior
Solution will send messages as it is received from collector.