Exporting failed: span tag key/value cannot be blank

[syamalaids]

Describe the bug I have instrumented my application with open telemetry. Once in a while I get the following error. Puzzling thing is it does not give me a clue which span is causing the issue. Could you guide me how I can find that out or get more details on the error?

I am using TanzuObservabilityExporter.

What version did you use? v0.42.0

What config did you use? receivers: otlp: protocols: grpc:

exporters: logging: logLevel: debug

Data sources: traces, metrics, logs

file: path: ./collector.json

tanzuobservability: traces:

hostname and 'customeTracingListenerPorts' of the Wavefront Proxy

   endpoint: "http://dev-proxy-temp.cyracomdev.com:30001"

processors: batch:

extensions: health_check: pprof: zpages:

service: extensions: [] pipelines: traces: receivers: [otlp] exporters: [logging, tanzuobservability] processors: [batch]

Environment OS: Windows Server 2012 R2 standard. Compiler(if manually compiled): (e.g., "go 14.2")

Additional context Add any other context about the problem here.

[jpkrohling]

cc @oppegard @thepeterstone @keep94

This message seems to be coming from a dependency of the Tanzu Observability Exporter:

$ go mod vendor
$ grep "cannot be blank" . -r
./vendor/github.com/wavefronthq/wavefront-sdk-go/senders/formatter.go:          return "", errors.New("metric point tag value cannot be blank")
./vendor/github.com/wavefronthq/wavefront-sdk-go/senders/formatter.go:          return "", errors.New("histogram tag value cannot be blank")
./vendor/github.com/wavefronthq/wavefront-sdk-go/senders/formatter.go:          return "", errors.New("span tag key/value cannot be blank")

[thepeterstone]

Thanks @syamalaids for the detailed report and @jpkrohling for the triage! I'll get a fix prioritized.

[oppegard]

@syamalaids we're going to improve the error message, but also check if there are empty span tags and drop them, so that the rest of the span can be exported to Wavefront.

In the meantime, I see you've enabled the Collector's logging exporter with debug level. You can search for STRING() in the log output and identify which span attributes/tags are empty. Once you've done that, you can use the attributes processor to delete the empty attributes or provide default values.

[syamalaids]

@oppegard Thank you I was able to find the culprit by searching for STRING(). I no longer see this error.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions[bot]]

Pinging code owners: @oppegard @thepeterstone @keep94. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

Based on the last message, I assume this has been sorted out. I'm closing this issue.