Trouble querying otel traces: "Cannot read properties of undefined (reading 'key')"

[toby181]

What happened: Viewing traces using the Lucene Query Type = Traces results in "Cannot read properties of undefined (reading 'key')".

Viewing the trace indices in raw data format works.

That's the request from the query inspector, I've removed the response.

{
    "request": {
      "url": "api/datasources/proxy/uid/b95aca2f-86e4-4bc6-966c-d52367ece91e/_msearch?max_concurrent_shard_requests=5",
      "method": "POST",
      "data": "{\"search_type\":\"query_then_fetch\",\"ignore_unavailable\":true,\"index\":\"otel-v1-apm-span-*\"}\n{\"size\":10,\"query\":{\"bool\":{\"must\":[{\"range\":{\"startTime\":{\"gte\":1687176062997,\"lte\":1687348862997}}},{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}],\"filter\":[],\"should\":[],\"must_not\":[]}},\"aggs\":{\"traces\":{\"terms\":{\"field\":\"traceId\",\"size\":100,\"order\":{\"_key\":\"asc\"}},\"aggs\":{\"latency\":{\"max\":{\"script\":{\"source\":\"\\n                if (doc.containsKey('traceGroupFields.durationInNanos') && !doc['traceGroupFields.durationInNanos'].empty) {\\n                  return Math.round(doc['traceGroupFields.durationInNanos'].value / 10000) / 100.0\\n                }\\n                return 0\\n                \",\"lang\":\"painless\"}}},\"trace_group\":{\"terms\":{\"field\":\"traceGroup\",\"size\":1}},\"error_count\":{\"filter\":{\"term\":{\"traceGroupFields.statusCode\":\"2\"}}},\"last_updated\":{\"max\":{\"field\":\"traceGroupFields.endTime\"}}}}}}\n",
      "hideFromInspector": false
    },

What you expected to happen: All traces are displayed in table format as described in plugin description: "View all traces: Query (Lucene) leave blank Lucene Query Type: Traces Rerun the query If necessary, select Table visualization type Clicking on a trace ID in the table opens that trace in the Explore panel Trace View"

How to reproduce it (as minimally and precisely as possible): Basically as described in the screenshot above, there's no query or any filtering done. It's necessary to have traces in otel format in OpenSearch.

Anything else we need to know?: The traces have the otel format, not jaeger.

Setup of the Grafana OpenSearch data source:

Environment:

• Grafana version: 9.5.2
• Plugin version: 2.6.2

Thanks in advance!

[sarahzinger]

@toby181 thanks for reporting this issue! At first glance I'm guessing your comment here "The traces have the otel format, not jaeger." is probably the issue. All of our test data for this new Traces feature has been using jaegar formatted data I think, and we may have missed a common use-case! Let's see if we can fix that.

A few questions that might help speed that process up:

• I'm curious are you able to see your opensearch traces in the trace analytics dashboard in the opensearch dashboard just fine? Or do you see any bugs there as well. Just want to double check that this data is understandable to opensearch first.
• Would you mind sending us some of the raw sample trace data, or the process you followed to generate it? If you do send us some data, just try to remove any sensitive info first before sharing.

Thanks so much for your help with this issue!

[toby181]

@sarahzinger Thanks for your feedback and sorry for my late response.

• Yes, we're able to see the traces within OpenSerach Dashboards. So far no issues with displaying them. All of our traces are shipped with the data-prepper and are therefore in otel format.
• Attached some examples. It's one traceId with 5 spans. I've extracted the data from the _source field of the index. (files are renamed to .txt)
• Raw traces can be found in the opentelemetry documentation: https://opentelemetry.io/docs/instrumentation/net/getting-started/ otel-trace-1.json.txt otel-trace-2.json.txt otel-trace-3.json.txt otel-trace-4.json.txt otel-trace-5.json.txt

[idastambuk]

Hi @toby181, sorry for the delay on this. Thanks for providing the raw traces. I need some more info about the response, as it seems that it comes a format we're not expecting. We should be able to support otel format, and when I make the same query in the otel-v1-apm-span-* index, I don't get the same error.

Can you make the query from a dashboard panel and paste the raw response from the network tab (you can empty the "hits" array or any other sensitive info)? One last thing - we recently released part of the migration to the backend response processing. This includes trace requests, but only for queries from the Explore tab. If you make the request from there, I wonder if you get a different response? Thanks so much for your patience on this!