[Bug]: Importing OTLP "File Exporter" JSON fails

[justfalter]

What happened?

jaegertracing/all-in-one 1.55 fails to import JSON generated by OTEL File Exporter.

Steps to reproduce

1. Spin up jaegertracing/all-in-one 1.55
2. Save trace example JSON data to test.json

   {"resourceSpans":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"scopeSpans":[{"scope":{},"spans":[{"traceId":"","spanId":"","parentSpanId":"","name":"operationA","startTimeUnixNano":"1581452772000000321","endTimeUnixNano":"1581452773000000789","droppedAttributesCount":1,"events":[{"timeUnixNano":"1581452773000000123","name":"event-with-attr","attributes":[{"key":"span-event-attr","value":{"stringValue":"span-event-attr-val"}}],"droppedAttributesCount":2},{"timeUnixNano":"1581452773000000123","name":"event","droppedAttributesCount":2}],"droppedEventsCount":1,"status":{"message":"status-cancelled","code":2}},{"traceId":"","spanId":"","parentSpanId":"","name":"operationB","startTimeUnixNano":"1581452772000000321","endTimeUnixNano":"1581452773000000789","links":[{"traceId":"","spanId":"","attributes":[{"key":"span-link-attr","value":{"stringValue":"span-link-attr-val"}}],"droppedAttributesCount":4},{"traceId":"","spanId":"","droppedAttributesCount":1}],"droppedLinksCount":3,"status":{}}]}]}]}
   {"resourceSpans":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"scopeSpans":[{"scope":{},"spans":[{"traceId":"","spanId":"","parentSpanId":"","name":"operationA","startTimeUnixNano":"1581452772000000321","endTimeUnixNano":"1581452773000000789","droppedAttributesCount":1,"events":[{"timeUnixNano":"1581452773000000424","name":"event-with-attr","attributes":[{"key":"span-event-attr","value":{"stringValue":"span-event-attr-val"}}],"droppedAttributesCount":2},{"timeUnixNano":"1581452773000000424","name":"event","droppedAttributesCount":2}],"droppedEventsCount":1,"status":{"message":"status-cancelled","code":2}},{"traceId":"","spanId":"","parentSpanId":"","name":"operationB","startTimeUnixNano":"1581452772000000343","endTimeUnixNano":"1581452773000001089","links":[{"traceId":"","spanId":"","attributes":[{"key":"span-link-attr","value":{"stringValue":"span-link-attr-val"}}],"droppedAttributesCount":3},{"traceId":"","spanId":"","droppedAttributesCount":4}],"droppedLinksCount":2,"status":{}}]}]}]}
   {"resourceSpans":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"scopeSpans":[{"scope":{},"spans":[{"traceId":"","spanId":"","parentSpanId":"","name":"operationA","startTimeUnixNano":"1581452772000000321","endTimeUnixNano":"1581452773000000789","droppedAttributesCount":1,"events":[{"timeUnixNano":"1581452773000000826","name":"event-with-attr","attributes":[{"key":"span-event-attr","value":{"stringValue":"span-event-attr-val"}}],"droppedAttributesCount":2},{"timeUnixNano":"1581452773000000826","name":"event","droppedAttributesCount":2}],"droppedEventsCount":1,"status":{"message":"status-cancelled","code":2}},{"traceId":"","spanId":"","parentSpanId":"","name":"operationB","startTimeUnixNano":"1581452772000200521","endTimeUnixNano":"1581452773000004789","links":[{"traceId":"","spanId":"","attributes":[{"key":"span-link-attr","value":{"stringValue":"span-link-attr-val"}}],"droppedAttributesCount":5},{"traceId":"","spanId":"","droppedAttributesCount":2}],"droppedLinksCount":3,"status":{}}]}]}]}
   {"resourceSpans":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"scopeSpans":[{"scope":{},"spans":[{"traceId":"","spanId":"","parentSpanId":"","name":"operationA","startTimeUnixNano":"1581452772000000321","endTimeUnixNano":"1581452773000000789","droppedAttributesCount":1,"events":[{"timeUnixNano":"1581452773000010925","name":"event-with-attr","attributes":[{"key":"span-event-attr","value":{"stringValue":"span-event-attr-val"}}],"droppedAttributesCount":2},{"timeUnixNano":"1581452773000010925","name":"event","droppedAttributesCount":2}],"droppedEventsCount":1,"status":{"message":"status-cancelled","code":2}},{"traceId":"","spanId":"","parentSpanId":"","name":"operationB","startTimeUnixNano":"1581452772000011821","endTimeUnixNano":"1581452772000012924","links":[{"traceId":"","spanId":"","attributes":[{"key":"span-link-attr","value":{"stringValue":"span-link-attr-val"}}],"droppedAttributesCount":2},{"traceId":"","spanId":"","droppedAttributesCount":2}],"droppedLinksCount":5,"status":{}}]}]}]}

3. Drag test.json onto Jaeger's Click or drag files to this area.
4. Get error:

   There was an error querying for traces:
   Error parsing JSON: JSON.parse: unexpected non-whitespace character after JSON data at line 2 column 1 of the JSON data

Expected behavior

I expect Jaeger to have loaded successfully loaded the Open Telemetry traces, per https://github.com/jaegertracing/jaeger/issues/4949

Relevant log output

No response

Screenshot

Additional context

The existing code only expects there to be a single JSON object entry in the OTEL File Exporter JSON output, but the file exporter specification says that the is in JSON lines format (file contains multiple JSON serialized objects, with a new-line separating each).

This is evident when looking at the test data in the original PR, as it is only a single JSON object entry. https://github.com/jaegertracing/jaeger/pull/5155/files

Jaeger backend version

1.55

SDK

No response

Pipeline

Example of the opentelemetry collector configuration I use to capture OTEL traces to OTEL file-exporter format:

extensions:
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
  memory_limiter:
    # 75% of maximum memory up to 2G
    limit_mib: 1536
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  file:
    path: /output/otel.json

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [file]

  extensions: [zpages]

Stogage backend

No response

Operating system

No response

Deployment model

No response

Deployment configs

No response

[yurishkuro]

Two issues with your data:

1) The input file is expected to be a single JSON object:

{
  "resourceSpans": ...
}

You have multiple objects (one per line) which do not have an equivalent representation in OTLP data types (it would be an array of TraceData object). It sounds like a problem worth solving.

2) All the trace/span IDs in the JSON are empty strings, which are not valid. So even if the structure was correct (you can try submitting just one line of that file), I suspect the parsing will then choke on the IDs.

[justfalter]

1. The input file is expected to be a single JSON object:

The implementation expects it to be a single JSON object, but the specification linked to by the origin feature request (#4949) states that the file is supposed to be in "JSON lines" format.

JSON lines file This file is a JSON lines file (jsonlines.org), and therefore follows those requirements:

• UTF-8 encoding
• Each line is a valid JSON value
• The line separator is \n
• The preferred file extension is jsonl

When I use the current file-exporter (https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/v0.96.0), it generates a file in "JSON lines" format.

2. All the trace/span IDs in the JSON are empty strings, which are not valid. So even if the structure was correct (you can try submitting just one line of that file), I suspect the parsing will then choke on the IDs.

In order to avoid this as a distraction, I'm including a trace that I've generated using opentelemetry's js SDK.

---

The opentelemetry collector contrib configuration:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    # The following ensures that one span is written per line.
    send_batch_size: 1
    send_batch_max_size: 1

exporters:
  file:
    path: /output/otel.json

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [file]

The generated trace otel.json. Note that there are 4 liness, one JSON object per line.

I can use the following jq to process the file into a single JSON object. The following concatenates all of the .resourceSpans arrays across each JSON object (per line), adding them to a single JSON object that Jaeger currently expects:

jq -c --slurp '{resourceSpans: map(.resourceSpans[])}' < otel.json > otel-out.json

Which gives me otel-out.json.

---

My expectation is that if Jaeger says that it will import OpenTelemetry JSON generated by the opentelemetry collector's file-export, then it should be able to handle a multi-line JSON file.

[NavinShrinivas]

A initial proposal: what if we get the frontend to add a [] and commas in between these json objects and loop over them with the same existing logic. Would this be a viable solution?

[varshith257]

@yurishkuro Any guidance on fixing it will be more helpful

[yurishkuro]

I will move to UI

[yurishkuro]

The current code that checks for OTLP format is here: https://github.com/jaegertracing/jaeger-ui/blob/d47fb0938a93cc468b89ffe6ee1feb7749804977/packages/jaeger-ui/src/utils/readJsonFile.tsx#L27

It effectively checks for JSON like {"resourceSpans":[...]}. I would suggest extending that code to:

1. also recognize[{"resourceSpans":[arr1]}, ... {"resourceSpans":[arrN]}]
2. join resourceSpans arrays into a single {"resourceSpans":[arr1..., arrN]} which then should be handled normally by the existing code

[Adarsh-jaiss]

• Hey @yurishkuro , I would like to work on this issue :)

[sfc-gh-kbregula]

I need this too. For now, I'm using a script like this that allows me to send all traces from a local file to Jagear.

import requests
import json
import argparse
from pathlib import Path

def send_post_request(url, json_data):
    headers = {'Content-Type': 'application/json'}
    response = requests.post(url, data=json.dumps(json_data), headers=headers)
    response.raise_for_status()
    return response

def process_file(input_file: Path, server_url: str):
    lines = input_file.read_text().splitlines()
    api_url = server_url.rstrip("/") + "/v1/traces"
    for line in lines:
        json_data = json.loads(line.strip())
        send_post_request(api_url, json_data)
    print("Finished")

def main():
    parser = argparse.ArgumentParser(description="Send saves traces to OLTP collector")
    parser.add_argument(
        '--server-url',
        type=str,
        default='http://localhost:4318',
        help="The server URL to send the POST requests to."
    )
    parser.add_argument('input_file', type=Path, help="The JSONL file to be processed.")

    args = parser.parse_args()

    process_file(args.input_file, args.server_url)

if __name__ == "__main__":
    main()

[yurishkuro]

@sfc-gh-kbregula if you're proficient in JS consider if you can bring https://github.com/jaegertracing/jaeger-ui/pull/2254 over the finish line.

[varshith257]

@yurishkuro I will take this up.

Lately I have searching this issue in Jaeger. But moved here 😅