Splunk HEC Exporter Transmission to HEC Endpoint Fails With Missing Body

[bdschaap]

Component(s)

exporter/splunkhec

What happened?

Description

Splunk HEC Exporter transmission to HEC endpoint fails when there is a missing body field. The otel logs data model lists body as an optional field. I'm using Faro Web SDK which has a log type (measurement) that does not contain a body field.

Steps to Reproduce

Transmit a log without a body field.

Expected Result

OTLP logs received and exported with a missing body field should not fail transmission to a HEC endpoint. Perhaps a default value should be set. Alternative is that the HEC endpoint (ie Splunk Forwarder, Splunk Enterprise, Splunk Cloud, etc) should accept a missing body field. I have encountered this behavior with a Splunk Forwarder.

Actual Result

400 response code is returned stating bad data format.

Collector version

opentelemetry-collector-contrib:0.88.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

exporters:
  splunk_hec/dev:
    token: "<guid>"
    endpoint: "https://splunkforwarder.mydomain.local:8088/services/collector/event"
    source: "otel"
    hec_metadata_to_otel_attrs:
      source: "service.name"        
    sourcetype: "faro"
    index: "dev"

Log output

No response

Additional context

No response

[github-actions[bot]]

Pinging code owners:

• exporter/splunkhec: @atoulme @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Agreed that the body of a log is optional, as stated in the spec here.

Can you share the full error message you're receiving? I'm wondering if this is possibly a duplicate of #29551.

[bdschaap]

Looks like #29551 was fixed in 0.89.0. I'm still on 0.88.0. I'll update first and report back.

[bdschaap]

Sorry I misread that. I'm guessing it will be in the next release. Will try when it's available.

[bdschaap]

I tried v0.91.0 and no longer receive an error but I can't find the log in Splunk. The log is still only transmitted and found in Splunk if I manually supply the body field in the log. I reverted to v0.88.0 to capture the error message below.

2023-12-12T14:41:22.416Z        error   exporterhelper/retry_sender.go:145      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "logs", "name": "splunk_hec", "error": "Permanent error: \"HTTP/1.1 400 Bad Request\\r\\n
Content-Length: 64\\r\\nConnection: Keep-Alive\\r\\nContent-Type: application/json; charset=UTF-8\\r\\nDate: Tue, 12 Dec 2023 14:41:21 GMT\\r\\nServer: Splunkd\\r\\nVary: Authorization\\r\\nX-Content-Type-Options: nosniff\\r\\nX-Frame-Options: SAMEORIGIN\\r\\n\\r\\n{
\\\"text\\\":\\\"Invalid data format\\\",\\\"code\\\":6,\\\"invalid-event-number\\\":0}\"", "dropped_items": 1}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/retry_sender.go:145
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send
        go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/logs.go:176
go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1
        go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/queue_sender.go:126
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1
        go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/internal/bounded_memory_queue.go:52

[atoulme]

"I can't find the log in Splunk" - how are you looking for it? Can you elaborate? Please open a support case with Splunk to follow up.

[bdschaap]

My Otel Collector sends logs to a Splunk Heavy Forwarder which then sends them to Splunk Cloud. I'm searching for the logs in Splunk Cloud. The logs with the missing body field cannot be found in the specified index and sourcetype on v0.91.0. If I manually edit the json to add the body field and then retransmit the log to my Otel Collector then I find that log in Splunk Cloud. I also checked the splunkd internal logs to see if I could find the reason but didn't find anything.

I'll open a support case with Splunk.

[atoulme]

To be precise, there is no such thing as a "body" field. You are referring to the content of the value associated with the "event" key in the JSON object sent as a HEC event. Please let me know if I'm missing something in this clarification.

[bdschaap]

Makes sense. I'm referring to the name of the field in Otel log data model. I believe the log is transformed to a HEC event by the Splunk HEC Exporter.

[atoulme]

This is resolved with https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/29551. Closing.