signalfx / splunk-otel-collector

Apache License 2.0
187 stars 154 forks source link

Cannot set a resource to a non-string value #1234

Closed dnitschke-splunk closed 2 years ago

dnitschke-splunk commented 2 years ago

Description

The documentation of the log data model at https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-resource states that the resource field is a map where the keys are of type string and the values of type any. However, trying to set a resource to a non-string value fails with error message

Failed to process entry        {"kind": "receiver", "name": "filelog/test_parsing", "operator_id": "$.regex_parser", "operator_type": "regex_parser", "error": {"description": "set parse_to: cannot set a resource to a non-string value"}, "action": "send", "entry": {"timestamp":"2022-02-11T11:45:52.559646658Z","body":null,"severity":0}}

I think this is due to type checking in https://github.com/open-telemetry/opentelemetry-log-collection/blob/89d49a3897633119bfe20628ff10f945d9741636/entry/resource_field.go#L44.

Setup

Steps to reproduce

Create a file /tmp/test_parsing.log with following content:

<30>1 2022-02-11T09:09:14.936012+00:00 myhost.local systemd 1 - -  splunk-otel-collector.service: Succeeded.

Monitor the file using a filelog receiver and write data to a new file /tmp/otel_parsing.out as configured in the agent_config.yaml below. Check OTel Collector logs using journalctl -f -u splunk-otel-collector.

agent_config.yaml

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  http_forwarder:
    ingress:
      endpoint: 0.0.0.0:6060
    egress:
      endpoint: "${SPLUNK_API_URL}"
  smartagent:
    bundleDir: "${SPLUNK_BUNDLE_DIR}"
    collectd:
      configDir: "${SPLUNK_COLLECTD_DIR}"
  zpages:
  memory_ballast:
    size_mib: ${SPLUNK_BALLAST_SIZE_MIB}

receivers:
  filelog/test_parsing:
    include: 
      - /tmp/test_parsing.log
    start_at: beginning
    include_file_name: false
    operators:
      - id: regex_parser
        type: regex_parser
        parse_from: $$body
        parse_to: $$resource['fields']
        regex: '^<(?P<priority>\d{1,3})>\d{1,2}\s(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+(?:[+-]\d{2}:\d{2}|Z))\s(?P<host>\S*)'

processors:
  batch:
  memory_limiter:
    check_interval: 2s
    limit_mib: ${SPLUNK_MEMORY_LIMIT_MIB}
  resourcedetection:

exporters:
  file/test_parsing:
    path: /tmp/otel_parsing.out

service:
  telemetry:
    logs:
      level: "debug"
  extensions: [health_check, http_forwarder, zpages, memory_ballast, smartagent]
  pipelines:
    logs:
      receivers:
        - filelog/test_parsing
      processors:
        - memory_limiter
        - batch
      exporters:
        - file/test_parsing
dnitschke-splunk commented 2 years ago

Seems to be addressed in https://github.com/open-telemetry/opentelemetry-log-collection/issues/190.