New component: `otlpjson` connector

ChrsMark commented 1 month ago

The purpose and use-cases of the new component

This new connector will be used in order to extract otlpjson from incoming Logs. The connector will be responsible for unpacking valid otlp json from Log records and specifically the Body field (this can be configurable).

This component will serve similarly to the otlpjsonfilereceiver but in a later stage in the pipeline. This is particularly useful when the otlpjson cannot be consumed directly by the aforementioned receiver. An example use-case can be found at https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/33846.

Example configuration for the component

Initially the configuration can be quite basic like in otlpjsonfilereceiver but we can expand later according to the needs.

connectors:
  otlpjson:

The connector can be used in a pipeline to extract Logs, Metrics and Traces like this:

receivers:
  filelog:
    include:
      - /var/log/foo.log

exporters:
  debug:

connectors:
  otlpjson:

service:
  pipelines:
    logs/raw:
      receivers: [filelog]
      exporters: [otlpjson]
    metrics/otlp:
      receivers: [otlpjson]
      exporters: [debug]
    logs/otlp:
      receivers: [otlpjson]
      exporters: [debug]
    traces/otlp:
      receivers: [otlpjson]
      exporters: [debug]

Telemetry data types supported

1) Logs to Logs 2) Logs to Metrics 3) Logs to Traces

Is this a vendor-specific component?

[ ] This is a vendor-specific component
[ ] If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
[ ] If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@djaglowski, @ChrsMark

Sponsor (optional)

@djaglowski (https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/33846#issuecomment-2228738273)

Additional context

1) Filing this issue after we shortly chatted about it with @zeitlinger on Slack. 2) A very basic PoC illustrating the idea can be found at https://github.com/ChrsMark/otlpjsonconnector.

infor7 commented 1 month ago

I'd love the idea. I have similar use case. In my company we are saving metrics into files with openTSDB format in PVC in kubernetes. Metrics are automatically picked up by sidecar which transfer it to our observability platform. OTLP metric example: put accessmanager.envoy.cluster.app-metrics-router-dev.zone.us-west1.us-west1-a.upstream_rq_time.P99 1721726558 93.2466666666666 direction=outbound port=4959 infra=mesh dc=us-west1-a namespace=accessmanager-dev pod_name=accessmanager-celery-flower-7f954746bf-qqmw8

I've tried to use stanza and build something like filelogreceiver but unfortunately that lib allows only to pick up the logs. If the otelcol lib will support metrics as well it should be pretty simple to write metric receiver with it.

ChrsMark commented 1 month ago

Tested that with https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.106.0. @djaglowski I'm not 100% sure about the process here but I assume we can close this issue since the initial implementation of the component has been shipped.

zeitlinger commented 1 month ago

thanks a lot for doing this!

open-telemetry / opentelemetry-collector-contrib