Data Prepper - Logs - HTTP Source

[yujie-zhou]

Hi,

I was trying to create a log ingestion flow from OpenTelemetry to OpenSearch via DataPrepper.

So far, I've only seen otel_metrics_sourceand otel_trace_sourcein the DataPrepper documentation, is there a source that supports otlpexporter of OpenTelemetry Collector?

I went with using http source in DataPrepper in the end hoping it would work with otlphttp exporter, but so far I'm getting this error:

2023-01-13 14:14:10 2023-01-13T19:14:10.672Z    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 1}
2023-01-13 14:14:10 2023-01-13T19:14:10.672Z    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"data-prepper:2021/v1/logs\": unsupported protocol scheme \"data-prepper\"", "interval": "3.465315768s"}
2023-01-13 14:14:25 2023-01-13T19:14:25.537Z    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"data-prepper:2021/v1/logs\": unsupported protocol scheme \"data-prepper\"", "interval": "14.167450749s"}
2023-01-13 14:14:39 2023-01-13T19:14:39.707Z    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"data-prepper:2021/v1/logs\": unsupported protocol scheme \"data-prepper\"", "interval": "29.737572211s"}

Below are my otel collector config and data prepper pipeline:

Otel Collector Config:

receivers:
  filelog:
    include: [ /etc/output/logs.log]

exporters:
  logging:
  otlphttp:
    endpoint: data-prepper:2021
    tls:
      insecure: true
      insecure_skip_verify: true

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [logging, otlphttp]

Data Prepper Pipeline:

logs-pipeline:
  source:
    http:
      ssl: false
      authentication:
        unauthenticated:
  processor:
    - grok:
        match:
          log: [ "%{COMMONAPACHELOG}" ]
  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        insecure: true
        username: admin
        password: admin
        index: apache_logs

I've tried changing log.log to log.json but I'm still getting the error. I'm putting everything in docker compose and use docker to run all my containers.

Can someone please give me some information and point me to the right direction? Thank you very much for the help!

[dlvenable]

@yujie-zhou ,

It sounds like you are looking for an OTel Logs ingestion pipeline. We will have that feature in Data Prepper 2.1 thanks to #1372. We are currently planning to release on Feb 7. But, feel free to try running from the latest source to see how this works out for you.

[milkfrog]

Good evening everyone!

I'm also waiting for this feature and I'm wondering when it is going to be available, I saw that the last tag on its docker hub is from 4 months ago.

Is there a new prevision on this?

Thank you so much!

[dlvenable]

Hello @milkfrog , we are planning to release 2.1 on Mar 2.