Improved otel_metrics_source documentation

[sbayer55]

Is your feature request related to a problem? Please describe. In Data Prepper v1.5.1 the otel_metrics_source cannot be used with the string_converter plugin.

Describe the solution you'd like Documentation on plugin/feature compatibility.

Describe alternatives you've considered (Optional) Enhance pipeline validation on startup to detect incomatibility issues.

Additional context

Related community forum question

pipelines.yaml

entry-pipeline:
  source:
    otel_metrics_source:
      ssl: false
  sink:
    - stdout:

otel-collector-config.yaml

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'data-prepper'
          scrape_interval: 10s
          metrics_path: "/metrics/prometheus"
          static_configs:
            - targets: ['host.docker.internal:4900']

exporters:
  otlp/2:
    endpoint: data-prepper:21891
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp/2]

prometheus.yaml

docker-compose.yaml

version: "3.9"

services:
  data-prepper:
    restart: unless-stopped
    image: opensearch-data-prepper:2.0.0-SNAPSHOT
    working_dir: /usr/share/data-prepper/
    volumes:
      - ./data-prepper/config/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml
      - ./data-prepper/config/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml
      - ../../shared-config/log4j2.properties:/usr/share/data-prepper/config/log4j.properties
    ports:
      - "2021:2021"
  otel-collector:
    restart: unless-stopped
    image: otel/opentelemetry-collector:0.40.0
    command: ["--config=/etc/otel-collector-config.yml"]
    volumes:
      - ./opentelemetry-collector/otel-collector-config.yml:/etc/otel-collector-config.yml
    depends_on:
      - data-prepper
    ports:
      - "4317:4317"
  prometheus:
    restart: unless-stopped
    image: prom/prometheus
    volumes:
      - ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml
    ports:
      - "9000:9000"
    depends_on:
      - otel-collector

I recommend using the smoke-tests docker-compose.yaml as a base for setup.

[anand3493]

@sbayer55 the copy_values processor also facing the same problem. Do you have workarounds for creating a custom field and assign value from existing field?