Route / Ingress for OTLP HTTP does not work

[pavolloffay]

The following collector CR

kubectl apply -f - <<EOF
kind: OpenTelemetryCollector
apiVersion: opentelemetry.io/v1alpha1
metadata:
  name: otel
spec:
  mode: deployment
  ingress: 
    type: route
    hostname: apps-crc.testing
    route:
      termination: insecure
  config: |

    receivers:
      otlp:
        protocols:
          http:
          grpc:

    exporters:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [logging]
EOF

creates this route:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  creationTimestamp: "2023-07-27T16:41:27Z"
  labels:
    app.kubernetes.io/instance: ploffay.otel
    app.kubernetes.io/managed-by: opentelemetry-operator
    app.kubernetes.io/name: otlp-http-otel-route
  name: otlp-http-otel-route
  namespace: ploffay
  ownerReferences:
  - apiVersion: opentelemetry.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: OpenTelemetryCollector
    name: otel
    uid: 036cc218-dc9e-4802-9403-c9b0985d193c
  resourceVersion: "313141"
  uid: d400e226-d756-45d3-97fa-7997e15ac262
spec:
  host: otlp-http.apps-crc.testing
  path: /otlp-http
  port:
    targetPort: otlp-http
  to:
    kind: Service
    name: otel-collector
    weight: 100
  wildcardPolicy: None

The path: /otlp-http is causing issues for setting OTEL_EXPORTER_OTLP_ENDPOINT (see https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md)

For instance:

docker run --rm -it --net=host ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.74.0 traces --otlp-http --otlp-endpoint=otlp-http.apps-crc.testing:80/otlp-http --otlp-insecure --duration 10s --rate 4                                                                                                   2 ↵ ploffay@fedora

2023-07-27T16:49:34.599Z    INFO    traces/traces.go:68 starting HTTP exporter
2023-07-27T16:49:34.599Z    INFO    traces/traces.go:126    generation of traces is limited {"per-second": 4}
2023/07/27 16:49:35 traces export: parse "http://otlp-http.apps-crc.testing:80%2Fotlp-http/v1/traces": invalid port ":80%2Fotlp-http" after host
2023/07/27 16:49:36 traces export: parse "http://otlp-http.apps-crc.testing:80%2Fotlp-http/v1/traces": invalid port ":80%2Fotlp-http" after host

Removing the path from route and the --otlp-endpoint fixed the issue.

[pavolloffay]

@frzifus do you remember why the collector is setting the path for the route?

[pavolloffay]

I was not able to report data to the OTLP HTTP endpoint via the route even from another OTEL collector:

The endpoint was set to http://otlp-http.apps-crc.testing/otlp-http or http://otlp-http.apps-crc.testing

 docker run --rm -it --net=host -v ${PWD}:/tmp ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.81.0 --config /tmp/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:

exporters:
  otlphttp:
    endpoint: http://otlp-http.apps-crc.testing/otlp-http
    tls:
      insecure: true

extensions:
  health_check:
  pprof:
  zpages:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [otlphttp]

After setting the path in the route to / the endpoint http://otlp-http.apps-crc.testing worked well.

[frzifus]

Same here, I wonder why the path is set. Let me verify this. It should be removed or there should be a rule to redirect.

[pavolloffay]

@frzifus I will unassign you I have started working on this

[pavolloffay]

I think this issue with path could be solved by URL rewrite annotation on the Route/Ingress object:

• https://docs.openshift.com/container-platform/4.11/networking/routes/route-configuration.html#nw-route-specific-annotations_route-configuration haproxy.router.openshift.io/rewrite-target:
• https://kubernetes.github.io/ingress-nginx/examples/rewrite/ nginx.ingress.kubernetes.io/rewrite-target

However, there are multiple ingress controllers and it does not seem reasonable to support all of them in the operator. The annotations can be as well specified in the collector CR, but it puts more configuration stress on users, they will have to understand which annotation to choose and how to rewrite the targets.

Following remote collector config

  otlphttp:
    endpoint: http://apps-crc.testing:80/otlp-http
    tls:
      insecure: true

Produces logs

2023-08-02T15:31:53.241Z    error   exporterhelper/queued_retry.go:391  Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "Permanent error: error exporting items, request to http://apps-crc.testing:80/otlp-http/v1/traces responded with HTTP Status Code 404", "dropped_items": 2}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
    go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/traces.go:126
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
    go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
    go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/internal/bounded_memory_queue.go:47

[pavolloffay]

I have tried the same setup with Ingress and run into the same issues.