The otlp/v1/logs API on the loki gateway returns 404.

[jaca-p]

Use Otel's automatic instrumentation.

The endpoint in mir and tempo works well, but the endpoint in loki returns 404. The endpoint address is correct.

Does anyone know why?

• instrumentation.yaml

  apiVersion: opentelemetry.io/v1alpha1
  kind: Instrumentation
  metadata:
  name: auto-instrumentation
  spec:
  exporter:
  endpoint: http://otel-collector-collector:4318
  propagators:
  - tracecontext
  - baggage
  - b3
  sampler:
  type: always_on
  java:
  image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest

• otel-collector yaml

  apiVersion: opentelemetry.io/v1alpha1
  kind: OpenTelemetryCollector
  metadata:
  name: otel-collector
  spec:
  mode: daemonset
  config: |
  receivers:
   otlp:
     protocols:
       grpc:
         endpoint: 0.0.0.0:4317
       http:
         endpoint: 0.0.0.0:4318
  processors:
   memory_limiter:
     check_interval: 1s
     limit_percentage: 75
     spike_limit_percentage: 15
   batch:
     send_batch_size: 10000
     timeout: 10s
  exporters:
   loki:
     endpoint: http://loki-loki-distributed-gateway.loki.svc/loki/api/v1/push # not use
   otlphttp/tempo:
     endpoint: http://tempo-distributor.tempo.svc:4318
   otlphttp/loki:
     endpoint: http://loki-loki-distributed-gateway.loki.svc:80/otlp
   otlphttp/mimir:
     endpoint: http://mimir-nginx.mimir.svc:80/otlp
     tls:
       insecure: true
   prometheusremotewrite/local:
     endpoint: http://mimir-nginx.mimir.svc:80/api/v1/push
     resource_to_telemetry_conversion:
       enabled: true
     tls:
       insecure: true
   debug:
  service:
   pipelines:
     traces:
       receivers: [otlp]
       processors: [memory_limiter, batch]
       exporters: [otlphttp/tempo, debug]
     logs:
       receivers: [otlp]
       processors: [memory_limiter, batch]
       exporters: [otlphttp/loki, debug]
     metrics:
       receivers: [otlp]
       processors: [memory_limiter, batch]
       exporters: [otlphttp/mimir, debug]

• otel-collector log

  2024-09-24T07:07:13.216Z info service@v0.107.0/service.go:221 Everything is ready. Begin running and processing data. 2024-09-24T07:08:53.407Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 16} 2024-09-24T07:08:53.407Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 6, "log records": 12} 2024-09-24T07:08:53.413Z error exporterhelper/queue_sender.go:92 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "otlphttp/loki", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = error exporting items, request to http://loki-loki-distributed-gateway.loki.svc:80/otlp/v1/logs responded with HTTP Status Code 404", "dropped_items": 12} go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1 go.opentelemetry.io/collector/exporter@v0.107.0/exporterhelper/queue_sender.go:92 go.opentelemetry.io/collector/exporter/internal/queue.(boundedMemoryQueue[...]).Consume go.opentelemetry.io/collector/exporter@v0.107.0/internal/queue/bounded_memory_queue.go:52 go.opentelemetry.io/collector/exporter/internal/queue.(Consumers[...]).Start.func1 go.opentelemetry.io/collector/exporter@v0.107.0/internal/queue/consumers.go:43 2024-09-24T07:09:03.408Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 56} 2024-09-24T07:09:43.222Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 23, "data points": 50}

• loki gateway log

  2024/09/24 07:08:53 [error] 11#11: *172 open() "/etc/nginx/html/otlp/v1/logs" failed (2: No such file or directory), client: 192.168.1.49, server: , request: "POST /otlp/v1/logs HTTP/1.1", host: "loki-loki-distributed-gateway.loki.svc:80"

[khanh1998]

I have the same issue with the latest version of Loki, I injected logs successfully via API /loki/api/v1/push, but when I tried the API /otlp/v1/logs it returned 404. I tried to curl the endpoint:

➜ curl -X POST "localhost:3100/otlp/v1/logs"
404 page not found

[khanh1998]

@jaca-p After I switched to image grafana/loki:3.2.0, it worked. When the API returned 404, I was use grafana/loki:latest.