OTLP HTTP logs endpoint with compression seems broken

Describe the bug OTLP HTTP logs endpoint with compression (gzip/zstd) gives an HTTP error 400. Unfortunately, Alloy doesn't display error messages like Quickwit does. However, I encountered the same behavior with a Python script, and I was able to see the error message. If I disable compression it works.

alloy logs

Jul 15 18:53:22 dhcp1 alloy[72606]: ts=2024-07-15T16:53:22.63928085Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.log error="not retryable error: Permanent error: rpc error: code = InvalidArgument desc = error exporting items, request to https://demo.quickwit.fr/api/v1/otlp/v1/logs responded with HTTP Status Code 400" dropped_items=1
Jul 15 18:53:22 dhcp1 alloy[72606]: ts=2024-07-15T16:53:22.639310127Z level=debug msg="Preparing to make HTTP request" component_path=/ component_id=otelcol.exporter.otlphttp.log url=https://demo.quickwit.fr/api/v1/otlp/v1/logs

python logs

INFO:root:Starting up
INFO:root:TEST LOG
ERROR:opentelemetry.exporter.otlp.proto.http._log_exporter:Failed to export logs batch code: 400, reason: {
  "message": "invalid OTLP request: failed to decode Protobuf message: invalid wire type value: 7"
}

Steps to reproduce (if applicable) Steps to reproduce the behavior:

Use a Python script (same issue, works only without compression).
Start Grafana Alloy with the OTLP HTTP endpoint and set compression (gzip/zstd, it doesn't matter which one).

Expected behavior A clear and concise description of what you expected to happen.

Configuration:

Using python script Install python requirement pip install opentelemetry-exporter-otlp-proto-http and launch the below script with REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt OTEL_EXPORTER_OTLP_LOGS_COMPRESSION=gzip python script.py. OTEL_EXPORTER_OTLP_LOGS_COMPRESSION can be set to none

Python script

# Support for instrumentation
import logging

# Instrumentation Libraries
from opentelemetry.sdk.resources import Resource
from opentelemetry.semconv.resource import ResourceAttributes

# Exporters
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter

# Logs Import
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry._logs import set_logger_provider

def initialize_otel(endpoint: str):
    logging.debug("Initialize OTEL")

    # Global resources
    resource = Resource.create(
        attributes={ResourceAttributes.SERVICE_NAME: 'debug_compression'})

    # Logging

    logger_provider = LoggerProvider(resource=resource)
    set_logger_provider(logger_provider)
    logger_provider.add_log_record_processor(
        BatchLogRecordProcessor(OTLPLogExporter(
            endpoint=endpoint)
        )
    )
    handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)

    # Use the OTLP logging handler to send logs.
    logging.getLogger().addHandler(handler)

    # Return the created providers
    return logger_provider

# Set up our own logging
logging.basicConfig(level="INFO")
logging.info("Starting up")

# Set up the OTEL agent to talk to the collector
LOG_PROVIDER = initialize_otel("https://demo.quickwit.fr/api/v1/otlp/v1/logs")

logging.info('TEST LOG')

Using Grafana alloy, disabling compression by setting compression = "none" works.

Alloy config

otelcol.exporter.otlphttp "log" {
        client {
                endpoint = "https://demo.quickwit.fr"
                compression = "gzip"
                tls {
                   insecure_skip_verify = true
                }
        }
        logs_endpoint = "https://demo.quickwit.fr/api/v1/otlp/v1/logs"
}

loki.source.journal "read" {
        forward_to = [
                otelcol.receiver.loki.converter.receiver,
        ]
        labels = {
                "job" = "journal",
        }
}

otelcol.receiver.loki "converter" {
        output {
                logs = [otelcol.exporter.otlphttp.log.input]
        }
}

Output of quickwit --version : Quickwit 0.8.1 (x86_64-unknown-linux-gnu 2024-03-29T14:09:41Z e6c5396)
The index_config.yaml : I use the default index configuration for OTLP

Quickwit config

[
  {
    "version": "0.8",
    "index_uid": "otel-logs-v0_7:01J1QRZ6BBB0N0KTKGQ1N6BY0X",
    "index_config": {
      "version": "0.8",
      "index_id": "otel-logs-v0_7",
      "index_uri": "s3://quickwit/indexes/otel-logs-v0_7",
      "doc_mapping": {
        "field_mappings": [
          {
            "name": "timestamp_nanos",
            "type": "datetime",
            "fast": true,
            "fast_precision": "milliseconds",
            "indexed": false,
            "input_formats": [
              "unix_timestamp"
            ],
            "output_format": "unix_timestamp_nanos",
            "stored": true
          },
          {
            "name": "observed_timestamp_nanos",
            "type": "datetime",
            "fast": false,
            "fast_precision": "seconds",
            "indexed": true,
            "input_formats": [
              "unix_timestamp"
            ],
            "output_format": "unix_timestamp_nanos",
            "stored": true
          },
          {
            "name": "service_name",
            "type": "text",
            "fast": {
              "normalizer": "raw"
            },
            "fieldnorms": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "severity_text",
            "type": "text",
            "fast": {
              "normalizer": "raw"
            },
            "fieldnorms": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "severity_number",
            "type": "u64",
            "coerce": true,
            "fast": true,
            "indexed": true,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "body",
            "type": "json",
            "expand_dots": true,
            "fast": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "default"
          },
          {
            "name": "attributes",
            "type": "json",
            "expand_dots": true,
            "fast": {
              "normalizer": "raw"
            },
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "dropped_attributes_count",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "trace_id",
            "type": "bytes",
            "fast": false,
            "indexed": true,
            "input_format": "hex",
            "output_format": "hex",
            "stored": true
          },
          {
            "name": "span_id",
            "type": "bytes",
            "fast": false,
            "indexed": true,
            "input_format": "hex",
            "output_format": "hex",
            "stored": true
          },
          {
            "name": "trace_flags",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "resource_attributes",
            "type": "json",
            "expand_dots": true,
            "fast": {
              "normalizer": "raw"
            },
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "resource_dropped_attributes_count",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "scope_name",
            "type": "text",
            "fast": false,
            "fieldnorms": false,
            "indexed": false,
            "stored": true
          },
          {
            "name": "scope_version",
            "type": "text",
            "fast": false,
            "fieldnorms": false,
            "indexed": false,
            "stored": true
          },
          {
            "name": "scope_attributes",
            "type": "json",
            "expand_dots": true,
            "fast": false,
            "indexed": false,
            "stored": true
          },
          {
            "name": "scope_dropped_attributes_count",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          }
        ],
        "tag_fields": [],
        "store_source": false,
        "index_field_presence": false,
        "timestamp_field": "timestamp_nanos",
        "mode": "strict",
        "max_num_partitions": 200,
        "tokenizers": []
      },
      "indexing_settings": {
        "commit_timeout_secs": 5,
        "docstore_compression_level": 8,
        "docstore_blocksize": 1000000,
        "split_num_docs_target": 10000000,
        "merge_policy": {
          "type": "stable_log",
          "min_level_num_docs": 100000,
          "merge_factor": 10,
          "max_merge_factor": 12,
          "maturation_period": "2days"
        },
        "resources": {
          "heap_size": "2.0 GB"
        }
      },
      "search_settings": {
        "default_search_fields": [
          "body.message"
        ]
      },
      "retention": null
    },
    "checkpoint": {
      "_ingest-api-source": {
        "ingest_partition_01J1QRZ4BW3XQ2RBDHFGEMRCME": "00000000000001705167",
        "ingest_partition_01J1RWEC95X1ZXYF9ZTT22HT9Z": "00000000000034089975",
        "ingest_partition_01J2K4H23VR2H0FKGXQVWYT5V8": "00000000000008372206",
        "ingest_partition_01J2SQ0TS33Y19SCRTNMZEW2P4": "00000000000003904812"
      },
      "_ingest-cli-source": {},
      "_ingest-source": {}
    },
    "create_timestamp": 1719858600,
    "sources": [
      {
        "version": "0.8",
        "source_id": "_ingest-source",
        "num_pipelines": 1,
        "enabled": false,
        "source_type": "ingest",
        "input_format": "json"
      },
      {
        "version": "0.8",
        "source_id": "_ingest-cli-source",
        "num_pipelines": 1,
        "enabled": true,
        "source_type": "ingest-cli",
        "input_format": "json"
      },
      {
        "version": "0.8",
        "source_id": "_ingest-api-source",
        "num_pipelines": 1,
        "enabled": true,
        "source_type": "ingest-api",
        "input_format": "json"
      }
    ]
  },
  {
    "version": "0.8",
    "index_uid": "otel-traces-v0_7:01J1QRZ6PJ9556XB15YNZMPV7T",
    "index_config": {
      "version": "0.8",
      "index_id": "otel-traces-v0_7",
      "index_uri": "s3://quickwit/indexes/otel-traces-v0_7",
      "doc_mapping": {
        "field_mappings": [
          {
            "name": "trace_id",
            "type": "bytes",
            "fast": true,
            "indexed": true,
            "input_format": "hex",
            "output_format": "hex",
            "stored": true
          },
          {
            "name": "trace_state",
            "type": "text",
            "fast": false,
            "fieldnorms": false,
            "indexed": false,
            "stored": true
          },
          {
            "name": "service_name",
            "type": "text",
            "fast": {
              "normalizer": "raw"
            },
            "fieldnorms": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "resource_attributes",
            "type": "json",
            "expand_dots": true,
            "fast": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "resource_dropped_attributes_count",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "scope_name",
            "type": "text",
            "fast": false,
            "fieldnorms": false,
            "indexed": false,
            "stored": true
          },
          {
            "name": "scope_version",
            "type": "text",
            "fast": false,
            "fieldnorms": false,
            "indexed": false,
            "stored": true
          },
          {
            "name": "scope_attributes",
            "type": "json",
            "expand_dots": true,
            "fast": false,
            "indexed": false,
            "stored": true
          },
          {
            "name": "scope_dropped_attributes_count",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "span_id",
            "type": "bytes",
            "fast": false,
            "indexed": true,
            "input_format": "hex",
            "output_format": "hex",
            "stored": true
          },
          {
            "name": "span_kind",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": true,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "span_name",
            "type": "text",
            "fast": {
              "normalizer": "raw"
            },
            "fieldnorms": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "span_fingerprint",
            "type": "text",
            "fast": false,
            "fieldnorms": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "span_start_timestamp_nanos",
            "type": "datetime",
            "fast": true,
            "fast_precision": "milliseconds",
            "indexed": false,
            "input_formats": [
              "unix_timestamp"
            ],
            "output_format": "unix_timestamp_nanos",
            "stored": true
          },
          {
            "name": "span_end_timestamp_nanos",
            "type": "datetime",
            "fast": false,
            "fast_precision": "seconds",
            "indexed": false,
            "input_formats": [
              "unix_timestamp"
            ],
            "output_format": "unix_timestamp_nanos",
            "stored": true
          },
          {
            "name": "span_duration_millis",
            "type": "u64",
            "coerce": true,
            "fast": true,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "span_attributes",
            "type": "json",
            "expand_dots": true,
            "fast": {
              "normalizer": "raw"
            },
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "span_dropped_attributes_count",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "span_dropped_events_count",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "span_dropped_links_count",
            "type": "u64",
            "coerce": true,
            "fast": false,
            "indexed": false,
            "output_format": "number",
            "stored": true
          },
          {
            "name": "span_status",
            "type": "json",
            "expand_dots": true,
            "fast": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "parent_span_id",
            "type": "bytes",
            "fast": false,
            "indexed": false,
            "input_format": "hex",
            "output_format": "hex",
            "stored": true
          },
          {
            "name": "events",
            "type": "array<json>",
            "expand_dots": true,
            "fast": {
              "normalizer": "raw"
            },
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          },
          {
            "name": "event_names",
            "type": "array<text>",
            "fast": false,
            "fieldnorms": false,
            "indexed": true,
            "record": "position",
            "stored": false,
            "tokenizer": "default"
          },
          {
            "name": "links",
            "type": "array<json>",
            "expand_dots": true,
            "fast": false,
            "indexed": true,
            "record": "basic",
            "stored": true,
            "tokenizer": "raw"
          }
        ],
        "tag_fields": [],
        "store_source": false,
        "index_field_presence": false,
        "timestamp_field": "span_start_timestamp_nanos",
        "mode": "strict",
        "max_num_partitions": 200,
        "tokenizers": []
      },
      "indexing_settings": {
        "commit_timeout_secs": 5,
        "docstore_compression_level": 8,
        "docstore_blocksize": 1000000,
        "split_num_docs_target": 10000000,
        "merge_policy": {
          "type": "stable_log",
          "min_level_num_docs": 100000,
          "merge_factor": 10,
          "max_merge_factor": 12,
          "maturation_period": "2days"
        },
        "resources": {
          "heap_size": "2.0 GB"
        }
      },
      "search_settings": {
        "default_search_fields": [
          "service_name",
          "span_name",
          "event_names"
        ]
      },
      "retention": null
    },
    "checkpoint": {
      "_ingest-api-source": {
        "ingest_partition_01J2SQ0TS33Y19SCRTNMZEW2P4": "00000000000000000011"
      },
      "_ingest-cli-source": {},
      "_ingest-source": {}
    },
    "create_timestamp": 1719858600,
    "sources": [
      {
        "version": "0.8",
        "source_id": "_ingest-cli-source",
        "num_pipelines": 1,
        "enabled": true,
        "source_type": "ingest-cli",
        "input_format": "json"
      },
      {
        "version": "0.8",
        "source_id": "_ingest-source",
        "num_pipelines": 1,
        "enabled": false,
        "source_type": "ingest",
        "input_format": "json"
      },
      {
        "version": "0.8",
        "source_id": "_ingest-api-source",
        "num_pipelines": 1,
        "enabled": true,
        "source_type": "ingest-api",
        "input_format": "json"
      }
    ]
  }
]

quickwit-oss / quickwit

OTLP HTTP logs endpoint with compression seems broken #5223