Keep getting ["error reading server preface: http2: frame too large"] with prometheus_remote_write output to thanos receiver

[Z4ck404]

Bug Report

Describe the bug

I am using fluenbit as sidecar to scrap local pods metrics, tag them and send them to a thanos receiver instance that will output them as tdsb block in cloud block storage. I am using the prometheus_remote_write output plugin to send send the metrics to the thanos receiving that in exposes a remote_write endpoint but I keep getting the error rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

To Reproduce

• Steps to reproduce the problem:

the pipeline configuration

    [INPUT]
      name                    prometheus_scrape
      host                    0.0.0.0
      port                    8080
      metrics_path            /metrics
      scrape_interval         5s

    [OUTPUT]
      Name                 prometheus_remote_write
      Host                 thanos-receiver-svc.logging.svc.cluster.local
      Port                 19291
      URI                  /api/v1/receive

fluentbitsidecar configuration:

  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Log_Level    info
        Parsers_File parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

Your Environment

• fluentbit version: 2.0.10
• thanos version: v0.34.1

[patrick-stephens]

Can you step up to the latest version (3.0.0) and confirm it still happens there?

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.