fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.81k stars 1.58k forks source link

Seeing Broken Connection error for prometheus_remote_write #9145

Open abhi670329 opened 2 months ago

abhi670329 commented 2 months ago

We are trying write the metrics from fluent bit to prometheus end point using prometheus_remote_write. We are seeing connection broken errors.

{3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] http_post result FLB_RETRY 3:00:26] [error] [output:prometheus_remote_write:prometheus_remote_write.1] ...:9090, HTTP status=0 3:00:26] [error] [http_client] broken connection to ...:9090 ? 3:00:26] [debug] [http_client] not using http_proxy for header 3:00:26] [debug] [upstream] KA connection #108 to ...:9090 is connected 3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] final payload size: 44771 3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] cmetric_id=0 decoded 0-65532 payload_size=44771 3:00:26] [debug] [task] destroy task=0x7ef84d83af60 (task_id=2) 3:00:26] [debug] [out flush] cb_destroy coro_id=8 3:00:26] [debug] [upstream] KA connection #106 to ...:443 is now available 3:00:26] [debug] [upstream] KA connection #106 to...:443 has been assigned (recycled) 3:00:26] [debug] [output:forward:forward.0] send options records=3 chunk='' 3:00:26] [debug] [output:forward:forward.0] request 1825 bytes to flush 3:00:26] [debug] [output:forward:forward.0] task_id=2 assigned to thread #0 3:00:26] [debug] [task] created task=0x7ef84d83af60 id=2 OK 3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] cmetrics msgpack size: 65532 3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] task_id=1 assigned to thread #0}



**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: Fluent Bit v3.0.6
* Configuration: 

[INPUT]
    Name prometheus_scrape
    host 0.0.0.0
    port 8080
    Tag issue-store
    metrics_path /api/metrics
    scrape_interval 10s

[OUTPUT]
    name prometheus_remote_write
    host *.*.*.*
    Port 9090
    match issue-store
    URI /api/v1/write
    log_response_payload true

* Environment name and version (e.g. Kubernetes? What version?):
* Server type and version:
* Operating System and version:Linux
*PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
* Filters and plugins:

**Additional context**
We are not able to write metrics to prometheus.
edsiper commented 2 months ago

what is the configuration of your Prometheus service ? please provide a full reproducible case, on that way we can speed up troubleshooting