We are trying write the metrics from fluent bit to prometheus end point using prometheus_remote_write. We are seeing connection broken errors.
{3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] http_post result FLB_RETRY
3:00:26] [error] [output:prometheus_remote_write:prometheus_remote_write.1] ...:9090, HTTP status=0
3:00:26] [error] [http_client] broken connection to ...:9090 ?
3:00:26] [debug] [http_client] not using http_proxy for header
3:00:26] [debug] [upstream] KA connection #108 to ...:9090 is connected
3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] final payload size: 44771
3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] cmetric_id=0 decoded 0-65532 payload_size=44771
3:00:26] [debug] [task] destroy task=0x7ef84d83af60 (task_id=2)
3:00:26] [debug] [out flush] cb_destroy coro_id=8
3:00:26] [debug] [upstream] KA connection #106 to ...:443 is now available
3:00:26] [debug] [upstream] KA connection #106 to...:443 has been assigned (recycled)
3:00:26] [debug] [output:forward:forward.0] send options records=3 chunk=''
3:00:26] [debug] [output:forward:forward.0] request 1825 bytes to flush
3:00:26] [debug] [output:forward:forward.0] task_id=2 assigned to thread #0
3:00:26] [debug] [task] created task=0x7ef84d83af60 id=2 OK
3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] cmetrics msgpack size: 65532
3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] task_id=1 assigned to thread #0}
**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: Fluent Bit v3.0.6
* Configuration:
[INPUT]
Name prometheus_scrape
host 0.0.0.0
port 8080
Tag issue-store
metrics_path /api/metrics
scrape_interval 10s
[OUTPUT]
name prometheus_remote_write
host *.*.*.*
Port 9090
match issue-store
URI /api/v1/write
log_response_payload true
* Environment name and version (e.g. Kubernetes? What version?):
* Server type and version:
* Operating System and version:Linux
*PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
* Filters and plugins:
**Additional context**
We are not able to write metrics to prometheus.
We are trying write the metrics from fluent bit to prometheus end point using prometheus_remote_write. We are seeing connection broken errors.
{3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] http_post result FLB_RETRY 3:00:26] [error] [output:prometheus_remote_write:prometheus_remote_write.1] ...:9090, HTTP status=0 3:00:26] [error] [http_client] broken connection to ...:9090 ? 3:00:26] [debug] [http_client] not using http_proxy for header 3:00:26] [debug] [upstream] KA connection #108 to ...:9090 is connected 3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] final payload size: 44771 3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] cmetric_id=0 decoded 0-65532 payload_size=44771 3:00:26] [debug] [task] destroy task=0x7ef84d83af60 (task_id=2) 3:00:26] [debug] [out flush] cb_destroy coro_id=8 3:00:26] [debug] [upstream] KA connection #106 to ...:443 is now available 3:00:26] [debug] [upstream] KA connection #106 to...:443 has been assigned (recycled) 3:00:26] [debug] [output:forward:forward.0] send options records=3 chunk='' 3:00:26] [debug] [output:forward:forward.0] request 1825 bytes to flush 3:00:26] [debug] [output:forward:forward.0] task_id=2 assigned to thread #0 3:00:26] [debug] [task] created task=0x7ef84d83af60 id=2 OK 3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] cmetrics msgpack size: 65532 3:00:26] [debug] [output:prometheus_remote_write:prometheus_remote_write.1] task_id=1 assigned to thread #0}