Remote write to Prometheus Server => No metrics

[P4R4DiSi4C]

Bug Report

Describe the bug Using node_exporter_metrics input with prometheus_remote_write output doesn't push metrics to Prometheus.

To Reproduce

• Launch prometheus server with following configuration systemd config

  ExecStart=/usr/local/bin/prometheus \
  --storage.tsdb.path=/srv/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=0 \
  --web.config.file=/etc/prometheus/web.yml \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.console.templates=/etc/prometheus/consoles \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url= \
  --web.enable-remote-write-receiver \
  --web.enable-admin-api \
  --config.file=/etc/prometheus/prometheus.yml

• Launch fluentbit with following configuration:

  
  [SERVICE]
  Flush                1
  Log_level            debug

[INPUT] Name node_exporter_metrics Tag node_exporter Scrape_interval 2

[OUTPUT] name prometheus_remote_write match * host xxxxxxx.yyyyy.zzzz port 9090 uri /api/v1/write log_response_payload true tls off tls.verify off

- Example log message if applicable:

Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] created task=0x7f210ba365a0 id=0 OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] task_id=0 assigned to thread #1 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetrics msgpack size: 148413 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=0 decoded 0-148413 payload_size=150> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] final payload size: 150139 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 has been assigned (recycled) Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [http_client] not using http_proxy for header Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] xxxxxx.yyyyyy.zzzzz:9090,> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 is now available Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] http_post result FLB_OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [out flush] cb_destroy coro_id=86 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] destroy task=0x7f210ba365a0 (task_id=0) Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing package_throttle_count: /sys/devices> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted ...skipping... Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] created task=0x7f210ba365a0 id=0 OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] task_id=0 assigned to thread #1 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetrics msgpack size: 148413 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=0 decoded 0-148413 payload_size=150> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] final payload size: 150139 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 has been assigned (recycled) Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [http_client] not using http_proxy for header Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] xxxxxx.yyyyyy.zzzzz:9090,> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 is now available Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] http_post result FLB_OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [out flush] cb_destroy coro_id=86 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] destroy task=0x7f210ba365a0 (task_id=0) Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing package_throttle_count: /sys/devices> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted ...skipping... Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] created task=0x7f210ba365a0 id=0 OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] task_id=0 assigned to thread #1 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetrics msgpack size: 148413 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=0 decoded 0-148413 payload_size=150> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] final payload size: 150139 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 has been assigned (recycled) Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [http_client] not using http_proxy for header Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] xxxxxx.yyyyyy.zzzzz:9090,> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 is now available Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] http_post result FLB_OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [out flush] cb_destroy coro_id=86 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] destroy task=0x7f210ba365a0 (task_id=0) Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing package_throttle_count: /sys/devices> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted ...skipping... Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] created task=0x7f210ba365a0 id=0 OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] task_id=0 assigned to thread #1 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetrics msgpack size: 148413 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=0 decoded 0-148413 payload_size=150> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] final payload size: 150139 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 has been assigned (recycled) Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [http_client] not using http_proxy for header Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] xxxxxx.yyyyyy.zzzzz:9090,> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 is now available Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] http_post result FLB_OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [out flush] cb_destroy coro_id=86 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] destroy task=0x7f210ba365a0 (task_id=0) Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing package_throttle_count: /sys/devices> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted ...skipping... Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:22 vmon01 fluent-bit[70707]: [2024/08/21 21:50:22] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] created task=0x7f210ba365a0 id=0 OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] task_id=0 assigned to thread #1 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetrics msgpack size: 148413 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=0 decoded 0-148413 payload_size=150> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] final payload size: 150139 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 has been assigned (recycled) Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [http_client] not using http_proxy for header Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] xxxxxx.yyyyyy.zzzzz:9090,> Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [upstream] KA connection #77 to xxxxxx.yyyyyy.zzzzz:9090 is now available Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] http_post result FLB_OK Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [out flush] cb_destroy coro_id=86 Aug 21 21:50:23 vmon01 fluent-bit[70707]: [2024/08/21 21:50:23] [debug] [task] destroy task=0x7f210ba365a0 (task_id=0) Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing package_throttle_count: /sys/devices> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] CPU is missing core_throttle_count: /sys/devices/sy> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] cannot read info from: /class/thermal/thermal_zone[> Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop0 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop4 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop5 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop6 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: loop7 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda1 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda2 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] skip device: sda3 Aug 21 21:50:24 vmon01 fluent-bit[70707]: [2024/08/21 21:50:24] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] NVMe storage is not mounted

**Expected behavior**
Expect to see node-exporter metrics on the Prometheus Server as the following: (none of those below come from fluentbit, as the instance matches my other clients pushing with remote_write through a Prometheus AGENT)
<img width="1412" alt="image" src="https://github.com/user-attachments/assets/c2707efd-5bed-4270-afde-696d487d687a">

**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: [fluent bit] version=3.1.6, commit=, pid=68819 and prometheus 2.54.0
* Environment name and version: Fluentbit on a VM in VMware ESXi 8.0 and a bare-metal machine with prometheus
* Operating System and version: Ubuntu 22.04.4 LTS

**Additional context**
I'm looking to replace my current setup which has clients pushing node-exporter's metrics to a Prometheus Server through a Prometheus Agent.
Using Fluentbit with remote write would allow me to have an all-in-one tool that I could install on my clients to not only push metrics to Prometheus but logs to ES as well.

[edsiper]

thanks for reporting this. we will troubleshoot the issue.

[cosmo0920]

I checked with the current master and prometheus 2.54.0. Prometheus 2.54.0 doesn't respond node_os_info and node_os_version in their queries. Prometheus 2.47.2 does respond them. Not sure how make the difference.

For Linux and other Unix like systems, we can use node_uname_info in Prometheus 2.54.0 with prometheus_remote_write output plugin.

[P4R4DiSi4C]

@cosmo0920 Thanks for having a look, the machines showing in the above screenshot are running Prom 2.54.0 in agent mode with Node-Exporter goversion="go1.22.3" version="1.8.1"

Maybe a difference in the node-exporter version / implementation since metrics exported are not all here ? If I install node-exporter in same VM as fluent-bit, I can see it's exporting node_os_info and node_os_version but missing on fluent-bit, same thing for node_pressure... Grafana Dashboard for node-exporter seems to be using these metrics (node_pressure_cpu_waiting_seconds_total, node_pressure_memory_waiting_seconds_total, node_pressure_io_waiting_seconds_total...)

I exported fluent-bit's node-exporter to see which metrics were being exposed and was indeed able to find those in my central prometheus so remote-write is 100% confirmed to be properly working but some metrics are missing

This bring another question to the table to make, how / would it be possible for fluent-bit to append similar "default" labels as Prometheus like:

• Environment
• Instance
• Job

Grafana's node-exporter dashboard uses "Instance" and "Job" so for fluent-bit to be/become a proper drop-in replacement to Prometheus Agent + Node-Exporter, it would be awesome to add those, what you think ?

[cosmo0920]

This bring another question to the table to make, how / would it be possible for fluent-bit to append similar "default" labels as Prometheus like:

* Environment

* Instance

* Job

How about using add_label to attach these metadata?

[OUTPUT]
    name                 prometheus_remote_write
    match                *
    host                 xxxxxxx.yyyyy.zzzz
    port                 9090
    uri                  /api/v1/write
    log_response_payload true
    tls                  off
    tls.verify           off
    add_label instance ${HOSTNAME}
    add_label job fluent-bit

[P4R4DiSi4C]

Yep that does the trick ! I can live with that :) Can we maybe add "add_label" to the documentation for "prometheus_remote_write" ? https://docs.fluentbit.io/manual/pipeline/inputs/prometheus-remote-write

For the missing metrics VS node-exporter, anything you thinking of ?