search

0glabs / 0g-storage-node

Apache License 2.0
92 stars 85 forks source link

Influxdb metrics pushing error #271

Open Hacker-web-Vi opened 1 week ago

Hacker-web-Vi commented 1 week ago

Hello! I'm trying to push zgs_node metrics to the influxdb bucket, but it doesn't work. Influxdb is up and running. I tried to upload some metrics manually using the same auth method as provided in the config below and it worked. There is no related error in the log.

zgs_node v0.7.4

#######################################################################
###                      Metrics Options                            ###
#######################################################################

[metrics]

# Whether to enable metrics.
enabled = true

# Interval to output metrics periodically, e.g. "10s", "30s" or "60s".
report_interval = "10s"

# File name to output metrics periodically.
file_report_output = "metrics.csv"

# Influxdb configurations to output metrics periodically.
influxdb_report_host = "http://168.119.10.134:8416"
influxdb_report_db = "0g-test"
influxdb_report_username = "0g-test"
influxdb_report_password = "test-0-0-0"

# Storage node name as a tag.
influxdb_report_node = "0g-test"

In trace logs I was able to find our influxdb host

0g-testnet-validator@trusted-point-1:~/.storage-node$ tail -f /home/0g-testnet-validator/.storage-node/log/zgs.log.$(TZ=UTC date +%Y-%m-%d) | grep 168.119.10.134
2024-11-14T19:47:17.073208Z TRACE hyper::client::pool: take? ("http", 168.119.10.134:8416): expiration = Some(90s)
2024-11-14T19:47:17.073216Z DEBUG hyper::client::pool: reuse idle connection for ("http", 168.119.10.134:8416)
2024-11-14T19:47:17.227662Z TRACE hyper::client::pool: put; add idle connection for ("http", 168.119.10.134:8416)
2024-11-14T19:47:17.227665Z DEBUG hyper::client::pool: pooling idle connection for ("http", 168.119.10.134:8416)
2024-11-14T19:47:31.735756Z TRACE hyper::client::pool: take? ("http", 168.119.10.134:8416): expiration = Some(90s)
2024-11-14T19:47:31.735774Z DEBUG hyper::client::pool: reuse idle connection for ("http", 168.119.10.134:8416)
2024-11-14T19:47:31.898813Z TRACE hyper::client::pool: put; add idle connection for ("http", 168.119.10.134:8416)
2024-11-14T19:47:31.898818Z DEBUG hyper::client::pool: pooling idle connection for ("http", 168.119.10.134:8416)
2024-11-14T19:47:39.209178Z TRACE hyper::client::pool: take? ("http", 168.119.10.134:8416): expiration = Some(90s)
boqiu commented 1 week ago

Basically, file_report_output is used for dev purpose, and need not to configure it.

As for reporting to influxdb, is there any stderr or logs? E.g.

Moreover, you could check the metrics via RPC:

curl -X POST --data '{"jsonrpc":"2.0","method":"admin_getMetrics","params":["metrics"],"id":1}' -H "Content-Type: application/json" http://127.0.0.1:5679 | jq -C ".result"

There is a metric metrics_report_failures records the number of failures to report metrics to influcdb.

boqiu commented 5 days ago

@Hacker-web-Vi Have your issue resolved?