paritytech / polkadot-runtime-prom-exporter

Prometheus exporter for polkadot runtime metrics
10 stars 3 forks source link

Exporter crashes when node is still syncing #7

Open jam206 opened 2 years ago

jam206 commented 2 years ago

When node is still syncing the exporter can fail when the blocks are not fully known on the node.

[22-01-19 11:01:09] info: connected to chain kusama
[22-01-19 11:01:19] info: starting block metric scrape at #5351936
/usr/src/polkadot-prom-exporter/build/index.js:290
        weightMetric.set({ class: "mandatory" }, weight.mandatory.toNumber());
                                                                  ^

TypeError: Cannot read properties of undefined (reading 'toNumber')
    at /usr/src/polkadot-prom-exporter/build/index.js:290:67
    at Generator.next (<anonymous>)
    at fulfilled (/usr/src/polkadot-prom-exporter/build/index.js:24:58)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at runNextTicks (node:internal/process/task_queues:65:3)
    at processImmediate (node:internal/timers:437:9)

For example exporter is trying to scrape block #5351936 when node only knows up to block #5349269

2022-01-19 11:54:16.298  INFO main sc_service::builder: 📦 Highest known block at #5349269

This causes TypeError above, better error handling should be implemented, possibly waiting? So the pod does not end up in a crash loop.