anclrii / Storj-Exporter

Prometheus exporter for monitoring Storj storage nodes
GNU General Public License v3.0
58 stars 19 forks source link

storagenodes are returning None as the payout #57

Closed anclrii closed 3 years ago

anclrii commented 3 years ago

My storj-exporters seem to have blown up this morning:

File "/home/smg/git/Storj-Exporter/storj-exporter.py", line 109, in add_payout_metrics data = self.node_data.get('payout', {}).get('currentMonth', None)

Seems like my storagenodes are returning None as the payout. Adding an if statement keep the exporter running, but with missing payout data.

--- a/storj-exporter.py
+++ b/storj-exporter.py
@@ -105,13 +105,14 @@ class StorjCollector(object):
   def add_payout_metrics(self):
     if 'payout' in self.storj_collectors:
       self.get_node_payout_data()
-      metric_name     = 'storj_payout_currentMonth'
-      data            = self.node_data.get('payout', {}).get('currentMonth', None)
-      documentation   = 'Storj estimated payouts for current month'
-      keys            = ['egressBandwidth', 'egressBandwidthPayout', 'egressRepairAudit', 'egressRepairAuditPayout', 'diskSpace', 'diskSpacePayout', 'heldRate', 'payout', 'held']
-      labels          = ['type']
-      metric_family   = GaugeMetricFamily
-      yield from self.dict_to_metric(data, metric_name, documentation, metric_family, keys, labels)
+      if self.node_data.get('payout', {}):
+        metric_name     = 'storj_payout_currentMonth'
+        data            = self.node_data.get('payout', {}).get('currentMonth', None)
+        documentation   = 'Storj estimated payouts for current month'
+        keys            = ['egressBandwidth', 'egressBandwidthPayout', 'egressRepairAudit', 'egressRepairAuditPayout', 'diskSpace', 'diskSpacePayout', 'heldRate', 'payout', 'held']
+        labels          = ['type']
+        metric_family   = GaugeMetricFamily
+        yield from self.dict_to_metric(data, metric_name, documentation, metric_family, keys, labels)

   def add_sat_metrics(self):
sgoldthorpe commented 3 years ago

Looks like it's confirmed as a storagenode bug on the 1st day of the month with the latest version(s). The exporter should be robust enough to cope with this anyway in case the bug isn't fixed or is re-introduced in the future.

https://forum.storj.io/t/what-is-it-error-console-endpoint-failed-to-encod/11513/8

chris-sanders commented 3 years ago

This is happening for me too:

Traceback (most recent call last):
  File "./storj-exporter.py", line 235, in <module>
    REGISTRY.register(StorjCollector())
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 26, in register
    names = self._get_names(collector)
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 66, in _get_names
    for metric in desc_func():
  File "./storj-exporter.py", line 190, in collect
    yield from self.add_payout_metrics()
  File "./storj-exporter.py", line 109, in add_payout_metrics
    data            = self.node_data.get('payout', {}).get('currentMonth', None)
AttributeError: 'NoneType' object has no attribute 'get'

Deploying on Kubernetes this is causing the pod to go into a crash loop because the exporter never comes up.