search

anclrii / Storj-Exporter

Prometheus exporter for monitoring Storj storage nodes
GNU General Public License v3.0
58 stars 19 forks source link

storageDaily null at start of month instead of 0 #38

Closed kevinkk525 closed 3 years ago

kevinkk525 commented 3 years ago

I guess you'll already have noticed that too since the dashboard looks pretty empty now xD

grafik

I "fixed" it locally by try: except: data=None Imho though this is an API error and I'll raise it in the storagenode repo too. If the storagenode doesn't have data for the current day, it should show 0 and not None. Or do you disagree?


Traceback (most recent call last):

  File "/usr/local/lib/python3.7/wsgiref/handlers.py", line 137, in run

    self.result = application(self.environ, self.start_response)

  File "/usr/local/lib/python3.7/site-packages/prometheus_client/exposition.py", line 52, in prometheus_app

    status, header, output = _bake_output(registry, accept_header, params)

  File "/usr/local/lib/python3.7/site-packages/prometheus_client/exposition.py", line 40, in _bake_output

    output = encoder(registry)

  File "/usr/local/lib/python3.7/site-packages/prometheus_client/openmetrics/exposition.py", line 14, in generate_latest

    for metric in registry.collect():

  File "/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py", line 82, in collect

    for metric in collector.collect():

  File "./storj-exporter.py", line 187, in collect

    yield from self.add_sat_metrics()

  File "./storj-exporter.py", line 119, in add_sat_metrics

    yield from self.add_extended_sat_metrics(sat)

  File "./storj-exporter.py", line 166, in add_extended_sat_metrics

    data            = sat['sat_data'].get('storageDaily', [None])[-1]

TypeError: 'NoneType' object is not subscriptable
anclrii commented 3 years ago

Haven't seen this to be honest. Is this on a new node? I'll try to reproduce but it seems like I missed handling this one properly. It relies on get to return default [None] in case storageDaily is missing. Though [None] has no [-1] element and so fails.

I think this is what happens here and so data = sat['sat_data'].get('storageDaily', [{}])[-1] should fix it, need to test.

kevinkk525 commented 3 years ago

I'm surprised you missed it, it occured on Nov 1st but you can wait for Dec 1st, then it'll happen again. It happened on 3 nodes, all rather old.

anclrii commented 3 years ago

oh you are right - checking history I have no metrics for start of the month as well :) that makes sense, thanks for reporting it 👍

anclrii commented 3 years ago

Reproduced the issue and this should fix it https://github.com/anclrii/Storj-Exporter/commit/f85ea79e529a0dc558651c0a53769b54902afd53

Indeed api just returns {"storageDaily": null} in that case and so [-1] fails as the value is not a list. I don't think there's a better option then try: except: as there's no safe way of retrieving list items similar to dictionary get method.

@kevinkk525 if you are interested, most commits are now auto building docker images and there's anclrii/storj-exporter:dev image available to test.

kevinkk525 commented 3 years ago

That looks good to me! Should work, my solution was similar, but only for this specific metric. It's good that you use a solution for multiple metrics. I'll give it a try on Dec 1st!

anclrii commented 3 years ago

This is now merged into master and latest image