search

FortDigital / covid-19

Collection script and Grafana Dashboard for tracking Progress of Covid-19
MIT License
18 stars 12 forks source link

Data retention? #13

Open rechena opened 4 years ago

rechena commented 4 years ago

I think this is more a question then an issue. But I've recently changed the retention on my influxDB. With this covid-19 script stopped working:

Traceback (most recent call last):
  File "/home/pi/covid-19/covid19.py", line 77, in <module>
    client.write_points(measurements)
  File "/home/pi/.local/lib/python3.7/site-packages/influxdb/client.py", line 599, in write_points
    consistency=consistency)
  File "/home/pi/.local/lib/python3.7/site-packages/influxdb/client.py", line 676, in _write_points
    protocol=protocol
  File "/home/pi/.local/lib/python3.7/site-packages/influxdb/client.py", line 410, in write
    headers=headers
  File "/home/pi/.local/lib/python3.7/site-packages/influxdb/client.py", line 369, in request
    raise InfluxDBClientError(err_msg, response.status_code)
influxdb.exceptions.InfluxDBClientError: 400: {"error":"partial write: points beyond retention policy dropped=40584"}

Looking briefly at the code it seems that everyday you pull all the data and rewrite the db, is this statement correct? Would it be possible to have a "incremental" or too much hassle?

Thanks

FortDigital commented 4 years ago

Hey,

Yeah when i wrote the script the datasources were changing so much that it was easier to just drop and recreate every time than attempt anything incremental.

I've not made any changes to this in quiet some time but if i get a chance ill take a look at adding an Incremental vs drop+Recreate switch to the script.

In terms of your error, partial writes are 'ok' in the sense that its just dropping the points that are past the retention period, probably just needs a try catch in there to allow for the error it returns.

Regards