search

alexnathanson / solar-protocol

A repository in development for a solar powered network of servers that host a distributed web platform. Project by Tega Brain, Alex Nathanson and Benedetta Piantella. Supported by Eyebeam, Mozilla, and CS&S
http://solarprotocol.net
220 stars 19 forks source link

How to know if missing data is due to an update #57

Closed Pixouls closed 1 year ago

Pixouls commented 1 year ago

I took note of several days when data for the Swarthmore server is missing that do not match the behavior of when a system has stopped taking data because the battery is running now (below 34%) and then comes back online when it is charged enough again (above 50% or so). Time is given in seconds based on unix time and converted to EST.

tracerData2023-02-02.csv System had downtime at the end of the day 64209.11098 (12:50:09 pm EST ) Battery percentage was 0.52

tracerData2023-02-26.csv System had downtime at the end of the day 65933.58273 (1:18:53 pm EST) Battery percentage was 0.57

tracerData2023-03-12.csv System had downtime during the day [3720.189992] (8:02:00 pm EST) These took place at, (array([59], dtype=int64),) Battery percentage was [0.38]

tracerData2023-03-18.csv System had downtime at the end of the day 57718.478581 (11:01:58 am EST) Battery percentage was 1.0

tracerData2023-03-23.csv System had downtime at the end of the day 37957.928276 (5:32:37 am EST) Battery percentage was 0.45

tracerData2023-04-05.csv System had downtime at the end of the day 50053.761868 (8:54:13 am EST) Battery percentage was 1.0

For all of these days, the data stops logging at some point during the day when it should still have plenty of battery, then data logging picks back up the next day's starting from midnight as if nothing happened.

2023-03-12 being an exception where the system goes offline but then comes back online within the same day.

2023-03-23 also only has one data point for that entire day (missing data from the morning even though the battery did not run out the previous day), which is really strange behavior.

Can we keep a public log on the github so we can keep track of dates where downtime is specifically related to an update/maintenance and compare them against our data files? I want to distinguish how much downtime is due to the battery discharging, due to an update/maintenance, or due to an error that needs to be debugged.

alexnathanson commented 1 year ago

I just completed an update of the csv_datalogger.py file, so recent issues may have been related to that. If the issue persists, we can look in to it more closely, but for the time being lets just see how this update runs. In general, its not practical for us to be logging maintenance activity manually, but generally maintenance would mean some minor update is being distributed via git so you could look at the history of when pulls occurred on your remote server to see if the timestamps are meaningful

Pixouls commented 1 year ago

I understand. Though as more people will be working with solar protocol data there may be others who encounter data missing without context that could be causing errors based on their assumptions in data processing and not know that the best thing to do for the time being is to look at the history, and what to do if it doesn't align with the history.