plan how data will be stored over time

[joshua-8]

The website should be able to get relatively up to date data (maybe even up to the minute), along with graphs over the past day or two, and longer term graphs over weeks and months. It would be useful to log quite frequent data somewhere that we could analyze (this issue is related to issues #8 and #9 ). Long term data shouldn't be only in RAM in case power is lost.

If we can get an sd card working, that would be great since it would provide lots of read and writable space that lasts even if the esp32 is reset. Sending data to a google sheets would be a great way to log data in a way that's more accessible to us than a card inside the station, but might not be as reliable of a place to keep data the esp32 might want to read back and work with.

[joshua-8]

we should probably log at least this data, though power could be split into voltage and current: battery voltage/capacity, solar panel power, inverter+system power draw

[joshua-8]

Here are my thoughts on a way to handle the data:

Every minute --- read data, update JSON for website, add data to daily buffers, log data to sd and/or google sheets there are 1440 minutes in a day, so even storing six 4byte floats every minute would only take 10% of an esp32's 328KB RAM

Every night --- calculate overall values for the day, add that to longer term buffers, log to sd and/or google sheets, optionally, clear daily buffers

Every month --- compile some overall values, maybe switch to new files on the sd card so they don't get too big, email report of overall values and data log?

Possibly keep backups of buffers to reload data if the esp32 gets reset (this might be easiest with additional sd card files)

[joshua-8]

The ideas above may need to be changed quite a bit now that the database is a SQLite database on a server

[joshua-8]

saved in a firebase realtime database, but still a similar plan