Create internal flash (LittleFS) logging functionality

timmbogner / Farm-Data-Relay-System

A system that uses ESP-NOW, LoRa, and other protocols to transport sensor data in remote areas without relying on WiFi.

MIT License

507 stars 114 forks source link

Create internal flash (LittleFS) logging functionality #45

Closed thefeiter closed 2 years ago

thefeiter commented 2 years ago

As discussed in #42 I implemented a possibility to log data to the internal SPIFFS Filesystem.

thefeiter commented 2 years ago

I have checked the date in the filesystem via a sketch. Seems to be working. But there is no convenience method to load the data onto your computer without uploading a different sketch.

A Http Fileserver maybe...

thefeiter commented 2 years ago

Works as expected. Retrieving the files shall be done sometime later.

timmbogner commented 2 years ago

Okedoke. A few thoughts as I contemplate this:

A major usefulness of this will be to automatically update the server with missed readings on reconnect. Especially since the filesystem isn't super easy to get files from, so let's leave it here for now. In the future there might be more we can do, like triggering the gateway to send the entire file.
I think the seconds as a back up isn't necessary. If it can't give us real timestamps then it shouldn't bother.
If I remember right, it's best to keep flash writes to a minimum, as rapid writes can corrupt the card. It might be better to buffer writes and make several at once. I'm not sure about this; its something to research. Will an SD card or flash hold up to a massive influx of individual readings? Have you tried running it against the Stress Test? I'm still looking for my ESP-CAM to test with.

thefeiter commented 2 years ago

A major usefulness of this will be to automatically update the server with missed readings on reconnect.

True

I think the seconds as a back up isn't necessary.

For the internal Flash, yes. Since one cannot easily read the files. But for the SD log it is still useful to be able to see not only in which order values came in but also how far apart time wise. And since I would keep the seconds since reset for the SD log I would also levae them in the Flash log.

its something to research

True

thefeiter commented 2 years ago

When logging to SD+FS with the stresstest only every 7th reading makes it to the sd card

When logging only to SD only every 2nd reading comes through

When logging only to FS every 8th reading makes it to the internal flash

Sooo, uhm yeah. Maybe buffering is a good idea. But I think the stresstest is going to extremes anyways, so for normal usage this should be ok for now. (for the already merged SD log)

timmbogner commented 2 years ago

I think the main thing that irks me is that the timestamp will be non-standardized. In the docs we'll have to say "it will look like this... except when it looks like this...". I think this functionality should assume the user has WiFi/internet and can get the time. If you're worried about power failures (losing both internet and rtc time) then implementing a battery backup is your option. I'll think more about this, but it's super-late and I'm going to sleep now. Thanks for all your help! Tomorrow I'm hoping to lay groundwork for the new packet confirmation feature, which is the first of the improvements Andreas called for in his video.

timmbogner commented 2 years ago

But I think the stresstest is going to extremes anyways, so for normal usage this should be ok for now.

True.

thefeiter commented 2 years ago

Now it is at least buffering the whole line to write. ~97% make it to the sd card with the stresstest. Writing to flash is now at 20%, this seems to be a weird bottleneck. I thought the internal FS would be the fastest...

thefeiter commented 2 years ago

So further buffering would be good. At this point I would need some help/do some reading how such a buffering is done best.

thefeiter commented 2 years ago

https://github.com/esp8266/Arduino/issues/7271

Devilbinder commented 2 years ago

Thanks for switching this to LittleFS. SPIFFS is a fragile headache of a file system.