search

timmbogner / Farm-Data-Relay-System

A system that uses ESP-NOW, LoRa, and other protocols to transport sensor data in remote areas without relying on WiFi.
MIT License
507 stars 114 forks source link

Add a logging function to write incoming Packets to SD card #42

Closed thefeiter closed 2 years ago

thefeiter commented 2 years ago

This came to my mind when my internet connection was interrupted and data was not sent to my mqtt broker.

SD cards work on SPI, so there has to be another Chip/slave select pin, but it is working good, since the Master does only talk to one slave at once (it is either using LoRa or the SD card)

For now I am just appending the JSON data to a hardcoded file whenever it is being sent over Serial.

This is meant to be able to retrieve sensor values when there is an Error with your internet connection or WiFi

TODO:

After Merge:

thefeiter commented 2 years ago

Any Ideas on getting some kind of timestamp without WiFi/NTP?

thefeiter commented 2 years ago

Whit this timestamp method the 1/10 seconds since reset are counted (based on millis()). This shold roll over after 4971 hours, this should be sufficient to retrieve lost sensor values.

Logs look like this atm:

0.20 : [{"id":11,"type":16,"data":3.265350819}]
4.70 : [{"id":2,"type":1,"data":26.25}]
8.10 : [{"id":4,"type":1,"data":28.5625}]
10.20 : [{"id":11,"type":16,"data":3.264254332}]
15.20 : [{"id":2,"type":1,"data":26.25}]
18.80 : [{"id":4,"type":1,"data":28.625}]
20.30 : [{"id":11,"type":16,"data":3.265350819}]
30.30 : [{"id":11,"type":16,"data":3.266447306}]
36.10 : [{"id":2,"type":1,"data":26.25}]
40.30 : [{"id":4100,"type":144,"data":28.625}]
40.30 : [{"id":11,"type":16,"data":3.266447306}]
46.60 : [{"id":2,"type":1,"data":26.25}]
50.40 : [{"id":11,"type":16,"data":3.264254332}]
60.40 : [{"id":11,"type":16,"data":3.259868383}]
67.60 : [{"id":2,"type":1,"data":26.25}]
70.40 : [{"id":11,"type":16,"data":3.266447306}]
timmbogner commented 2 years ago

Cool! I've thought about some kind of SD logger but I ran into the same conundrum: without the context of time, a list of data readings is pretty meaningless. This is probably the best way to make do. If we add this, I think it would be good to have it only run if MQTT is down. Also, can you establish the filename when calling the function? SDsend("fdrs_log.csv"); It may also be a good idea to implement writing to other filesystems as well like internal flash. FSsend("fdrs_log.csv", SD); CSV might be a more portable format for the data, although I'm not sure anymore. Maybe somewhere farther down the line we can add a method of sending the missed readings with timestamps over MQTT when a connection is reestablished, but there are several other features that I want to implement first.

thefeiter commented 2 years ago

If we add this, I think it would be good to have it only run if MQTT is down.

Yep, so this would only be used in an MQTT gateway.

Also, can you establish the filename when calling the function?

Good idea!

It may also be a good idea to implement writing to other filesystems as well like internal flash

For internal Flash I would have do do some reading, this is new to me. But it would be much easier since you don't need another SD card breakout board.

CSV might be a more portable format for the data

Good, it is at least easier to use than json.

Maybe somewhere farther down the line we can add a method of sending the missed readings with timestamps over MQTT when a connection is reestablished, but there are several other features that I want to implement first.

Maybe syncing an internal clock when internet is available and using this as a timestamp for when internet is down.

thefeiter commented 2 years ago

I changed a lot in this commit

thefeiter commented 2 years ago

Improvement ideas appreciated.

thefeiter commented 2 years ago

This seems to be the best option: If wifi is activated, we keep time by using an NTPClient which also keeps time when the connection is interrupted via millis and is able to return a Unix epoch timestamp. If wifi is off, it falls back to my original millis timekeeping.

I reduced the resolution to seconds to produce similar timestamps like the NTPClient.

thefeiter commented 2 years ago

This NTPClient timekeeping can also be used in other functions.

thefeiter commented 2 years ago 
1657140888,3,1,20.94
1657140891,2,1,13.75
1657140894,4,1,20.06
1657140894,11,16,3.27
1657140900,3,1,21.06
1657140902,2,1,13.75
1657140904,11,16,3.27
1657140905,4,1,20.06
thefeiter commented 2 years ago

From my side, I am ready for merge.

thefeiter commented 2 years ago

From my side, I am ready for merge.

Well, maybe now I am... :grin:

timmbogner commented 2 years ago

In the flurry of PRs today, the LoRa setup macro names changed and now there are conflicts. Sorry, I should have just pulled it this morning. Or pulled it first, though that still probably would have cause conflicts? Still learning here.

thefeiter commented 2 years ago

The time client tries to update every 60 seconds only. (internally) If needed one can adjust this in the setup. Updating once might be too inaccurate because of the possible drift of millis().

08.07.2022 04:26:19 Timm Bogner @.***>:

@.**** commented on this pull request.

In FDRS_Gateway/FDRS_Gateway.ino[https://github.com/timmbogner/Farm-Data-Relay-System/pull/42#discussion_r916412299]:

}

  • client.loop();
  • client.loop(); // for recieving incoming messages and maintaining connection
  • timeClient.update(); //update internal clock if possible

Is this updating every loop? Should we maybe just update the time when it connects?

— Reply to this email directly, view it on GitHub[https://github.com/timmbogner/Farm-Data-Relay-System/pull/42#pullrequestreview-1032335941], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AK7SSGXCSO4EL5XE2U4XHBLVS6GUPANCNFSM52Q45BUQ]. You are receiving this because you authored the thread.[Verfolgungsbild][https://github.com/notifications/beacon/AK7SSGXEZRXJNPKUXDRZSIDVS6GUPA5CNFSM52Q45BU2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOHWEDERI.gif]