Mqtt send gateway timestamp and resend packets logged to SD/Flash

[thefeiter]

Since I have created the SD and flash logging functionality I would now like to add that when the Mqtt connection is up again the logged values will be sent.

For this I would like to send the internal unix timestamp with every Mqtt package to be able to send the missed packets backdated.

I will start working and do some testing but if there are any Ideas, feel free...

[thefeiter]

MQTT messages will look like this:

[{"id":5,"type":16,"data":3.335526228,"time":1658352635},{"id":5,"type":1,"data":23.3125,"time":1658352635}]

[aviateur17]

Sounds good. Is there a max file size configurable or limits to the number of messages? Do we do a checksum or hash on the file to make sure the data is valid or how do we check for invalid data? Do the current MQTT packets send data to the broker with timestamps - I'm thinking no but maybe some work has been done on that. (Looks like you just handled that) I'll keep thinking on this.

[timmbogner]

This looks fine! I've started work on pinging functionality, which will lead to pairing and then controlling devices: the-big-change

[thefeiter]

I can Imagine a configurable file size/ message count limit for the logging could be implemented. Can you tell me more specifically what your intention behind this would be? Is it that likely for the file to be corrupted so that we need a checksum? I guess one could just write a checksum to the end of the line in the file, but I don't think that this is neccessary.

[thefeiter]

Sending the timestamps works good and they don't deviate from the ones on my backend.

[aviateur17]

My intention for the file size/message number limit would be to prevent the file from being so large that it surpasses the filesystem limitation or micro SD card/flash size. Perhaps if something "goes wrong" and there is a file size limit then you can prevent the whole file from being junk. Safety stuff.

Regarding checksum/hash, gotta think about what happens in worst case scenario. What if the controller reboots or loses power while writing to the filesystem? What if the flash goes bad? Would be nice to have some indication if something isn't working as it should.

When writing code I try to implement not only the functionality but also error checking and warnings if things aren't working as they should.

[thefeiter]

prevent the file from being so large that it surpasses the filesystem limitation or micro SD card/flash size

Yep, I am with you here, what is the best way to handle the overflow? A circular buffer in a file takes quite some cpu time.

Would be nice to have some indication if something isn't working as it should.

Then I think the best approach is to verify each line of the log before resending it on MQTT.

[timmbogner]

what is the best way to handle the overflow?

So, the situation is that the gateway has been disconnected from MQTT for so long that is has filled its filesystem with logs and is out of space. Can we just give up if it reaches this point? 🤔😌

If you want to try to add more contingency plans, I'm not going to stop you. But I'm perfectly happy with it filling up, then closing the file and being done.

[timmbogner]

I think i'll leave the integrity checks up to you guys. It's probably a good thing to be thinking about, but you can always save it for later if it isn't causing problems.

[thefeiter]

Can we just give up if it reaches this point?

Yup, I think that is sufficient.

[thefeiter]

So, I now added a file size limit and a crc confirmation for each log entry. Is there anything to add? If not, I am ready for merge.

[timmbogner]

Here's something I just thought of... do you think sending backup data under a different MQTT topic would be a good idea? There would probably be a different mechanism on the front-end to handle the backlog, so it would make sense not to send it in the same topic as current data.

In the future, perhaps we can extend this file IO functionality to purposely buffer data for long periods of time. Like, if a farmer only wants a remote station to save up data, then release it at midnight every night to the next gateway. So one gateway can be on and collecting data, but the (for example) three repeater gateways are set to turn on only between 00:00 & 00:30 in the morning. The collector gateway is then scheduled to release its buffer at 00:05. This would save hella battery!

We must keep in mind that the clock will drift without an external RTC, however. I have ideas about a way to keep/update time system-wide, but it will come after a few other features and abilities are available.

[thefeiter]

do you think sending backup data under a different MQTT topic would be a good idea

Kind of, yes. I actually did this for testing, but since my system mainly consists of a listener which enters all incoming readings into my database with the timestamp of the reading, there is no need to handle it seperately.

I will add a config option for the backlog topic and leave it at fdrs/data as default.

[timmbogner]

I'll get to this soon, don't worry!