[Feature] Select FIFO / FILO when the buffer is full

[NorbNorb]

Great project so far! I have a feature request and I want to share a little context first: after the buffer is full, incoming messages are currently dropped until there are slots available for storage again. Imagine I want to set the buffer high enough to store messages that come in every minute, and I want to be able to fill the buffer for at least a month (60 24 30). If this time span is exceeded and the connection is restored later, I would rather be interested in the latest data and not the oldest from a month ago.

What if you had the ability to specify how the buffer is emptied, even with a full buffer? In the current implementation it is FIFO (first in - first out). However, in my example above, it would be more desirable to have a FILO (first in - last out).

BTW: is there any technical or practical limitation on the buffer size?

[colinl]

You seem to be asking for two things.

Firstly, that if the buffer is full then you would like to add new messages to the buffer, and drop the oldest message so that the buffer retains the most recent data rather than the oldest data. I can understand that this might be useful.

Secondly, that when the connection is re-established then you would like it to send the buffer in reverse order. I don't understand why this would be useful.

In terms of buffer capacity it is only limited by memory availability (and disk availability if persistent context is in use). Assuming that you are not running on something like Pi Zero then one might assume that at least 1 GB of memory might be available. If, for example, each message were to consumes 100 bytes then 1GB will hold 10 million records. There should be no problem storing your 43,200 messages. Do make sure that your messages do not have any large properties attached to them though. Feed the messages into a debug node set to Output Complete Message and make sure there are no significant sized properties there.

[NorbNorb]

Thank you for your reply! You are right, I have mixed up two things. My main concern is that when the buffer is full, (in my use case) the oldest record is overwritten first, as it's not as important as the latest data.

The second issue has no particular usefulness, except that with a large buffer you get the current state first at the moment the connection is restored, and then the large data gap closes over time. But this can also be seen as an implementation detail and has nothing to do with my actual concern.

Thanks also for the thoughts on the buffer size!

[colinl]

Is this still an issue for you, given the fact that you can save a huge number of messages in context if necessary?

[NorbNorb]

Hi! Well, it's a feature request. The most important thing is that the basic functionality works even with a very large amount of buffered messages. This is just a suggestion for improvement, especially in case you have buffered thousands of messages but want your monitoring to see that the device is back online and can provide the latest data. So this request is about how to fill large time gaps, especially if it takes a long time to retransmit the old messages.

[colinl]

Do you actually have a need for this feature?

[NorbNorb]

Yes, my use case is that devices can go offline for several weeks or even months. The internet connection is usually okay but not guaranteed to have a large bandwidth. As said, this is an enhancement, not a must-have.