search

mcollina / mqtt-level-store

Store your in-flight MQTT message on Level, for Node
MIT License
24 stars 11 forks source link

Question: Are there any data size limitations? #14

Open dushyantbangal opened 5 years ago

dushyantbangal commented 5 years ago

I am talking about the total size of data that I can have offline in base network connectivity is lost.

I was using mqtt-nedb-store but nedb is does not support more than 256 MB as it needs to parse the file to json, and 256MB is the limit of js string.

Here it looks like the data is being written to multiple small files, which I think avoids that issue.

So are there any limitations that I should know of?

robertsLando commented 1 year ago

I have created a new store that could help: https://github.com/robertsLando/mqtt-jsonl-store

dushyantbangal commented 1 year ago

@robertsLando I might be wrong, but it looks like the db on which you have built this on is writing the data to a single file. In that case it might have the issue I have mentioned above. I did use mqtt-level-store extensively and didnt find any such limitation.

robertsLando commented 1 year ago

Is your problem related to offline packets storage? If so I personally create a custom store for them and handle my own. You can write them in all the files you want.

Also mind that a size limit is not really necessary as all mqtt storage db use messageId as key and that is a 16bit number. It means there could be at most 2^16-1 packets in your db then them will be overriden. This is why I use a custom store based on the one I have build