Persistence Layer : MongoDB -> NeDB

[tdurand]

No, we won't run MongoDB on android (for now ;-) )

https://github.com/louischatriot/nedb sounds perfect , we already used it for the twitter bot.. lightweight and it was working very well ( @b-g recommended this back in the days 🙌 )

TODO:

• test if fully compatible with current DBManager API (as NeDB as the same API as mongodb) : https://github.com/opendatacam/opendatacam/blob/master/server/db/DBManager.js by just switching dependencies
• if so, at a parameter in the config.json to choose if we want to use NeDB or MongoDB
• if not, do the same + build an adapter

[b-g]

OK + great! :)

Just: Could we test whether a heavy recording eg. an entire days of 24h (which I guess will result in big files) will still work for all features? Happy to do the stress test with my phone once there is a alpha.

[tdurand]

yep of course...

Also maybe I could properly evaluate what would be the cost of actually running mongodb on android, there is a SDK maybe it is not very hard.. : https://docs.mongodb.com/realm/android/install

[tdurand]

PS: beta coming !! tomorrow or friday !

[tdurand]

yep of course...

Also maybe I could properly evaluate what would be the cost of actually running mongodb on android, there is a SDK maybe it is not very hard.. : https://docs.mongodb.com/realm/android/install

Hum. this seems to be MongoDB Realm.. which is for now unclear to me what is it , is it free, is it the same API .. anyways.. for now I will just go to down the route to the fast and simple for the beta... and let's see if we need to move to more robustness later on

[b-g]

👍! Looking forward! :)

[tdurand]

NB: first alpha ready: https://github.com/opendatacam/opendatacam-mobile/issues/3#issuecomment-747726946

update on this.. turn out to be a little more complex than expected.. as always .. but working ! yeah

• API isn't quite the same as MongoDB but it had all the features we needed .. so was able to make it work after trial an error, documented "diff" : https://github.com/opendatacam/opendatacam/compare/development...evo-addnedbconnector , basically wrote a new DBManager with same API .. good thing is that is super trivial now to have an option in the config.json to choose between MongoDB or NeDB for persistence
• every 5 min we run a compact database otherwise it grows really fast (200 MB for 10 mins).. but after compacting it takes only 50 kB from my test for each 5 min (it takes 100ms to compact ;-) ).. so for 24h I think it should take just a few MBs of space for a recording of this size.. but I need to benchmark better.. Also I think it might be the time to add an option to only persist counter data and not each frame of the tracker if we want.. most people don't care of the raw tracker data.

First impression is that it sounds good enough for the first android version.. but let's see after more tests

[b-g]

👍

Also I think it might be the time to add an option to only persist counter data and not each frame of the tracker if we want.. most people don't care of the raw tracker data.

Yes. We kind have for this also the swich off pathfinder setting ... but it would be a pitty to disable the pathfinder mode on default as maybe people won't notice them ... hence just a new switch "Record Raw Paths: y/n" (n = default)