search

servalproject / serval-dna

The Serval Project's core daemon that implements Distributed Numbering Architecture (DNA), MDP, VoMP, Rhizome, MeshMS, etc.
http://servalproject.org
Other
170 stars 81 forks source link

Database slowdown over time #133

Open gh0st42 opened 6 years ago

gh0st42 commented 6 years ago

We used serval to distribute sensor data, taking snapshots every 1-5 minutes. This included about 5 journals (temperature, pressure, etc) per sensor node (2 in total) and a 8MP image taken every now and then. After a few days serval became almost unusable. Listing all rhizome entries takes quite long (rest and cmdline) and inserting new content is almost impossible (all done via rest but one insert takes over a minute). One node had around 5000 entries and the other around 2800. Restarting serval-dna or calling rhizome clean does not really make it better. We still had plenty diskspace and inodes left on the devices...

I'm not sure whether the problem is the growing journals or the large number of images. Since you guys have more long term experience and heavily use meshms (which are just fancy journals) the bug is probably more related to the total number of files.

lakeman commented 6 years ago

Turning on debug.profiling and repeating each of the restful requests might help to pin down where the time is going.

lakeman commented 6 years ago

We should probably create subfolders in the rhizome blob directory to limit the size of any single folder.