opensensorhub / osh-core

OpenSensorHub Core Modules
http://docs.opensensorhub.org
Mozilla Public License 2.0
33 stars 17 forks source link

H2 database continues to grow regardless of Automatic Purge Policy settings #231

Open TheRestOfMe opened 1 year ago

TheRestOfMe commented 1 year ago

First encountered with PiAware driver. The size of the DB file on disk continues to grow indefinitely, even when Auto Purge settings are set. When viewing DB timeline in sensorhub/admin databasae UI, the Time Range bar chart shows the correct time window. Attaching screencap of bar chart for this DB. DB is now 3.4G for only 1-2 hours of data (should be much smaller):

pi@piaware:/opt/osh-node-2.4.6 $ ll -h connected_systems.db -rw-r--r-- 1 pi pi 3.4G Jan 17 00:16 connected_systems.db

AutoPurge section in config.json: "autoPurgeConfig": { "objClass": "org.sensorhub.impl.database.system.MaxAgeAutoPurgeConfig", "maxRecordAge": 3600.0, "enabled": true, "purgePeriod": 3600.0 },

TimeRangeBarChart

TheRestOfMe commented 1 year ago

Was going to make the db file from the Alabama PI available. It has been running for one week and was up to 3.7G. When I stopped the OSH process, it shrunk down to 27M. That was with "Compact on Close" true. So maybe that is a useful clue. I also farmed the log files and show there were ~6240 unique FOIs added during the week that it was running.

alexrobin commented 1 year ago

Ok, can you keep this file, and start building another one for a week after you turn off compact on close so I can compare them? Thanks.

alexrobin commented 1 year ago

Can you also attach the complete JSON config of the storage so I can see the other settings?

TheRestOfMe commented 1 year ago

I was able to stop the OSH Pi in Austin without 'Compact on Close' set. It was running for ten days, up to ~4G and ~6600 unique FOIs. I am uploading the db file to my Google drive now. I'll post a link once it completes.

Config file for that instance is here: https://drive.google.com/file/d/1frrwdmw0Xs_9hKh4-ig3CB7dURriL3p1/view?usp=sharing

TheRestOfMe commented 1 year ago

DB File: https://drive.google.com/file/d/1og3kx-BvgxYbrv3N25cjxDE8x3ROj0UA/view?usp=sharing