search

openhab / openhab1-addons

Add-ons for openHAB 1.x
Eclipse Public License 2.0
3.43k stars 1.71k forks source link

[MapDB Persistence] Add periodic database compaction #5840

Open nbrealey opened 5 years ago

nbrealey commented 5 years ago

The mapdb persistence database grows in size and fills the disk if the everyUpdate strategy is used.

Expected Behavior

The database should not grow in size if the keys do not change. The database should be compacted automatically if there is a lot of wasted space.

Current Behavior

The mapdb persistence database grows in size and fills the disk. Restarting the openhab2 service causes the database to be compacted. I saw the problem with String items.

Steps to Reproduce

  1. Configure mapdb persistence with to save some String items using the everyUpdate strategy.
  2. Watch the database grow in size: du -h -s /var/lib/openhab2/persistence/mapdb (location of database may vary)

Context

The virtual disk on the VM running openHAB became full. I was using openHAB to implement multiple heating schedules on my Drayton Wiser system when the manufacturer only provided a single shedule. See https://community.openhab.org/t/drayton-wiser-thermostat-binding/35640/320

Your Environment

openHAB 2.5.0 Build #1557 and Build #1558 (openHAB 2.5.0-SNAPSHOT version) from yum repository.

MapDB Persistence: persistence-mapdb - 1.14.0.SNAPSHOT

CentOS 7.6 on VirtualBox (5.2.256) VM on XigmaNAS 11.2.04.6536 (FreeBSD 11.2-RELEASE-P9).

davidgraeff commented 5 years ago

I cannot flag this as "bug" as it really is no bug :/ The openHAB persistence layer is not yet prepared to constrain persistence data to a specific quota.

The same with log files btw.

5iver commented 5 years ago

Since you are using the 1.x version of the binding, you should post your issue here... https://github.com/openhab/openhab1-addons/issues. There is a 2.x version of mapdb that was available in ESH, and I see it in the repo, but I don't see it yet in Paper UI (using S1557).

davidgraeff commented 5 years ago

@openhab-5iver All persistence services are 1.x bundles. The map 2.x bundle was never used and is not part of the karaf feature file.

But Kai expressed the will to move them over to this repo soonish, so it is the correct repository to report such an issue.

5iver commented 5 years ago

But the reported issue is for mapdb 1.14, so how would that be resolved in this repo?

davidgraeff commented 5 years ago

I cannot yet be resolved that is true. But the issue is existing and where do we keep track of it if not here? Nobody will watch the OH1 repo for such an issue, I assume? Especially not if the files have been moved over.

5iver commented 5 years ago

There are 1.x maintainers and it hasn't been moved yet... https://github.com/openhab/openhab1-addons/tree/master/bundles/persistence/org.openhab.persistence.mapdb. :smile: And the issue may not exist in the 2.x version...

davidgraeff commented 5 years ago

Of course it does, the 2.x version is the 1.x version with renamed package names. But we can of course hope that OH 1 maintainers implement a fix although I don't know how, because the core would need to be changed. I try to transfer the issue now.

Edit: I'm not allowed to because I have no write access to OH1-addons.

davidgraeff commented 5 years ago

@kaikreuzer Can you move this Issue to the OH1 repo?

kaikreuzer commented 5 years ago

Moved.

9037568 commented 5 years ago

So the easy fix is, don't use everyUpdate if your use case doesn't make it meaningful, as here.

Compacting the DB occasionally is an interesting feature request, though, so I've relabelled this as a request for that.

kaikreuzer commented 5 years ago

Not using everyUpdate isn't really a good option as this is the main usecase of the mapdb database.

Note that MapDB isn't a normal database, which would store time series. Instead, it only holds the very last value for an item. With that in mind, it is indeed a bit weird if the database size is growing over time.

9037568 commented 5 years ago

Ok, that's a wrinkle.

@nbrealey can you get a trace log for this and attach it?