dbsize grows instead of reduce when removing large amount of data!

[saveriocastellano]

I'm experiencing a very weird behaviour of ssdb.. I have a large ssdb database (about 100Gb) that contains millions of keys, hashes and sorted sets. Because I have embedded in the key names a count that represents the time, I'm able to know what key/hash/sorted-set I need to remove (because the data can be expired given it is too old) from the name of the key.

So I have created a scripts that cycles through the key names (using scan, zlist, and hlist commands) and determines if the keys has to be removed from its name.

I have run the script and it has removed millions of keys, hashes and sorted sets, however very surprisingly the size of the database has increased instead of decreasing!

I do understand that every now and then ssdb triggers compresssion, and i do understand that ssdb doesn't write things immediately in disk... but still after running my script for so long and removing so much data i really expect the db size to decrease.

Can you please suggest what could be the cause?

[saveriocastellano]

could this be related to the issue of leveldb described here: https://github.com/google/leveldb/issues/603

?

[saveriocastellano]

I also saw this issue and test case:

https://github.com/ideawu/ssdb/blob/master/deps/leveldb-1.20/issues/issue178_test.cc

so I wonder, is this related to the problem I described?

[ideawu]

Hi, this issue is related to leveldb. As I know, deleted keys will not be actually removed from disk immediately, they may be removed after a compaction or may not. It depends on leveldb's policy.

[saveriocastellano]

while it is acceptable that they are not removed immediately, for me it is very crucial to make sure that they will eventually be removed.. otherwise an ever growing database will be unusable for me. Could you please tell me how to set leveldb policy in a way that deleted keys will be removed for sure?

[ideawu]

Manually invoke a compact operation will actually remove deleted keys from disk, execute compact in ssdb-cli. PS: do compact when server is not busy.

[saveriocastellano]

Unfortunately my server is busy 24h a day :(

Actually I’m running 2 masters in “mirror” and My server only uses one of them actively. So I was wondering, will it be better to run “compact” on the second master, the one that it is not actively used? Will this immediately affect the performance on the primary master? And after having compacted the data on the second master, will the data on the first master also reduce through syncing ?

[ideawu]

Compact on a server will not affect on another server.

[saveriocastellano]

Does the “compact” command of ssdb support key ranges and type (kv,hash,sorted set, list)? If it does then I could slowly split the compaction in smaller ranges in order not to affect the performance too much

[saveriocastellano]

@ideawu just as it might be useful for others, considering that I'm running with a master-master configuration with sync=mirror, I was able to solve this by doing the following:

• removing data on MasterB (the one that is normally not actively used)
• wait for MasterA and MasterB to be fully in sync again
• run "compact" command on MasterB (this won't affect performance as this is not actively used)
• switch my application to use MasterB
• run "compact" command on MasterA
• switch back my application to use MasterA

after doing the above the size of my database was reduced from all the data that I had deleted.