Ability to purge deleted object nodes after a set time period

[shuther]

Overview

If I remove everything from an actual .kdbx file (including recycle bin) and save it (or use save as backup), I notice that the file is not decreasing. There is no compact process so I suspect that everything that was deleted (as per the user request) is still present (and could be recovered?).

Steps to Reproduce

1. duplicate a kdbx file
2. open the new one and delete everything
3. empty and delete the recycle bin
4. Save or save database backup
5. check the size of the output

Expected Behavior

an empty database should not be 6Mb

Actual Behavior

file size stay the same

Context

steps followed to investigate another issue on keepass-rs

KeePassXC - 2.7.8 Revision: f6757d3

Operating System: Windows

[droidmonkey]

6 MB? Did you have a huge attachment on an entry?

[phoerious]

Go to settings, set history size to 0, save, then set it back to whatever it was.

[droidmonkey]

Technically clearing the recycle bin should wipe out the history since the entry nodes themselves are removed at that point.

[droidmonkey]

I just added a bunch of large attachments and custom icons to entries and deleted them (from the recycle bin) and the size went from 4 MB -> 60 KB. So you must have not actually deleted the items or you are looking at the wrong file.

[shuther]

The file is 6,232,643 bytes. For me it is clearly empty (or maybe I am missing something) I can't send you the file but let me know what I could share with you to solve it.

[droidmonkey]

Show the file properties sheet (from your file manger) for:

[shuther]

is it what you need?

[droidmonkey]

Well that sure is interesting

[droidmonkey]

Do you have anything listed in previous versions? I'm not 100% sure, but that might be counting against your size.

[shuther]

nothing in previous version; this file was obtained as a save as Backup but I had the same issue with regular save.

I tried to disable the compression and save as backup and the file is even larger (9Mb). I don't feel confortable sending you the file but I may run a debug version of keepassxc (on windows or linux)? I am able to replicate the issue on a Mac.

[shuther]

Going back to the reason why I opened this issue (https://github.com/Narigo/keepass-diff/issues/76); could it be that my file is "corrupted" with a silent error, and keepassxc is not purging the file due to a silent error or a bug in the recent version of the parser?

[droidmonkey]

My only thought is entry attachments, there must be one "stuck" somehow to a phantom entry. Attachments are very odd in the kdbx specification and there have been corruption points in the past with other programs, but that was usually attachments getting replaced randomly, not stuck.

Can you try export to xml to see if there are any entries dumped?

[phoerious]

Attachments should be rewritten entirely on save.

[shuther]

so export in CSV returns nothing export in XML returns a lot of (3/4 of the file):

            <DeletedObject>
                <UUID>eVV21DDsrEylbYmQ15p+8Q==</UUID>
                <DeletionTime>YZwE3g4AAAA=</DeletionTime>
            </DeletedObject>

the 1st part is about the icons. I don't see anything related to the attachments.

[droidmonkey]

Deleted object lines are fully expected.

[shuther]

The xml exported file of the empty kdbx is 9MB; not compressed the kdbx file is 9MB The full DB exported as xml is 28MB. The kdbx not compressed is 29MB The 2 files, compressed are about 6MB (same size).

1. Since I emptied the recycle bin, I would not have expected DeletedObject?
2. Is there an option to import the xml into a new keepassxc database? I wanted to compare the size of the database to see if there is anything else in the kdbx?
3. My understanding is that the keepass entries are removed from the kdbx file (maybe some headers stay), so no space saving is expected. Therefore, everything is fine? I let you decide if my point 1 is a concern or not.

[droidmonkey]

9MB of what? That is a LOT of XML. How many entries did you delete, over 1 million?

[shuther]

I have 4700 entries (maybe I am an extreme user). With about 123 characters for each deleted line in XML, it is about 5.7Mb of XML for this section. There is a big part for the icons so it sounds right for me?

[michaelk83]

4700*123=570k. Did you miss a zero somewhere?

[droidmonkey]

That still doesn't add up, but what happens when you go to Database -> Database Settings -> Maintenance tab. Then purge unused icons. I coupd buy you have 8MB of unused favicons.

[shuther]

correct, I made a mistake with a 0 :( So here are the new stats:

• 3190 icons, about 830 characters each (for the 1st one in the list) = 2.6MB, but the full XML section is 8.7MB. I identified some section that are 20K for one icon.
• 7225 deleted objects * 123 characters (I have some with 130 characters) = 1MB

So in term of size of the file, it all comes from the icons. :) When I purge the icons from keepass-xc, the file is now 200KB Would it be possible to purge the DeletedObject?

[droidmonkey]

No you cannot purge deleted objects, you would never want to do that in practice to all of them. When compressed those xml entries constitute a negligible size.

However, being able to clear deleted objects over a year old (for example) would be a nice maintenance item in the database settings. As long as there is little chance for those entries to be merged back in.