enhance borg2 compact stats

[ThomasWaldmann]

When building a ChunkIndex it currently starts from refcount=0 and then sets refcount=MAX_VALUE if a chunk is used.

That's how most of borg2 works now: it doesn't do refcounting anymore, just a boolean "do we have chunk X".

For better deduplication stats in borg compact, we could deviate from that in just borg compact and do precise refcounting without any additional effort.

Before persisting the ChunkIndex, we then need to set refcounts to MAX_VALUE, similar as we clean up the size values.

To consider:

• what do we win?
• we have the total size also in the archive metadata and can sum up all of these.
• do we want to add per-directory stats / analytics? maybe even 2-pass stuff?

[ThomasWaldmann]

Comment about what's interesting for practical usage: https://github.com/borgbackup/borg/issues/122#issuecomment-125700186

[awgcooper]

This would definitely be useful: https://github.com/borgbackup/borg/issues/122#issuecomment-125915021

Question: if compression and/or obfuscation is enabled, would the size stats be given for the native file, pre-compression etc?

[awgcooper]

Something else, not sure if it relates specifically to this: let's say I have a specific file backed up. I know this because it appears in a list contents of the most recent archive. Let's say I wanted to eliminate this file from the whole repo, how would I do that. Would I simply delete the first instance of it being backup up and by doing so that would automatically eliminate all dedups? If so, how would I find it? Fusermout?

[ThomasWaldmann]

@awgcooper No, it does not work like that.

But you can use borg recreate to rewrite all the archives that contain the unwanted file (or the directory). Just be very careful with that and first use --dry-run --list to see if it does what you want.