JackSlateur
commented
1 year ago Hello @trofi
When you do a command like this: duperemove -hr --hashfile=/tmp/hashes some-directory, duperemove will list and process the stuff in some-directory but will also check files found in the hashfile (do they still exist, maybe they changed etc)
For your use case, which is, as I understand it, some sort of append-only filesystem, all that work is useless and expensive
Could you check the quick code I push in the this branch ?
It adds a --ignore-existing-files options for you to check
As a test, I created a directory that contains 100 sub-directory, with 1000 files each (to 100000 files in total) The hashfile has been built:
0.49 [jack:/tmp/mnt] hashstats /tmp/hashes
Raw header info for "/tmp/hashes":
version: 3.0 block_size: 131072
num_files: 100000 num_hashes: 0
Print top 10 hashes (this may take some time)Running it from master, with just one directory as a target:
0.42 [jack:/tmp/mnt] /usr/bin/time -v duperemove -hr --hashfile=/tmp/hashes 1 > /dev/null
Command being timed: "duperemove -hr --hashfile=/tmp/hashes 1"
User time (seconds): 0.22
System time (seconds): 0.78
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.02
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 40576
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 9575
Voluntary context switches: 8
Involuntary context switches: 151
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
0.57 [jack:/tmp/mnt] /usr/bin/time -v strace -fty duperemove -hr --hashfile=/tmp/hashes 1 |& wc -l
508305And with the option:
0.24 [jack:/tmp/mnt] /usr/bin/time -v duperemove -hr --hashfile=/tmp/hashes 1 --ignore-existing-files >/dev/null
Command being timed: "duperemove -hr --hashfile=/tmp/hashes 1 --ignore-existing-files"
User time (seconds): 0.08
System time (seconds): 0.03
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.12
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 22656
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 5086
Voluntary context switches: 8
Involuntary context switches: 14
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
0.77 [jack:/tmp/mnt] /usr/bin/time -v strace -fty duperemove -hr --hashfile=/tmp/hashes 1 --ignore-existing-files |& wc -l
13165
trofi
I use
btrfsfilesystem (compression enabled). I have about 30M files in 75K directories. It's a growing cache of build artifacts. It grows a directory at a time and has quite a bit of duplicates (50% to 70%). Most of the similar files have only partial match: ~20 bytes of ~1MB differ (different identifiers embedded).I wondered if I could use
duperemoveincrementally one directory at a time. But look like at least by defaultduperemovedoes a lot of work when tried to deduplicate one file by reading (and maybe extracting extents?) for all known files.The following seems to be enough to illustrate slowdown: we create N files and add files one by one into database (real use would be a directory-at-a-time):
I'll change
1000to increase linearly to see how long it takes to process:It's not a clean test as it also measures time to create the files. File creation is minor though: it takes 9ms to create 5k files.
I think we see a quadratic slowdown here. Is it an intended (required by design) behaviour? Or possibly an omission somewhere and incremental mode could be faster?
Thanks!