sahib
commented
5 years ago Hello @Blindfreddy,
Firstly, rmlint is AWESOME, thanks, I'd been looking for a great dedup tool like this for years !!!
Good to hear. But we've been also around for a few years now. :smile:
Running on an odroid hc1, rmlint spawns several processes (seems one per cpu core) and each of them uses much more memory than the -u limit. Changing the -u value does not seem to have any effect. Memory consumption grows over time.
This is pretty weird. rmlint definitely only works as a single process, so I don't quite understand where the several processes in your htop are coming from. You're sure it's not showing threads (it has different pids, but just to make sure)?
Regarding -u: This option is sadly not exact science (and will never be). It has the most effect when used together with -p (paranoid mode). There we do not store hashes but actual file contents, which potentially use a lot more memory. For normal operation, this switch does not do very much. The actual cause of your memory issue are the several processes. Any clue why? How do you start rmlint?
Blindfreddy
Firstly, rmlint is AWESOME, thanks, I'd been looking for a great dedup tool like this for years !!!
But, I either don't understand the -u switch or it doesn't work properly. No matter what I specify for -u, all system memory is consumed and the oom_reaper kills rmlint after a while.
Running on an odroid hc1, rmlint spawns several processes (seems one per cpu core) and each of them uses much more memory than the -u limit. Changing the -u value does not seem to have any effect. Memory consumption grows over time.
Here is the htop output to demonstrate
root@odhc1:/etc/monit# htop
....
Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||897M/1.95G] Tasks: 97, 121 thr; 11 running
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 18702 andrev 20 0 595M 437M 4036 S 772. 21.9 54:02.25 rmlint -e -u 128M HomeVideo/ Camcorder/ 19870 andrev 20 0 595M 437M 4036 R 98.7 21.9 0:06.68 rmlint -e -u 128M HomeVideo/ Camcorder/ 19886 andrev 20 0 595M 437M 4036 R 98.7 21.9 0:04.11 rmlint -e -u 128M HomeVideo/ Camcorder/ 19744 andrev 20 0 595M 437M 4036 R 98.1 21.9 0:23.77 rmlint -e -u 128M HomeVideo/ Camcorder/ 19825 andrev 20 0 595M 437M 4036 R 92.4 21.9 0:11.42 rmlint -e -u 128M HomeVideo/ Camcorder/ 19847 andrev 20 0 595M 437M 4036 R 65.8 21.9 0:05.05 rmlint -e -u 128M HomeVideo/ Camcorder/ 19698 andrev 20 0 595M 437M 4036 R 65.2 21.9 0:19.76 rmlint -e -u 128M HomeVideo/ Camcorder/ 19859 andrev 20 0 595M 437M 4036 R 62.7 21.9 0:05.96 rmlint -e -u 128M HomeVideo/ Camcorder/ 19841 andrev 20 0 595M 437M 4036 S 45.6 21.9 0:07.03 rmlint -e -u 128M HomeVideo/ Camcorder/ 19891 andrev 20 0 595M 437M 4036 R 38.6 21.9 0:01.48 rmlint -e -u 128M HomeVideo/ Camcorder/ 19754 andrev 20 0 595M 437M 4036 R 24.7 21.9 0:16.99 rmlint -e -u 128M HomeVideo/ Camcorder/ 18727 andrev 20 0 595M 437M 4036 S 14.6 21.9 0:40.87 rmlint -e -u 128M HomeVideo/ Camcorder/ 18728 andrev 20 0 595M 437M 4036 S 9.5 21.9 0:43.00 rmlint -e -u 128M HomeVideo/ Camcorder/ 19849 root 20 0 9856 4872 4040 S 6.3 0.2 0:00.23 /usr/sbin/sshd -D -R 29002 root 20 0 5080 2092 1320 R 1.9 0.1 18:10.34 htop 2501 plex 20 0 248M 1524 0 S 0.6 0.1 1h57:28 /usr/lib/plexmediaserver/Plex DLNA Server
...