search

sahib / rmlint

Extremely fast tool to remove duplicates and other lint from your filesystem
http://rmlint.rtfd.org
GNU General Public License v3.0
1.86k stars 128 forks source link

Error : unable to push thread to pool - continue or quit large job? #570

Closed james-cook closed 2 years ago

james-cook commented 2 years ago

I am getting sporadic messages during the run of the command (not script) ERROR: Unable to push thread to pool 0x32687020: Error creating thread: Resource temporarily unavailable The command is in its last "Matching..." phase. The command itself is still running and finding matches as usual.

This is a LARGE job, which has been running for several days. So I'd "prefer" to continue and run the script if possible. This assumes the error just leads to possible dropped match checks and not false matches or.... Any advice appreciated. Should I instead e.g. quit and search for sets of files (e.g. by slowly decreasing "-S" "from to" size) instead.

_

Env. is underpowered(!) - Rasberry Pi4 with 4GB RAM - recentish compile (Feb 2022)

james-cook commented 2 years ago

in the end the command itself crashed:

ERROR: Unable to push thread to pool 0x30b8a7c0: Error creating thread: Resource temporarily unavailable
ERROR: Unable to push thread to pool 0x30b8a7c0: Error creating thread: Resource temporarily unavailable
?¦¦¦¦¦¦¦¦¦¦¦¦ERROR: Unable to push thread to pool 0x32685280: Error creating thread: Resource temporarily unavailable
ERROR: Unable to push thread to pool 0x32685280: Error creating thread: Resource temporarily unavailable
ERROR: Unable to push thread to pool 0x32685280: Error creating thread: Resource temporarily unavailable
?¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦?Matching (265262 dupes of 64398../../../glib/gmem.c:135: failed to allocate 4194304 bytesTrace/breakpoint trap

) I checked the rmlint.sh file and it looked "OK" - (not complete but "OK"). I am hardlinking and not deleting so the risks are "lower". I went ahead and ran the incomplete script with "-pxd". I used-p (paranoid) so each file is checked to be a real match in case the error above was leading to false matches.

To solve this problem "properly" I will now add some temporary swap space (a swap file on my HDD (not the SD card!) ) just for the duration of the rmlint command - and re-run the original command.

For those interested, on raspberry pi 4/4GB these were my commands:

# go to the mounted HDD directory:
cd /srv/dev-disk-by-label-OMV2/
ls
sudo fallocate -l 8G swapfile-temp
ls -lh swapfile-temp
sudo chmod 600 swapfile-temp
ls -lh swapfile-temp
sudo mkswap swapfile-temp
sudo swapon -show
sudo swapon swapfile-temp
sudo swapon -show
free -h