search

kornelski / dssim

Image similarity comparison simulating human perception (multiscale SSIM in Rust)
https://kornel.ski/dssim
GNU Affero General Public License v3.0
1.09k stars 70 forks source link

Controlling/limiting the CPU and Memory usage #106

Closed jouven closed 2 years ago

jouven commented 2 years ago

One of the features of this program is "Takes advantage of multi-core CPUs.", is there a way to limit or control that? this program is mostly unusable with parallel-like software (https://www.gnu.org/software/parallel/) because the core usage is not predictable as far as I know. I use a script to convert images depending on their dssim values and I've seen a single process of dssim at 1200 CPU% in htop, using 12 cores, which even if it's for a second or two, it makes parallelisation of dssim a problem, because it slows the computer substantially, this is on a 3900x with 32GB of ram, then again the images are pretty big resolution wise, 5xxx X 7xxx, which leads me to the issue of the memory usage, is there a way to know how much memory will it use? (knowing the image resolution/color depth), for the images I've previously mention, I've seen a single dssim process use 6-7GB of ram, still, ram usage is easier to control using parallel

kornelski commented 2 years ago

Yes, you can set RAYON_NUM_THREADS environment variable.

This will work for almost every Rust program, because that's the config of one of the most popular multi-threading Rust libraries.

jouven commented 2 years ago

After some testing it seems I can make it work using RAYON_NUM_THREADS plus parallel's -j to limit the CPU, to limit the memory parallel's --memsuspend seems to work. After tweaking these options the scripts executing in parallel don't end halfway anymore, before they were running out of resources

Thanks for the help