memory usually not enough when batch processing more than 10 images

[ruudygh]

memory usually not enough when batch processing more than 10 images. I have 64g of ram. When the process is starting to go into the Masker step, my RAM "commited memory" will exceed 65gb and thrown out error and crash my pc for a while.

[VoxelCubes]

That is highly unusual, and really shouldn't be possible. Did you change anything in the profile?

If so, please attach the profile file, you'll find the location by reading the log, or finding it near the config file (these are not the same). If begins like this:

[General]

# Preferred file type to save the cleaned image as.
# If no file type is specified, the original file type will be used.
preferred_file_type = 

# Preferred file type to save the mask as.
# Only image formats that allow for transparency are supported.
preferred_mask_file_type = .png

Please attach one or more sample images that cause this problem so I can diagnose the issue. It may be related to the size and type of image file, so I cannot figure out the problem without this.

Unfortunately there is no way to limit the number of threads currently, as I haven't had any such issues when handling several hundred images, due to only as many as there are threads being loaded at a time, meaning it'll make due with 1 to 3 GB on 16 thread machines.

[ruudygh]

Strangly when I test it out in a virtual machine (win10 with only 8gb of ram assigned), the number of image i can batch process are around 50 before it reach max commit memory.

In comparision when processing using host vs using guest machine, the comit memory rise much faster with host machine than with guest machine. So i think there might be some issue on my host machine. Since the images im processing are not suitable to public so i wont be uploading here lol. But i have test it with few comics from different artists they came to the same problem.

The setting on both machine: windows page file are set to 0 (not sure if this is the cause) Using Panel Cleaner binary .exe. No use gpu. Default profile. Nothing changed. Cleaning. Save output. (checked Mask or isolated text)

The error messages are a bit different everytime.

[VoxelCubes]

Those error messages do quite clearly confirm it's running out of memory, nothing more, unfortunately. You could send some sample images that exhibit the problem to my email address voxel.aur@gmail.com for me to maybe figure out what's wrong, but as it stands, the only other option here is to offer an option to disable multiprocessing, which would slow down the work by a lot, but at least finish, if the issue is caused by something going wrong with that.

[VoxelCubes]

Version 2.9 will allow you to set individual thread limits for masking and denoising. Additionally, it will warn you ahead of time if you're running low on ram. This way you can turn down those threading limits in the profile to still be able to clean without the OS killing the process. How much ram you use depends on the size of the images. Something above 4000x7000 will result in significantly more ram used. If you still come across such ridiculously high ram usage, please send me the images causing these issues so I can investigate. Sorry it took so long, should've done this sooner. Release coming this month, may or may not consider waiting on #110 (PSD support).

[ruudygh]

Version 2.9 will allow you to set individual thread limits for masking and denoising. Additionally, it will warn you ahead of time if you're running low on ram. This way you can turn down those threading limits in the profile to still be able to clean without the OS killing the process. How much ram you use depends on the size of the images. Something above 4000x7000 will result in significantly more ram used. If you still come across such ridiculously high ram usage, please send me the images causing these issues so I can investigate. Sorry it took so long, should've done this sooner. Release coming this month, may or may not consider waiting on #110 (PSD support).

Thanks for notify. Actually i found the cause of the memory problem few weeks ago. I have my windows11 set to have 0 page file / virtual memory, and that is the cause. I changed the page file to minimum 8gb and maximum 96gb, PanelCleaner is working fine after that.

[VoxelCubes]

Having no swap space for overflowing RAM would only be an issue if you really did fill up all 64GB of RAM somehow. But using swap a lot makes the computer really slow, so it isn't ideal to rely on it.

The only way I could imagine pcleaner filling 64GB is if you have 16 or more cores + super high res images, like 8000x14000 pixels, if it isn't some strange bug I still haven't found.

But if you don't have any issues currently, then I guess that's all good then. Thanks for letting me know.

[ruudygh]

I have amd 5900x with 24 cores. The comic im trying to process are only within 1600x2400. But it could also be the problem of my PC. I use 3d app like blender and maya everyday, most of the time the committed memory would fill up about twice faster than the actual memory. So If i want to fully utilize my 64gb memory, i need to set the page file to at least around 128gb. Not sure if this is normal 😅

[VoxelCubes]

24 cores would definitely spawn a ton of threads for cleaning/denoising, which would eat ram like crazy. Makes sense. Once 2.9 is released around the end of the month, you'll be able to limit the number of threads those use and you'll never have ram issues again, I hope. You could run from source right now, it's all done except for the psd merge request. But that's a little more complicated than downloading an exe. But if you use pip to get cuda acceleration already, it wouldn't be much harder.