hako-mikan / sd-webui-supermerger

model merge extention for stable diffusion web ui
GNU Affero General Public License v3.0
743 stars 112 forks source link

RAM usage keeps increasing continuously when I use the XYZ Plot. #311

Closed Ketorami closed 6 months ago

Ketorami commented 10 months ago

RAM usage keeps increasing continuously when I use the XYZ Plot. I've noticed that the more I merge models in the XY Plot, the more RAM it consumes. Sometimes my computer freezes for a while due to high RAM usage.

There are instances where errors occur during the XY Plot or when I plot the XY graph for the second time after the first plot. These errors often indicate that there is insufficient memory or sometimes there is no specific error message, just a prompt saying [ Press any key to continue... ]

4 (1) 4 (2) 4 (4) 4 (8) 4 (9)

This problem is similar to the 'Load settings from' button. If I click on it before performing XY Merge, the problem arises.

I also have another issue: the 'Unload Model' button is not functioning.

Ketorami commented 9 months ago

I understand the problem now. It seems that the issue stems from the extension Kohya Hires.Fix. When you click on Dis-activate, the problem does not occur. However, i would like to use it alongside Kohya Hires.Fix. Is there a way to fix this, or is it a problem with the Hires.Fix extension? Is there a way to make SuperMerge work with Kohya Hires.Fix?

nCoderGit commented 8 months ago

Your VRAM is likely getting exhausted due to the high resolution with Hires.Fix (and/or batch size set too high).

The newer NVIDIA drivers have a feature called CUDA - Sysmem Fallback Policy which will start offloading to system RAM when VRAM reaches a certain threshold. It prevents out-of-memory crashes (at least until your RAM is also full), but system memory is awefully slow in comparison, like 10x and more. And all you see as a user is some heavy slowdown without any real indicator to tell you if the graphics driver is actually offloading at the moment.

I'd choose Prefer No Sysmem Fallback in NVIDIA's control panel (only for [webui path]\venv\Scripts\python.exe, or globally to disable fallback for all applications) to get an instant OOM because then you know what resolution and batch size your 8GB 2060 Super can actually handle. However, keep in mind that this will put a hard 8GB limit to what models you can load, so if you get OOMs while training or using SDXL you might wanna enable it again.

hako-mikan commented 6 months ago

If the problem only occurs when using kohya.hiresfix, then it is not an issue with supermerger. This is because the function to generate images uses the Web-UI's feature, not SuperMerger's.

Ketorami commented 6 months ago

Alright then, if that's the case, I will close this.