search

arenasys / stable-diffusion-webui-model-toolkit

A Multipurpose toolkit for managing, editing and creating models.
MIT License
508 stars 34 forks source link

Bad memory management #9

Closed DevilaN closed 1 year ago

DevilaN commented 1 year ago

Each time loading model and saving "clean" version takes more and more RAM. I can barely clean up two models before hitting ceiling with swap partition being full. Possible memory leak?

arenasys commented 1 year ago

Are you seeing any error? Theres a memory leak that can happen if the webui destroys and recreates the extension (like when doing "Reload UI"), leaving the loaded model still in memory.

DevilaN commented 1 year ago

Nothing error-like in console. I do not have any python background so I cannot help with debug without any help. One thing which I suspect is that garbage collector is not running properly. I am using https://github.com/AbdBarho/stable-diffusion-webui-docker repo to run SD. There has been some issues reported with Python 3.10.x like this one: https://github.com/python/cpython/issues/98801 and @AbdBarho's repo is using Python v3.10.9.

Maybe there could be some simple fix like calling gc.collect() in strategic places (where?) to test it?

arenasys commented 1 year ago

Can try, i added the GC calls where needed in 4c99665a9723dc96b976dca1648d9a9487f547e5. Pythons normal behavior is to hold onto memory after the model is freed, and use this reserved memory before requesting more.

DevilaN commented 1 year ago

Nope. Still same behavior :( But I've found some pattern during tests. It seems that python has some memory threshold before it is freeing memory and for unknown reasons it is also taking swap space into account as available memory. So it is exhausting 16GB regular RAM, then using swap (20GB reserved) until it hits some limit and then memory do not grow anymore.

This of course is not your code fault, so I am closing this issue, but many people are using docker or other container solutions to run SD, so it would be nice if there is some info for them about this python memory management issues.

Now I have to make some research how to force python not to use entire RAM+SWAP for memory management purposes :(

Thank you one more time!

junekhan commented 10 months ago

@DevilaN Hi there. Have you made any progress in improving memory management?