search

lllyasviel / stable-diffusion-webui-forge

GNU Affero General Public License v3.0
7.79k stars 746 forks source link

The last days Forge on Runpod always OutOfMemoryError: #1984

Open carlmoss22 opened 23 hours ago

carlmoss22 commented 23 hours ago

After 1 or 2 generations i always get this error on runpod. it worked before. seems like it's not clearing the memory.

OutOfMemoryError: CUDA out of memory. Tried to allocate 268.00 MiB. GPU 0 has a total capacity of 23.64 GiB of which 34.81 MiB is free. Process 1193254 has 23.60 GiB memory in use. Of the allocated memory 22.60 GiB is allocated by PyTorch, and 528.55 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

MysticDaedra commented 14 hours ago

I'm getting OOMs running on local using Cuda allocator, have to restart every couple of generations.

alienzed commented 11 hours ago

getting lots of crashes... reverting to f5190349b43b6da91b039d5fa6cb85e23d0e5830 restores stability in my case.

carlmoss22 commented 3 hours ago

getting lots of crashes... reverting to f519034 restores stability in my case.

cool, but how to do this on runpod?