Out of memory problem - Githubissues

SoftwareLogico commented 1 year ago

I have a RTX 3060 with 12GB VRAM and it works perfect for every model including the 768 one.

but with this one even using the --lowvram parameter, I get out of memory error

set COMMANDLINE_ARGS=--listen --no-half --precision full --port 9999 --disable-safe-unpickle --deepdanbooru --reinstall-torch --reinstall-xformers --enable-insecure-extension-access --lowvram

ERROR:

Traceback (most recent call last): File "F:\auto1111\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "F:\auto1111\venv\lib\site-packages\gradio\blocks.py", line 1015, in process_api result = await self.call_function( File "F:\auto1111\venv\lib\site-packages\gradio\blocks.py", line 833, in call_function prediction = await anyio.to_thread.run_sync( File "F:\auto1111\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "F:\auto1111\venv\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "F:\auto1111\venv\lib\site-packages\anyio_backends_asyncio.py", line 867, in run result = context.run(func, *args) File "F:\auto1111\extensions\stable-diffusion-webui-instruct-pix2pix\scripts\instruct-pix2pix.py", line 143, in generate with torch.no_grad(), autocast("cuda"), model.ema_scope(): File "C:\Users\ramse\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 135, in enter return next(self.gen) File "F:\auto1111\modules\models\diffusion\ddpm_edit.py", line 185, in ema_scope self.model_ema.store(self.model.parameters()) File "F:\auto1111\repositories\stable-diffusion-stability-ai\ldm\modules\ema.py", line 66, in store self.collected_params = [param.clone() for param in parameters] File "F:\auto1111\repositories\stable-diffusion-stability-ai\ldm\modules\ema.py", line 66, in self.collected_params = [param.clone() for param in parameters] torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 0; 12.00 GiB total capacity; 11.14 GiB already allocated; 0 bytes free; 11.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

illtellyoulater commented 1 year ago

I think I've read on the original repo it was tested with a 18+ GB VRAM GPU... I have a 12 GB too and I'm going to try this now, let's see how it goes... but if if doesn't work I really hope someone will find a way to optimize it...

ppongpeauk commented 1 year ago

I think I've read on the original repo it was tested with a 18+ GB VRAM GPU... I have a 12 GB too and I'm going to try this now, let's see how it goes... but if if doesn't work I really hope someone will find a way to optimize it...

I'm running this extension on an RTX 2060S 8GB with xformers installed, and it seems to be working just fine for the first few generations, until I eventually get an OOM error. Perhaps the issue is a memory leak?

SoftwareLogico commented 1 year ago

I think I've read on the original repo it was tested with a 18+ GB VRAM GPU... I have a 12 GB too and I'm going to try this now, let's see how it goes... but if if doesn't work I really hope someone will find a way to optimize it...

I think I've read on the original repo it was tested with a 18+ GB VRAM GPU... I have a 12 GB too and I'm going to try this now, let's see how it goes... but if if doesn't work I really hope someone will find a way to optimize it...

I'm running this extension on an RTX 2060S 8GB with xformers installed, and it seems to be working just fine for the first few generations, until I eventually get an OOM error. Perhaps the issue is a memory leak?

for me it works just once.

when i try for 2nd time i got an OOM error.

then i have to reopen the AUTO1111 bat

mediastacks commented 1 year ago

Also started to get this very error after the recent update, can't gen anything with pix2pix now regardless of the resolution

The first version from 2 days ago was working great for me

Other models still work perfectly (txt2img, im2img), so I guess there is something about the pix2pix extension update?

(Thank you for the amazing extension btw!)

nerdyrodent commented 1 year ago

Which version of A1111 are you using? If you use commit 6cff4401 does everything work again?

SoftwareLogico commented 1 year ago

Which version of A1111 are you using? If you use commit 6cff4401 does everything work again?

I installed it today with the commit 9beb794

but everything works fine except that.

it only runs once then the next gives the OOM error.

makiritare commented 1 year ago

had the same problem and did what @nerdyrodent suggested with git reset --hard 6cff4401 and everything seems to work again, will test some more later (using a 3060 with 12gb)

Lenowin777 commented 1 year ago

Just wanted to say that I was having the same issues with out of memory errors after updating, and going back to 6cff4401 seems to have fixed it.

DenkingOfficial commented 1 year ago

Agree, there is a memory leak I can use this extension to edit image only once before OOM Looks like extension tries to load model again without freeing VRAM after successful generation

RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 8.00 GiB total capacity; 7.21 GiB already allocated; 0 bytes free; 7.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I have RTX2060S (8GB), using webui without xformers

Klace commented 1 year ago

Agree, there is a memory leak I can use this extension to edit image only once before OOM Looks like extension tries to load model again without freeing VRAM after successful generation

RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 8.00 GiB total capacity; 7.21 GiB already allocated; 0 bytes free; 7.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I have RTX2060S (8GB), using webui without xformers

Yep, I'll fix this up.

Klace commented 1 year ago

Can someone try this branch to see if it's resolved: https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix/tree/Mem_Alloc_Fix

For some reason I wasn't experiencing the same issue, but hopefully the change I made will help.

I now have it clear the cache too after generating so viewing by os should show the memory free up. Please let me know if there's still any issues and when they occur. I ran a batch of 20 images @ 2048 and then switched models, everything seems ok.

Thanks!

Klace commented 1 year ago

Also it does seem Auto's repo is having model loading issues: https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/5d14f282c2812888275902be4b552681f942dbfd

This may contribute to some funky behavior as well

DenkingOfficial commented 1 year ago

Can someone try this branch to see if it's resolved: https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix/tree/Mem_Alloc_Fix

Just updated extension to the latest version, still getting OOM error

GIF demonstration:

OOM

Klace commented 1 year ago

Hmmm that's odd... Does it happen with the regular pix2pix model?

The latest update should purge the cache and free up the memory. I'll keep looking.

On Sat, Jan 28, 2023, 1:47 PM Andrew Shershnev @.***> wrote:

Can someone try this branch to see if it's resolved: https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix/tree/Mem_Alloc_Fix

Just updated extension to the latest version, still getting OOM error

GIF demonstration:

[image: OOM] https://user-images.githubusercontent.com/38957619/215285405-cfda3d98-023b-4eb0-8916-0dcd52580982.gif

— Reply to this email directly, view it on GitHub https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix/issues/21#issuecomment-1407462075, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGGO6I7IEWZI5URTMZQQEGLWUVSVTANCNFSM6AAAAAAUI3TK2Q . You are receiving this because you commented.Message ID: @.*** .com>

DenkingOfficial commented 1 year ago

Hmmm that's odd... Does it happen with the regular pix2pix model?

Yes, it is not working too

And it looks like cache in VRAM isn't purged

mediastacks commented 1 year ago

Also updated and still getting the error

Klace commented 1 year ago

OK pushed another version. For those having the problem please let me know your videocard and VRAM

makiritare commented 1 year ago

@Klace i just tried and still after running it once it stop working, im using a nvidia Geforce RTX 3060 (12gb) Untitled

April93 commented 1 year ago

i just stumbled upon this extension and tried installing it. I'm also getting out of memory errors. I'm on a 6gb vram card (1660ti). I normally run the webui with --medvram (and sometimes with it off) and that works fine. However this extension doesn't seem to work even with --lowvram enabled. I tried downloading a pruned model from here but that didn't seem to help. Made sure to update to latest auto ui (works fine for everything else), along with just downloading the latest extension.

Edit: Someone suggested rolling the webui back to commit ee0a0da3244123cb6d2ba4097a54a1e9caccb687 and this fixes the problem for me, and I'm able to generate pics fine.

DenkingOfficial commented 1 year ago

OK pushed another version. For those having the problem please let me know your videocard and VRAM

Nope, still not working :(

RTX2060S (8 GB VRAM)

Log:

Processing 1 image(s)
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:07<00:00,  2.76it/s]
Processing 1 image(s)
Traceback (most recent call last):
  File "C:\Software\Other Software\sdautomatic111\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\Software\Other Software\sdautomatic111\venv\lib\site-packages\gradio\blocks.py", line 1015, in process_api
    result = await self.call_function(
  File "C:\Software\Other Software\sdautomatic111\venv\lib\site-packages\gradio\blocks.py", line 833, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Software\Other Software\sdautomatic111\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Software\Other Software\sdautomatic111\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\Software\Other Software\sdautomatic111\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\Software\Other Software\sdautomatic111\extensions\stable-diffusion-webui-instruct-pix2pix\scripts\instruct-pix2pix.py", line 223, in generate
    with torch.no_grad(), autocast("cuda"), model.ema_scope():
  File "C:\Software\Programming\Python310\lib\contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "C:\Software\Other Software\sdautomatic111\modules\models\diffusion\ddpm_edit.py", line 185, in ema_scope
    self.model_ema.store(self.model.parameters())
  File "C:\Software\Other Software\sdautomatic111\repositories\stable-diffusion-stability-ai\ldm\modules\ema.py", line 66, in store
    self.collected_params = [param.clone() for param in parameters]
  File "C:\Software\Other Software\sdautomatic111\repositories\stable-diffusion-stability-ai\ldm\modules\ema.py", line 66, in <listcomp>
    self.collected_params = [param.clone() for param in parameters]
RuntimeError: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 8.00 GiB total capacity; 7.19 GiB already allocated; 0 bytes free; 7.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFProcessing 1 image(s)

mediastacks commented 1 year ago

OK pushed another version. For those having the problem please let me know your videocard and VRAM

1080 Ti 11gb

Processing 1 image(s)
Traceback (most recent call last):
  File "A:\AI\Images\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "A:\AI\Images\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1015, in process_api
    result = await self.call_function(
  File "A:\AI\Images\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 833, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "A:\AI\Images\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "A:\AI\Images\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "A:\AI\Images\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "A:\AI\Images\stable-diffusion-webui\extensions\stable-diffusion-webui-instruct-pix2pix\scripts\instruct-pix2pix.py", line 223, in generate
    with torch.no_grad(), autocast("cuda"), model.ema_scope():
  File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "A:\AI\Images\stable-diffusion-webui\modules\models\diffusion\ddpm_edit.py", line 185, in ema_scope
    self.model_ema.store(self.model.parameters())
  File "A:\AI\Images\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\ema.py", line 66, in store
    self.collected_params = [param.clone() for param in parameters]
  File "A:\AI\Images\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\ema.py", line 66, in <listcomp>
    self.collected_params = [param.clone() for param in parameters]
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.00 GiB total capacity; 10.15 GiB already allocated; 0 bytes free; 10.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Mary-Usagi commented 1 year ago

OK pushed another version. For those having the problem please let me know your videocard and VRAM

I'm having a similar issue, though on 512px I can generate around 3 to 4 images before I run OOM and have to restart the UI.

NVIDIA RTX 2080 S - 8GB

gitaiuser20 commented 1 year ago

I have a RTX 3060 laptop, git reset --hard 6cff4401 fixes it and everything works fine, but the current version is still always out of memory for me, I'm not even getting it to work once like some of the others

Klace commented 1 year ago

mmm I don't see 6cff4401 in my commit history, can you guys link it?

Klace commented 1 year ago

oh it's auto's commit

He changed the way the model was loaded from my quick hack of the config to using the full configuration file, perhaps it uses more memory?

DenkingOfficial commented 1 year ago

He changed the way the model was loaded from my quick hack of the config to using the full configuration file, perhaps it uses more memory?

Yes, looks like this

VOOM108 commented 1 year ago

I just updated the extension and the issue remains. I have a 3060 12GB

paulescobar commented 1 year ago

I'm a noob and running this for the first time - so forgive if I describe something wrong. But I have the same issue on my installation. I've run other models and add-ons without problems - only this one gives me immediate error.

My system:

Windows 10 x64
Nvidia 1660 TI GPU

Latest updates via Automatic1111 Launcher

↺   Updating Extension: stable-diffusion-webui-instruct-pix2pix
✓   SUCCESS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
i   Arguments are now: --medvram --xformers --precision full --no-half --opt-split-attention

I've tried disabling those "arguments" one by one - even until none of them are present. But I keep getting errors on Pix2Pix, regardless.

Whenever I try to use Pix2Pix via Automatic1111, I get this kind of error:

SD-Webui API layer loaded
Loading weights [fbc31a67aa] from A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\models\Stable-diffusion\instruct-pix2pix-00-22000.safetensors
Creating model from config: A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\configs\instruct-pix2pix.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(1): bad_prompt
Model loaded in 6.2s (create model: 1.2s, apply weights to model: 1.9s, move model to device: 3.0s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Traceback (most recent call last):
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1015, in process_api
    result = await self.call_function(
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 833, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\extensions\stable-diffusion-webui-instruct-pix2pix\scripts\instruct-pix2pix.py", line 151, in generate
    model.eval().to(shared.device)
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\pytorch_lightning\core\mixins\device_dtype_mixin.py", line 113, in to
    return super().to(*args, **kwargs)
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 927, in to
    return self._apply(convert)
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
    module._apply(fn)
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
    module._apply(fn)
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
    module._apply(fn)
  [Previous line repeated 4 more times]
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 602, in _apply
    param_applied = fn(param)
  File "A:\DataBase\Programs\-- Automatic 1111 --\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 925, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 114.00 MiB (GPU 0; 6.00 GiB total capacity; 5.13 GiB already allocated; 0 bytes free; 5.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Connum commented 1 year ago

had the same problem and did what @nerdyrodent suggested with git reset --hard 6cff4401 and everything seems to work again, will test some more later (using a 3060 with 12gb)

When I reset to that commit, I don't see the pix2pix tab at all - am I doing something wrong?

SoftwareLogico commented 1 year ago

I tested today Jan 30 2023 and it worked. on my RTX 3060 12GB Windows. I did several test and no OOM problem with the last 667c646

Thank you I really appreciate you guys paid attention to us

mediastacks commented 1 year ago

Confirm it's working now for me too Huge thank you for fixing it

paulescobar commented 1 year ago

I'm a noob and running this for the first time - so forgive if I describe something wrong. But I have the same issue on my installation. I've run other models and add-ons without problems - only this one gives me immediate error.

My system:
Windows 10 x64

Nvidia 1660 TI GPU
Latest updates via Automatic1111 Launcher

I just updated and can also confirm Pix2Pix is working properly now. Even with all the arguments (specified in my previous post) enabled. Thank you to the developer!

Klace commented 1 year ago

Thanks here goes to auto

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/aa4688eb8345de583070ca9ddb4c6f585f06762b

Klace / stable-diffusion-webui-instruct-pix2pix

Out of memory problem #21