search

lllyasviel / Fooocus

Focus on prompting and generating
GNU General Public License v3.0
41.51k stars 5.9k forks source link

[Bug]: "Run out of memory" in LOW_VRAM adding an aspect ratios in only the config.txt file. #2830

Closed Davide-R0 closed 6 months ago

Davide-R0 commented 6 months ago

Checklist

What happened?

Hello!

This error happen in case the aspect ratios is set in only the config.txt file, if the same aspect ratio is set even or only in modules/config.py the problem doesn't occur. Probably not GPU (GTX1650 4Gb VRAM) trouble.

The error that come out is torch.cuda.OutOfMemoryError.

Steps to reproduce the problem

  1. GPU Nvidia GTX1650, maybe enought <=6Gb VRAM gpu (not sure)
  2. add an aspect ratios in config.txt of "1920*1080"
  3. generate an image in Quality mode
  4. wait the generation, the error come at the end

What should have happened?

work, no errors

What browsers do you use to access Fooocus?

Brave, Other

Where are you running Fooocus?

Locally

What operating system are you using?

Linux 6.6.21-gentoo

Console logs

Warning: Ran out of memory when regular VAE decoding, retrying with tiled VAE decoding.
Traceback (most recent call last):
  File "/home/davide/Code/Fooocus/ldm_patched/modules/sd.py", line 241, in decode
    pixel_samples[x:x+batch_number] = torch.clamp((self.first_stage_model.decode(samples).to(self.output_device).float() + 1.0) / 2.0, min=0.0, max=1.0)
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/ldm/models/autoencoder.py", line 202, in decode
    dec = self.decoder(dec, **decoder_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/ldm/modules/diffusionmodules/model.py", line 639, in forward
    h = self.up[i_level].upsample(h)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/ldm/modules/diffusionmodules/model.py", line 72, in forward
    x = self.conv(x)
        ^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/modules/ops.py", line 42, in forward
    return super().forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 460, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.98 GiB. GPU 0 has a total capacty of 3.80 GiB of which 101.56 MiB is free. Including non-PyTorch memory, this process has 3.44 GiB memory in use. Of the allocated memory 2.82 GiB is allocated by PyTorch, and 558.95 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/davide/Code/Fooocus/modules/async_worker.py", line 913, in worker
    handler(task)
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/modules/async_worker.py", line 816, in handler
    imgs = pipeline.process_diffusion(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/modules/default_pipeline.py", line 380, in process_diffusion
    decoded_latent = core.decode_vae(vae=target_vae, latent_image=sampled_latent, tiled=tiled)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/modules/core.py", line 162, in decode_vae
    return opVAEDecode.decode(samples=latent_image, vae=vae)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/contrib/external.py", line 269, in decode
    return (vae.decode(samples["samples"]), )
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/modules/sd.py", line 244, in decode
    pixel_samples = self.decode_tiled_(samples_in)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/modules/sd.py", line 211, in decode_tiled_
    (ldm_patched.modules.utils.tiled_scale(samples, decode_fn, tile_x // 2, tile_y * 2, overlap, upscale_amount = self.downscale_ratio, output_device=self.output_device, pbar = pbar) +
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/modules/utils.py", line 418, in tiled_scale
    ps = function(s_in).to(output_device)
         ^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/modules/sd.py", line 209, in <lambda>
    decode_fn = lambda a: (self.first_stage_model.decode(a.to(self.vae_dtype).to(self.device)) + 1.0).float()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/ldm/models/autoencoder.py", line 202, in decode
    dec = self.decoder(dec, **decoder_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/ldm/modules/diffusionmodules/model.py", line 639, in forward
    h = self.up[i_level].upsample(h)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/ldm/modules/diffusionmodules/model.py", line 72, in forward
    x = self.conv(x)
        ^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/ldm_patched/modules/ops.py", line 42, in forward
    return super().forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 460, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davide/Code/Fooocus/fooocus_env/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB. GPU 0 has a total capacty of 3.80 GiB of which 101.56 MiB is free. Including non-PyTorch memory, this process has 3.44 GiB memory in use. Of the allocated memory 3.13 GiB is allocated by PyTorch, and 238.82 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Additional information

mashb1t commented 6 months ago

Won't fix (also can't fix tbh). What you can try is to manually set the VAE to FP16 with --vae-in-fp16, but this should be the default anyways, see console log. The experimental flag --vae-in-cpu might also work, see https://github.com/lllyasviel/Fooocus/blob/main/ldm_patched/modules/model_management.py#L564 In case this doesn't work: please use the supported default resolutions SDXL has been trained on for best results, see https://github.com/lllyasviel/Fooocus/pull/1617, or get better hardware so VAE fits on your VRAM.