search

comfyanonymous / ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.
https://www.comfy.org/
GNU General Public License v3.0
56.21k stars 5.95k forks source link

flux still OOM #4894

Closed TigerHH6866 closed 2 months ago

TigerHH6866 commented 2 months ago

Expected Behavior

flux+cn+lora, when ksamper it OOM

sys info

Total VRAM 24253 MB, total RAM 773393 MB pytorch version: 2.1.2+cu121 xformers version: 0.0.23.post1 Set vram state to: HIGH_VRAM Device: cuda:0 NVIDIA GeForce RTX 3090 : cudaMallocAsync Using xformers cross attention

flux.1_t2i_cn_lora.json

Actual Behavior

just run a flux workflow ,1 lora and 1 cn

Steps to Reproduce

just run a flux workflow ,1 lora and 1 cn

Debug Logs

model_path is /root/autodl-tmp/ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/lllyasviel/Annotators/ControlNetHED.pth
Using xformers attention in VAE
Using xformers attention in VAE
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLUX
Requested to load FluxClipModel_
Loading 1 new model
loaded completely 0.0 4777.53759765625 True
clip missing: ['text_projection.weight']
Requested to load FluxClipModel_
Loading 1 new model
Requested to load FluxClipModel_
Loading 1 new model
Requested to load Flux
Loading 1 new model
ERROR lora diffusion_model.double_blocks.10.img_mod.lin.weight Allocation on device 0 would exceed allowed memory. (out of memory)
!!! Exception during processing !!! Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated     : 23.21 GiB
Requested               : 32.00 MiB
Device limit            : 23.68 GiB
Free (according to CUDA): 20.69 MiB
PyTorch limit (set by user-supplied memory fraction)
                        : 17179869184.00 GiB
Traceback (most recent call last):
  File "/root/autodl-tmp/ComfyUI/execution.py", line 323, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "/root/autodl-tmp/ComfyUI/execution.py", line 198, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "/root/autodl-tmp/ComfyUI/execution.py", line 169, in _map_node_over_list
    process_inputs(input_dict, i)
  File "/root/autodl-tmp/ComfyUI/execution.py", line 158, in process_inputs
    results.append(getattr(obj, func)(**inputs))
  File "/root/autodl-tmp/ComfyUI/custom_nodes/x-flux-comfyui/nodes.py", line 348, in sampling
    mm.load_model_gpu(model)
  File "/root/autodl-tmp/ComfyUI/comfy/model_management.py", line 559, in load_model_gpu
    return load_models_gpu([model])
  File "/root/autodl-tmp/ComfyUI/comfy/model_management.py", line 545, in load_models_gpu
    cur_loaded_model = loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
  File "/root/autodl-tmp/ComfyUI/comfy/model_management.py", line 327, in model_load
    raise e
  File "/root/autodl-tmp/ComfyUI/comfy/model_management.py", line 323, in model_load
    self.real_model = self.model.patch_model(device_to=patch_model_to, lowvram_model_memory=lowvram_model_memory, load_weights=load_weights, force_patch_weights=force_patch_weights)
  File "/root/autodl-tmp/ComfyUI/comfy/model_patcher.py", line 427, in patch_model
    self.load(device_to, lowvram_model_memory=lowvram_model_memory, force_patch_weights=force_patch_weights, full_load=full_load)
  File "/root/autodl-tmp/ComfyUI/comfy/model_patcher.py", line 393, in load
    self.patch_weight_to_device(weight_key, device_to=device_to)
  File "/root/autodl-tmp/ComfyUI/comfy/model_patcher.py", line 324, in patch_weight_to_device
    out_weight = comfy.float.stochastic_rounding(out_weight, weight.dtype, seed=string_to_seed(key))
  File "/root/autodl-tmp/ComfyUI/comfy/float.py", line 63, in stochastic_rounding
    output[i:i+slice_size].copy_(manual_stochastic_round_to_float8(value[i:i+slice_size], dtype, generator=generator))
  File "/root/autodl-tmp/ComfyUI/comfy/float.py", line 37, in manual_stochastic_round_to_float8
    abs_x[:] = calc_mantissa(abs_x, exponent, normal_mask, MANTISSA_BITS, EXPONENT_BIAS, generator=generator)
  File "/root/autodl-tmp/ComfyUI/comfy/float.py", line 5, in calc_mantissa
    mantissa_scaled = torch.where(
torch.cuda.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated     : 23.21 GiB
Requested               : 32.00 MiB
Device limit            : 23.68 GiB
Free (according to CUDA): 20.69 MiB
PyTorch limit (set by user-supplied memory fraction)
                        : 17179869184.00 GiB

Got an OOM, unloading all loaded models.
Prompt executed in 10.56 seconds

Other

No response

comfyanonymous commented 2 months ago

Update pytorch to 2.4.1

ZeroCool22 commented 1 month ago

Update pytorch to 2.4.1

Steps to do it?

(base) C:\Users\ZeroCool22\Desktop\SwarmUI\dlbackend\comfy\ComfyUI>pip list
Package                       Version
----------------------------- ---------------
alabaster                     0.7.12
anaconda-client               1.11.1
anaconda-navigator            2.4.0
anaconda-project              0.11.1
annotated-types               0.6.0
anyio                         3.5.0
appdirs                       1.4.4
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
arrow                         1.2.3
astroid                       2.14.2
astropy                       5.1
asttokens                     2.0.5
atomicwrites                  1.4.0
attrs                         22.1.0
Automat                       20.2.0
autopep8                      1.6.0
Babel                         2.11.0
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
backports.tempfile            1.0
backports.weakref             1.0.post1
bcrypt                        3.2.0
beautifulsoup4                4.11.1
binaryornot                   0.4.4
black                         22.6.0
bleach                        4.1.0
bokeh                         2.4.3
Bottleneck                    1.3.5
brotlipy                      0.7.0
certifi                       2024.7.4
cffi                          1.15.1
chardet                       4.0.0
charset-normalizer            2.0.4
click                         8.0.4
cloudpickle                   2.0.0
clyent                        1.2.2
colorama                      0.4.6
colorcet                      3.0.1
comm                          0.1.2
conda                         23.1.0
conda-build                   3.23.3
conda-content-trust           0.1.3
conda-pack                    0.6.0
conda-package-handling        2.0.2
conda_package_streaming       0.7.0
conda-repo-cli                1.0.27
conda-token                   0.4.0
conda-verify                  3.4.2
constantly                    15.1.0
contourpy                     1.0.5
cookiecutter                  1.7.3
cryptography                  39.0.1
cssselect                     1.1.0
cycler                        0.11.0
cytoolz                       0.12.0
daal4py                       2023.0.2
dask                          2022.7.0
datashader                    0.14.4
datashape                     0.5.4
debugpy                       1.5.1
decorator                     5.1.1
defusedxml                    0.7.1
diff-match-patch              20200713
dill                          0.3.6
distributed                   2022.7.0
docstring-to-markdown         0.11
docutils                      0.18.1
entrypoints                   0.4
et-xmlfile                    1.1.0
executing                     0.8.3
fastapi                       0.103.0
fastjsonschema                2.16.2
filelock                      3.9.0
flake8                        6.0.0
Flask                         2.2.2
flit_core                     3.6.0
fonttools                     4.25.0
fsspec                        2022.11.0
future                        0.18.3
gensim                        4.3.0
glob2                         0.7
greenlet                      2.0.1
h5py                          3.7.0
HeapDict                      1.0.1
holoviews                     1.15.4
huggingface-hub               0.10.1
hvplot                        0.8.2
hyperlink                     21.0.0
idna                          3.4
imagecodecs                   2021.8.26
imageio                       2.26.0
imagesize                     1.4.1
imbalanced-learn              0.10.1
importlib-metadata            4.11.3
incremental                   21.3.0
inflection                    0.5.1
iniconfig                     1.1.1
intake                        0.6.7
intervaltree                  3.1.0
ipykernel                     6.19.2
ipython                       8.10.0
ipython-genutils              0.2.0
ipywidgets                    7.6.5
isort                         5.9.3
itemadapter                   0.3.0
itemloaders                   1.0.4
itsdangerous                  2.0.1
jedi                          0.18.1
jellyfish                     0.9.0
Jinja2                        3.1.2
jinja2-time                   0.2.0
jmespath                      0.10.0
joblib                        1.1.1
json5                         0.9.6
jsonschema                    4.17.3
jupyter                       1.0.0
jupyter_client                7.3.4
jupyter-console               6.6.2
jupyter_core                  5.2.0
jupyter-server                1.23.4
jupyterlab                    3.5.3
jupyterlab-pygments           0.1.2
jupyterlab_server             2.19.0
jupyterlab-widgets            1.0.0
keyring                       23.4.0
kiwisolver                    1.4.4
lazy-object-proxy             1.6.0
libarchive-c                  2.9
llvmlite                      0.39.1
locket                        1.0.0
lxml                          4.9.1
lz4                           3.1.3
Markdown                      3.4.1
MarkupSafe                    2.1.1
matplotlib                    3.7.0
matplotlib-inline             0.1.6
mccabe                        0.7.0
menuinst                      1.4.19
mistune                       0.8.4
mkl-fft                       1.3.1
mkl-random                    1.2.2
mkl-service                   2.4.0
mock                          4.0.3
mpmath                        1.2.1
msgpack                       1.0.3
multipledispatch              0.6.0
munkres                       1.1.4
mypy-extensions               0.4.3
navigator-updater             0.3.0
nbclassic                     0.5.2
nbclient                      0.5.13
nbconvert                     6.5.4
nbformat                      5.7.0
nest-asyncio                  1.5.6
networkx                      2.8.4
nltk                          3.7
notebook                      6.5.2
notebook_shim                 0.2.2
numba                         0.56.4
numexpr                       2.8.4
numpy                         1.23.5
numpydoc                      1.5.0
openpyxl                      3.0.10
packaging                     22.0
pandas                        1.5.3
pandocfilters                 1.5.0
panel                         0.14.3
param                         1.12.3
paramiko                      2.8.1
parsel                        1.6.0
parso                         0.8.3
partd                         1.2.0
pathlib                       1.0.1
pathspec                      0.10.3
patsy                         0.5.3
pep8                          1.7.1
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        9.4.0
pip                           24.2
pkginfo                       1.8.3
platformdirs                  2.5.2
plotly                        5.9.0
pluggy                        1.0.0
ply                           3.11
pooch                         1.4.0
poyo                          0.5.0
prometheus-client             0.14.1
prompt-toolkit                3.0.36
Protego                       0.1.16
psutil                        5.9.0
ptyprocess                    0.7.0
pure-eval                     0.2.2
py                            1.11.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycodestyle                   2.10.0
pycosat                       0.6.4
pycparser                     2.21
pyct                          0.5.0
pycurl                        7.45.1
pydantic                      2.5.3
pydantic_core                 2.14.6
PyDispatcher                  2.0.5
pydocstyle                    6.3.0
pyerfa                        2.0.0
pyflakes                      3.0.1
Pygments                      2.11.2
PyHamcrest                    2.0.2
PyJWT                         2.4.0
pylint                        2.16.2
pylint-venv                   2.3.0
pyls-spyder                   0.4.0
PyNaCl                        1.5.0
pyodbc                        4.0.34
pyOpenSSL                     23.0.0
pyparsing                     3.0.9
PyQt5                         5.15.7
PyQt5-sip                     12.11.0
PyQtWebEngine                 5.15.4
pyrsistent                    0.18.0
PySocks                       1.7.1
pytest                        7.1.2
python-dateutil               2.8.2
python-lsp-black              1.2.1
python-lsp-jsonrpc            1.0.0
python-lsp-server             1.7.1
python-slugify                5.0.2
python-snappy                 0.6.1
pytoolconfig                  1.2.5
pytz                          2022.7
pyviz-comms                   2.0.2
PyWavelets                    1.4.1
pywin32                       305.1
pywin32-ctypes                0.2.0
pywinpty                      2.0.10
PyYAML                        6.0
pyzmq                         23.2.0
QDarkStyle                    3.0.2
qstylizer                     0.2.2
QtAwesome                     1.2.2
qtconsole                     5.4.0
QtPy                          2.2.0
queuelib                      1.5.0
regex                         2022.7.9
requests                      2.28.1
requests-file                 1.5.1
rope                          1.7.0
Rtree                         1.0.1
ruamel.yaml                   0.17.21
ruamel.yaml.clib              0.2.6
ruamel-yaml-conda             0.17.21
scikit-image                  0.19.3
scikit-learn                  1.2.1
scikit-learn-intelex          20230228.214818
scipy                         1.10.0
Scrapy                        2.8.0
seaborn                       0.12.2
Send2Trash                    1.8.0
service-identity              18.1.0
setuptools                    65.6.3
sip                           6.6.2
six                           1.16.0
smart-open                    5.2.1
sniffio                       1.2.0
snowballstemmer               2.2.0
sortedcontainers              2.4.0
soupsieve                     2.3.2.post1
Sphinx                        5.0.2
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
spyder                        5.4.1
spyder-kernels                2.4.1
SQLAlchemy                    1.4.39
stack-data                    0.2.0
starlette                     0.27.0
statsmodels                   0.13.5
sympy                         1.11.1
tables                        3.7.0
tabulate                      0.8.10
TBB                           0.2
tblib                         1.7.0
tenacity                      8.0.1
terminado                     0.17.1
text-unidecode                1.3
textdistance                  4.2.1
threadpoolctl                 2.2.0
three-merge                   0.1.1
tifffile                      2021.7.2
tinycss2                      1.2.1
tldextract                    3.2.0
tokenizers                    0.11.4
toml                          0.10.2
tomli                         2.0.1
tomlkit                       0.11.1
toolz                         0.12.0
torch                         2.4.0+cu124
torchaudio                    2.4.0+cu124
torchvision                   0.19.0+cu124
tornado                       6.1
tqdm                          4.64.1
traitlets                     5.7.1
transformers                  4.24.0
Twisted                       22.2.0
twisted-iocpsupport           1.0.2
typing_extensions             4.11.0
ujson                         5.4.0
Unidecode                     1.2.0
urllib3                       1.26.14
w3lib                         1.21.0
watchdog                      2.1.6
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              0.58.0
Werkzeug                      2.2.2
whatthepatch                  1.0.2
wheel                         0.38.4
widgetsnbextension            3.5.2
win-inet-pton                 1.1.0
wincertstore                  0.2
wrapt                         1.14.1
xarray                        2022.11.0
xformers                      0.0.28.dev887
xlwings                       0.29.1
yapf                          0.31.0
zict                          2.1.0
zipp                          3.11.0
zope.interface                5.4.0
zstandard                     0.19.0

(base) C:\Users\ZeroCool22\Desktop\SwarmUI\dlbackend\comfy\ComfyUI>

What commend i need to run?

CUDA Windows/Linux:

conda install pytorch=2.4.1 torchvision torchaudio pytorch-cuda=12.4 -c pytorch-test -c nvidia

Or

PIP CUDA: Windows/Linux:

pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/test/cu124

hanggun commented 1 month ago

I have updated the torch to 2.4.1 and test on both 3090 and 4090, but i can not use highvram mode to use flux workflows, even without lora and controlnet, how can i use highvram