search

lllyasviel / ControlNet

Let us control diffusion models!
Apache License 2.0
30.52k stars 2.74k forks source link

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED #711

Closed wzgpeter closed 1 month ago

wzgpeter commented 1 month ago

A100 platform cuda118 run command: python gradio_canny2image.py

log show as below:

ad. result_gallery = gr.Gallery(label='Output', show_label=False, elem_id="gallery").style(grid=2, height='auto') gradio_canny2image.py:98: GradioDeprecationWarning: The 'grid' parameter will be deprecated. Please use 'grid_cols' in the constructor instead. result_gallery = gr.Gallery(label='Output', show_label=False, elem_id="gallery").style(grid=2, height='auto') Running on local URL: http://0.0.0.0:5000 IMPORTANT: You are using gradio version 3.38.0, however version 4.29.0 is available, please upgrade.

Running on public URL: https://49b5cf058a69109bd7.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces) curr-time: 0:00:00.001951 Global seed set to 1706203096 Data shape for DDIM sampling is (1, 4, 104, 64), eta 0.0 Running DDIM Sampling with 20 timesteps DDIM Sampler: 0%| | 0/20 [00:00<?, ?it/s] Traceback (most recent call last): File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/gradio/routes.py", line 442, in run_predict output = await app.get_blocks().process_api( File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/gradio/blocks.py", line 1389, in process_api result = await self.call_function( File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/gradio/blocks.py", line 1094, in call_function prediction = await anyio.to_thread.run_sync( File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 2357, in run_sync_in_worker_thread return await future File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 864, in run result = context.run(func, args) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/gradio/utils.py", line 703, in wrapper response = f(args, kwargs) File "gradio_canny2image.py", line 59, in process samples, intermediates = ddim_sampler.sample(ddim_steps, num_samples, File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, *kwargs) File "/home/ControlNet/cldm/ddim_hacked.py", line 103, in sample samples, intermediates = self.ddim_sampling(conditioning, size, File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(args, kwargs) File "/home/ControlNet/cldm/ddim_hacked.py", line 163, in ddim_sampling outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps, File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, kwargs) File "/home/ControlNet/cldm/ddim_hacked.py", line 190, in p_sample_ddim model_t = self.model.apply_model(x, t, c) File "/home/ControlNet/cldm/cldm.py", line 337, in apply_model control = self.control_model(x=x_noisy, hint=torch.cat(cond['c_concat'], 1), timesteps=t, context=cond_txt) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(args, kwargs) File "/home/ControlNet/cldm/cldm.py", line 288, in forward guided_hint = self.input_hint_block(hint, emb, context) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, *kwargs) File "/home/ControlNet/ldm/modules/diffusionmodules/openaimodel.py", line 86, in forward x = layer(x) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 458, in forward return self._conv_forward(input, self.weight, self.bias) File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 454, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

pip list show as below:

Package Version

absl-py 2.1.0 aiofiles 23.2.1 aiohappyeyeballs 2.4.3 aiohttp 3.10.9 aiosignal 1.3.1 altair 5.4.1 annotated-types 0.7.0 antlr4-python3-runtime 4.8 anyio 4.5.0 async-timeout 4.0.3 attrs 24.2.0 cachetools 5.5.0 certifi 2024.8.30 charset-normalizer 3.4.0 click 8.1.7 contourpy 1.1.1 cycler 0.12.1 diffusers 0.30.3 einops 0.3.0 exceptiongroup 1.2.2 fastapi 0.115.0 ffmpy 0.4.0 filelock 3.16.1 fonttools 4.54.1 frozenlist 1.4.1 fsspec 2024.9.0 ftfy 6.2.3 future 1.0.0 google-auth 2.35.0 google-auth-oauthlib 1.0.0 gradio 3.38.0 gradio_client 1.3.0 grpcio 1.66.2 h11 0.14.0 httpcore 1.0.6 httpx 0.27.2 huggingface-hub 0.25.2 idna 3.10 importlib_metadata 8.5.0 importlib_resources 6.4.5 Jinja2 3.1.4 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 kiwisolver 1.4.7 lightning-utilities 0.11.7 linkify-it-py 2.0.3 Markdown 3.7 markdown-it-py 2.2.0 MarkupSafe 2.1.5 matplotlib 3.7.5 mdit-py-plugins 0.3.3 mdurl 0.1.2 mpmath 1.3.0 multidict 6.1.0 narwhals 1.9.2 networkx 3.1 numpy 1.24.4 nvidia-cublas-cu11 11.11.3.6 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu11 11.8.87 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu11 11.8.89 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu11 11.8.89 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu11 9.1.0.70 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu11 10.9.0.58 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu11 10.3.0.86 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu11 11.4.1.48 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu11 11.7.5.86 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu11 2.20.5 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.6.77 nvidia-nvtx-cu11 11.8.86 nvidia-nvtx-cu12 12.1.105 oauthlib 3.2.2 omegaconf 2.1.1 open-clip-torch 2.0.2 opencv-python 4.10.0.84 orjson 3.10.7 packaging 24.1 pandas 2.0.3 pillow 10.4.0 pip 24.2 pkgutil_resolve_name 1.3.10 propcache 0.2.0 protobuf 5.28.2 pyasn1 0.6.1 pyasn1_modules 0.4.1 pycryptodome 3.21.0 pydantic 2.9.2 pydantic_core 2.23.4 pyDeprecate 0.3.1 pydub 0.25.1 Pygments 2.18.0 pyparsing 3.1.4 python-dateutil 2.9.0.post0 python-multipart 0.0.12 pytorch-lightning 1.5.0 pytz 2024.2 PyYAML 6.0.2 referencing 0.35.1 regex 2024.9.11 requests 2.32.3 requests-oauthlib 2.0.0 rich 13.9.2 rpds-py 0.20.0 rsa 4.9 ruff 0.6.9 safetensors 0.4.5 semantic-version 2.10.0 setuptools 75.1.0 shellingham 1.5.4 six 1.16.0 sniffio 1.3.1 starlette 0.38.6 sympy 1.13.3 tensorboard 2.14.0 tensorboard-data-server 0.7.2 timm 1.0.9 tokenizers 0.12.1 tomlkit 0.12.0 torch 2.4.1+cu118 torchaudio 2.4.1+cu118 torchmetrics 1.4.2 torchvision 0.19.1+cu118 tqdm 4.66.5 transformers 4.19.2 triton 3.0.0 typer 0.12.5 typing_extensions 4.12.2 tzdata 2024.2 uc-micro-py 1.0.3 urllib3 2.2.3 uvicorn 0.31.1 wcwidth 0.2.13 websockets 11.0.3 Werkzeug 3.0.4 wheel 0.44.0 xformers 0.0.28.post1 yarl 1.14.0 zipp 3.20.2