search

niklasku / IF-webui

DeepFloyd IF web UI
Creative Commons Zero v1.0 Universal
28 stars 4 forks source link

CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasGemmStridedBatchedExFix #6

Open Maoweicao opened 1 year ago

Maoweicao commented 1 year ago

I run it in a server with Tesla M40. In loadding model,it's show

Getting T5 embs... Traceback (most recent call last): File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/gradio/routes.py", line 437, in run_predict output = await app.get_blocks().process_api( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/gradio/blocks.py", line 1352, in process_api result = await self.call_function( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/gradio/blocks.py", line 1077, in call_function prediction = await anyio.to_thread.run_sync( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run result = context.run(func, args) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/gradio/helpers.py", line 602, in tracked_fn response = fn(args) File "/media/maoweicao/DATA1/IF-webui/webui.py", line 208, in generate_images result = dream( File "/media/maoweicao/DATA1/IF-webui/deepfloyd_if/pipelines/dream.py", line 64, in dream t5_embs = t5.get_text_embeddings(prompt) File "/media/maoweicao/DATA1/IF-webui/deepfloyd_if/modules/t5.py", line 96, in get_text_embeddings text_encoder_embs = self.model( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward output = old_forward(*args, *kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1846, in forward encoder_outputs = self.encoder( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1040, in forward layer_outputs = layer_module( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward output = old_forward(*args, *kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 673, in forward self_attention_outputs = self.layer[0]( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward output = old_forward(*args, kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 579, in forward attention_output = self.SelfAttention( File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward output = old_forward(args, kwargs) File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 509, in forward scores = torch.matmul( RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmStridedBatchedExFix(handle, opa, opb, (int)m, (int)n, (int)k, (void*)&falpha, a, CUDA_R_16BF, (int)lda, stridea, b, CUDA_R_16BF, (int)ldb, strideb, (void*)&fbeta, c, CUDA_R_16BF, (int)ldc, stridec, (int)num_batches, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)

so this model only running in with support FP16 compute card?

Maoweicao commented 1 year ago

I swith to local RTX2080TI 22G and RTX3070 8G，this is work normally.It's look like did't support no FP16 support compute card. image