File ~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, *kwargs)
1530 return self._compiled_call_impl(args, **kwargs) # type: ignore[misc]
...
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
The cuda version is 12.2. and the environment dependency is:
I keep encountering the following error when I ran the last block of inference.ipynb:
RuntimeError Traceback (most recent call last) Cell In[14], line 1 ----> 1 output = model.generate(graph, input_tokens) 2 print(output)
File ~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:115, in context_decorator..decorate_context(*args, kwargs)
112 @functools.wraps(func)
113 def decorate_context(*args, *kwargs):
114 with ctx_factory():
--> 115 return func(args, kwargs)
File ~/3D-MoLM/model/blip2_llama_inference.py:85, in Blip2Llama.generate(self, graph_batch, text_batch, do_sample, num_beams, max_length, min_length, max_new_tokens, min_new_tokens, repetition_penalty, length_penalty, num_captions) 70 @torch.no_grad() 71 def generate( 72 self, (...) 83 num_captions=1, 84 ):
---> 85 graph_embeds, graph_masks = self.graph_encoder(*graph_batch) 86 graph_embeds = self.ln_graph(graph_embeds) 87 query_tokens = self.query_tokens.expand(graph_embeds.shape[0], -1, -1)
File ~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, *kwargs) 1530 return self._compiled_call_impl(args, **kwargs) # type: ignore[misc] ... RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.The cuda version is 12.2. and the environment dependency is:
Package Version
absl-py 2.1.0 accelerate 0.32.1 aiohttp 3.9.5 aiosignal 1.3.1 altair 5.3.0 annotated-types 0.7.0 antlr4-python3-runtime 4.9.3 asttokens 2.4.1 async-timeout 4.0.3 attrs 23.2.0 bleach 6.1.0 blinker 1.8.2 blis 0.7.11 braceexpand 0.1.7 Brotli 1.1.0 cachetools 5.3.3 catalogue 2.0.10 certifi 2024.7.4 cffi 1.16.0 cfgv 3.4.0 charset-normalizer 3.3.2 click 8.1.7 cloudpathlib 0.18.1 cmake 3.30.0 comm 0.2.2 confection 0.1.5 contextlib2 21.6.0 contexttimer 0.3.3 contourpy 1.2.1 cycler 0.12.1 cymem 2.0.8 debugpy 1.8.2 decorator 5.1.1 decord 0.6.0 deepspeed 0.12.2 distlib 0.3.8 docker-pycreds 0.4.0 einops 0.8.0 exceptiongroup 1.2.0 executing 2.0.1 fairscale 0.4.4 filelock 3.15.4 fonttools 4.53.1 frozenlist 1.4.1 fsspec 2024.6.1 ftfy 6.2.0 gitdb 4.0.11 GitPython 3.1.43 gmpy2 2.1.5 h2 4.1.0 hjson 3.1.0 hpack 4.0.0 huggingface-hub 0.23.4 hyperframe 6.0.1 identify 2.6.0 idna 3.7 imageio 2.34.2 importlib_metadata 8.0.0 iopath 0.1.10 ipykernel 6.29.5 ipython 8.26.0 jedi 0.19.1 Jinja2 3.1.4 joblib 1.4.2 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 jupyter_client 8.6.2 jupyter_core 5.7.2 kaggle 1.6.14 kiwisolver 1.4.5 langcodes 3.4.0 language_data 1.2.0 lazy_loader 0.4 lightning-utilities 0.11.3.post0 lit 18.1.8 lmdb 1.5.1 marisa-trie 1.2.0 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.9.1 matplotlib-inline 0.1.7 mdurl 0.1.2 ml_collections 0.1.1 mpmath 1.3.0 multidict 6.0.5 murmurhash 1.0.10 nest_asyncio 1.6.0 networkx 3.3 ninja 1.11.1.1 nodeenv 1.9.1 numpy 1.26.4 nvidia-cublas-cu11 11.10.3.66 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu11 11.7.101 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu11 8.5.0.96 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu11 10.9.0.58 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu11 10.2.10.91 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu11 11.4.0.1 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu11 11.7.4.91 nvidia-cusparse-cu12 12.1.0.106 nvidia-ml-py 12.555.43 nvidia-nccl-cu11 2.14.3 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.5.82 nvidia-nvtx-cu11 11.7.91 nvidia-nvtx-cu12 12.1.105 omegaconf 2.3.0 opencv-python-headless 4.5.5.64 opendatasets 0.1.22 packaging 24.1 pandas 2.2.2 parso 0.8.4 peft 0.11.1 pexpect 4.9.0 pickleshare 0.7.5 pillow 10.4.0 pip 24.1.2 platformdirs 4.2.2 plotly 5.22.0 portalocker 2.10.0 pre-commit 3.7.1 preshed 3.0.9 prompt_toolkit 3.0.47 protobuf 5.28.0rc1 psutil 6.0.0 ptyprocess 0.7.0 pure-eval 0.2.2 py-cpuinfo 9.0.0 pyarrow 16.1.0 pycocoevalcap 1.2 pycocotools 2.0.8 pycparser 2.22 pydantic 2.8.2 pydantic_core 2.20.1 pydeck 0.9.1 Pygments 2.18.0 pynvml 11.5.3 pyparsing 3.1.2 PySocks 1.7.1 python-dateutil 2.9.0 python-magic 0.4.27 python-slugify 8.0.4 pytorch-lightning 2.0.7 pytz 2024.1 PyYAML 6.0.1 pyzmq 26.0.3 rdkit 2024.3.3 referencing 0.35.1 regex 2024.5.15 requests 2.32.3 rich 13.7.1 rpds-py 0.19.0 safetensors 0.4.3 salesforce-lavis 1.0.2 scikit-image 0.24.0 scikit-learn 1.5.1 scipy 1.14.0 sentencepiece 0.2.0 sentry-sdk 2.9.0 setproctitle 1.3.3 setuptools 69.5.1 shellingham 1.5.4 six 1.16.0 smart-open 7.0.4 smmap 5.0.1 spacy 3.7.5 spacy-legacy 3.0.12 spacy-loggers 1.0.5 srsly 2.4.8 stack-data 0.6.2 streamlit 1.36.0 sympy 1.13.0 tenacity 8.5.0 tensorboardX 2.6.2.2 text-unidecode 1.3 thinc 8.2.5 threadpoolctl 3.5.0 tifffile 2024.7.2 timm 0.4.12 tokenizers 0.19.1 toml 0.10.2 toolz 0.12.1 torch 2.3.1 torch_geometric 2.5.3 torchaudio 2.3.0 torchmetrics 1.4.0.post0 torchvision 0.18.0 tornado 6.4.1 tqdm 4.66.4 traitlets 5.14.3 transformers 4.42.4 triton 2.3.1 typer 0.12.3 typing_extensions 4.12.2 tzdata 2024.1 unicore 0.0.1 urllib3 2.2.2 virtualenv 20.26.3 wandb 0.17.4 wasabi 1.1.3 watchdog 4.0.1 wcwidth 0.2.13 weasel 0.4.1 webdataset 0.2.86 webencodings 0.5.1 wheel 0.43.0 wrapt 1.16.0 yarl 1.9.4 zipp 3.19.2 zstandard 0.23.0
I have tried to switch to the pytorch version of 2.3.0 and have the error:
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling
cublasGemmStridedBatchedEx(handle, opa, opb, (int)m, (int)n, (int)k, (void*)&falpha, a, CUDA_R_16BF, (int)lda, stridea, b, CUDA_R_16BF, (int)ldb, strideb, (void*)&fbeta, c, CUDA_R_16BF, (int)ldc, stridec, (int)num_batches, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)It works if I use the cpu instead of gpu. However, cpu generates responses too slow.
Could you provide some advice on how to resolve this error? Thanks.
btw: I tried to install the environment with the requirements.txt but the pip failed.