RuntimeError: CUDA error: device-side assert triggered

[Yazooliu]

Hi Team，

I meet the error during run rum_demo.py.

OS Environment: Centos python version: Python 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.

StartUp method: nohup python run_demo.py

Error info in nohup.out Traceback (most recent call last): File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/gradio/routes.py", line 442, in run_predict output = await app.get_blocks().process_api( File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/gradio/blocks.py", line 1392, in process_api result = await self.call_function( File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/gradio/blocks.py", line 1097, in call_function prediction = await anyio.to_thread.run_sync( File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run result = context.run(func, args) File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/gradio/utils.py", line 703, in wrapper response = f(args, **kwargs) File "/home/llm/app/CodeGeeX2-6B/source/CodeGeeX2/run_demo_CodeGeeX2.py", line 117, in predict set_random_seed(seed) File "/home/llm/app/CodeGeeX2-6B/source/CodeGeeX2/run_demo_CodeGeeX2.py", line 104, in set_random_seed torch.manual_seed(seed) File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/torch/random.py", line 40, in manual_seed torch.cuda.manual_seed_all(seed) File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/torch/cuda/random.py", line 113, in manual_seed_all _lazy_call(cb, seed_all=True) File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/torch/cuda/init.py", line 183, in _lazy_call callable() File "/home/llm/miniconda3/envs/CodeGeeX2_env/lib/python3.10/site-packages/torch/cuda/random.py", line 111, in cb default_generator.manual_seed(seed) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

I try the method as following: add CUDA_LAUNCH_BLOCKING=1 in run_demo.py

I am not sure the reason and the method is correct or not?? If correct, could I pull the PR?

BestRegards Yazhou

[Stanislas0]

Which torch version do you use? And the CUDA version?

[Yazooliu]

Which torch version do you use? And the CUDA version?

show you the pip list detail info as following: Package Version

---

accelerate 0.21.0 aiofiles 23.1.0 aiohttp 3.8.5 aiosignal 1.3.1 altair 5.0.1 annotated-types 0.5.0 anyio 3.7.1 asttokens 2.2.1 async-timeout 4.0.2 attrs 23.1.0 backcall 0.2.0 certifi 2023.7.22 charset-normalizer 3.2.0 click 8.1.6 cmake 3.27.0 contourpy 1.1.0 cpm-kernels 1.0.11 cycler 0.11.0 decorator 5.1.1 exceptiongroup 1.1.2 executing 1.2.0 fastapi 0.100.0 ffmpy 0.3.1 filelock 3.12.2 fonttools 4.41.1 frozenlist 1.4.0 fsspec 2023.6.0 gradio 3.39.0 gradio_client 0.3.0 h11 0.14.0 httpcore 0.17.3 httpx 0.24.1 huggingface-hub 0.16.4 idna 3.4 ipython 8.14.0 jedi 0.18.2 Jinja2 3.1.2 jsonschema 4.18.4 jsonschema-specifications 2023.7.1 kiwisolver 1.4.4 latex2mathml 3.76.0 linkify-it-py 2.0.2 lit 16.0.6 Markdown 3.4.4 markdown-it-py 2.2.0 MarkupSafe 2.1.3 matplotlib 3.7.2 matplotlib-inline 0.1.6 mdit-py-plugins 0.3.3 mdtex2html 1.2.0 mdurl 0.1.2 mpmath 1.3.0 multidict 6.0.4 networkx 3.1 numpy 1.25.1 nvidia-cublas-cu11 11.10.3.66 nvidia-cuda-cupti-cu11 11.7.101 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cudnn-cu11 8.5.0.96 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.2.10.91 nvidia-cusolver-cu11 11.4.0.1 nvidia-cusparse-cu11 11.7.4.91 nvidia-nccl-cu11 2.14.3 nvidia-nvtx-cu11 11.7.91 orjson 3.9.2 packaging 23.1 pandas 2.0.3 parso 0.8.3 pexpect 4.8.0 pickleshare 0.7.5 Pillow 10.0.0 pip 23.1.2 prompt-toolkit 3.0.39 protobuf 4.23.4 psutil 5.9.5 ptyprocess 0.7.0 pure-eval 0.2.2 pydantic 1.10.9 pydantic_core 2.3.0 pydub 0.25.1 Pygments 2.15.1 pyparsing 3.0.9 python-dateutil 2.8.2 python-multipart 0.0.6 pytz 2023.3 PyYAML 6.0.1 referencing 0.30.0 regex 2023.6.3 requests 2.31.0 rpds-py 0.9.2 safetensors 0.3.1 semantic-version 2.10.0 sentencepiece 0.1.99 setuptools 67.8.0 six 1.16.0 sniffio 1.3.0 sse-starlette 1.6.1 stack-data 0.6.2 starlette 0.27.0 sympy 1.12 tokenizers 0.13.3 toolz 0.12.0 torch 2.0.1 tqdm 4.65.0 traitlets 5.9.0 transformers 4.30.2 triton 2.0.0 typing_extensions 4.7.1 tzdata 2023.3 uc-micro-py 1.0.2 urllib3 2.0.4 uvicorn 0.23.1 wcwidth 0.2.6 websockets 11.0.3 wheel 0.38.4 yarl 1.9.2

NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.6 V100 32G GPU/Card

[Yazooliu]

Which torch version do you use? And the CUDA version?

Could I PR to fix this issue ? Thanks

BestRegards Yazhou

[Yazooliu]

PR is arise to fix this issue , please review and verify

[Yazooliu]

No response ？ for my PR?