Internal assertion in CUDA Graph Utils

[MarcSeebold]

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. Clone repo
2. Run ./start.sh

Expected behavior Image generation works

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu 22.04.2 LTS
• Browser: Firefox 110 (x64)
• Version: Easy Diffusion 2.5

GPU Info NVIDIA RTX 1080 NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0

Error Message

Error: Unexpected Read Error:

Error:undefined
EventInfo: {
    "reader": {}
}

Error: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus21889

Additional context Related issues:

• https://github.com/pytorch/torchdynamo/issues/1743
• https://github.com/huggingface/transformers/issues/20752
• https://discuss.pytorch.org/t/reporting-bug-internal-assert-failed-at-c-w-b-windows-pytorch-aten-src-aten-native-cuda-reduce-cuh-929-please-report-a-bug-to-pytorch/149795

Stacktrace

11:46:13.620 ERROR cuda:0 Traceback (most recent call last):                                                                             task_manager.py:295
  File "/home/marc/Programs/stable-diffusion-ui/ui/easydiffusion/task_manager.py", line 287, in thread_render                                               
    task.response = renderer.make_images(task.render_request, task.task_data, task.buffer_queue, task.temp_images, step_callback)                           
  File "/home/marc/Programs/stable-diffusion-ui/ui/easydiffusion/renderer.py", line 34, in make_images                                                      
    images, seeds = make_images_internal(req, task_data, data_queue, task_temp_images, step_callback)                                                       
  File "/home/marc/Programs/stable-diffusion-ui/ui/easydiffusion/renderer.py", line 50, in make_images_internal                                             
    images, user_stopped = generate_images_internal(req, task_data, data_queue, task_temp_images, step_callback,                                            
task_data.stream_image_progress)                                                                                                                            
  File "/home/marc/Programs/stable-diffusion-ui/ui/easydiffusion/renderer.py", line 70, in generate_images_internal                                         
    images = generate_images(context, callback=callback, **req.dict())                                                                                      
  File "/home/marc/Programs/stable-diffusion-ui/installer_files/env/lib/python3.8/site-packages/sdkit/generate/image_generator.py", line                    
43, in generate_images                                                                                                                                      
    seed_everything(seed)                                                                                                                                   
  File "/home/marc/Programs/stable-diffusion-ui/installer_files/env/lib/python3.8/site-packages/pytorch_lightning/utilities/seed.py",                       
line 69, in seed_everything                                                                                                                                 
    torch.manual_seed(seed)                                                                                                                                 
  File "/home/marc/Programs/stable-diffusion-ui/installer_files/env/lib/python3.8/site-packages/torch/random.py", line 40, in                               
manual_seed                                                                                                                                                 
    torch.cuda.manual_seed_all(seed)                                                                                                                        
  File "/home/marc/Programs/stable-diffusion-ui/installer_files/env/lib/python3.8/site-packages/torch/cuda/random.py", line 113, in                         
manual_seed_all                                                                                                                                             
    _lazy_call(cb, seed_all=True)                                                                                                                           
  File "/home/marc/Programs/stable-diffusion-ui/installer_files/env/lib/python3.8/site-packages/torch/cuda/__init__.py", line 165, in                       
_lazy_call                                                                                                                                                  
    callable()                                                                                                                                              
  File "/home/marc/Programs/stable-diffusion-ui/installer_files/env/lib/python3.8/site-packages/torch/cuda/random.py", line 111, in cb                      
    default_generator.manual_seed(seed)                                                                                                                     
RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph                     
CaptureStatus21889

[allen-adastra]

I too have encountered this issue with stable-baselines3

RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus21993

Cuda version:

(poetry) nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

[Eyshika]

Any solution @allen-adastra ?

[JeLuF]

Can you please provide a complete log of the output of ./start.sh?