CUDA error arch_list[-1] += '+PTX' IndexError: list index out of range

[Klendat]

I installed all the requirements and when I try to run

python main.py --text "a hamburger" --workspace trial -O

the error is

File "C:\Users\Klendat\anaconda3\envs\dream3d\lib\site-packages\torch\utils\cpp_extension.py", line 1780, in _get_cuda_arch_flags arch_list[-1] += '+PTX' IndexError: list index out of range

Running cuda toolkit 11.6

[ckcollab]

I'm also getting this in Windows 11 with Cuda 12.0, more output:

(stable-dreamfusion) C:\Users\eric\src\stable-dreamfusion>python main.py --text "a hamburger" --workspace trial -O --sd_version 1.5
Namespace(text='a hamburger', negative='', O=True, O2=False, test=False, save_mesh=False, eval_interval=10, workspace='trial', guidance='stable-diffusion', seed=0, iters=10000, lr=0.001, warm_iters=500, min_lr=0.0001, ckpt='latest', cuda_ray=True, max_steps=512, num_steps=64, upsample_steps=32, update_extra_interval=16, max_ray_batch=4096, albedo=False, albedo_iters=1000, uniform_sphere_rate=0.5, bg_radius=1.4, density_thresh=10, blob_density=10, blob_radius=0.3, fp16=True, backbone='grid', optim='adan', sd_version='1.5', hf_key=None, w=64, h=64, jitter_pose=False, bound=1, dt_gamma=0, min_near=0.1, radius_range=[1.0, 1.5], fovy_range=[40, 70], dir_text=True, suppress_face=False, angle_overhead=30, angle_front=60, lambda_entropy=0.0001, lambda_opacity=0, lambda_orient=0.01, lambda_smooth=0, gui=False, W=800, H=800, radius=3, fovy=60, light_theta=60, light_phi=0, max_spp=1)
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.0'
Traceback (most recent call last):
  File "C:\Users\eric\src\stable-dreamfusion\gridencoder\grid.py", line 10, in <module>
    import _gridencoder as _backend
ModuleNotFoundError: No module named '_gridencoder'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\eric\src\stable-dreamfusion\main.py", line 110, in <module>
    model = NeRFNetwork(opt)
  File "C:\Users\eric\src\stable-dreamfusion\nerf\network_grid.py", line 49, in __init__
    self.encoder, self.in_dim = get_encoder('tiledgrid', input_dim=3, log2_hashmap_size=16, desired_resolution=2048 * self.bound)
  File "C:\Users\eric\src\stable-dreamfusion\encoding.py", line 69, in get_encoder
    from gridencoder import GridEncoder
  File "C:\Users\eric\src\stable-dreamfusion\gridencoder\__init__.py", line 1, in <module>
    from .grid import GridEncoder
  File "C:\Users\eric\src\stable-dreamfusion\gridencoder\grid.py", line 12, in <module>
    from .backend import _backend
  File "C:\Users\eric\src\stable-dreamfusion\gridencoder\backend.py", line 31, in <module>
    _backend = load(name='_grid_encoder',
  File "C:\Users\eric\.conda\envs\stable-dreamfusion\lib\site-packages\torch\utils\cpp_extension.py", line 1284, in load
    return _jit_compile(
  File "C:\Users\eric\.conda\envs\stable-dreamfusion\lib\site-packages\torch\utils\cpp_extension.py", line 1508, in _jit_compile
    _write_ninja_file_and_build_library(
  File "C:\Users\eric\.conda\envs\stable-dreamfusion\lib\site-packages\torch\utils\cpp_extension.py", line 1610, in _write_ninja_file_and_build_library
    _write_ninja_file_to_build_library(
  File "C:\Users\eric\.conda\envs\stable-dreamfusion\lib\site-packages\torch\utils\cpp_extension.py", line 2014, in _write_ninja_file_to_build_library
    cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
  File "C:\Users\eric\.conda\envs\stable-dreamfusion\lib\site-packages\torch\utils\cpp_extension.py", line 1780, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'
IndexError: list index out of range

[kalyco]

@ckcollab @Klendat did you find a resolution for this? I'm also running CUDA 12 on Ubuntu 22.0.4. Wanted to run CUDA 11.6 for a consistent env, but looked like it wasn't supported for 22.

[ckcollab]

@kalyco no I ended up using InvokeAI which worked instantly without many problems.

[kalyco]

@ckcollab nice! Fwiw I solved it on my end. Added the environmental settings in my .bashrc

export TORCH_CUDA_ARCH_LIST="YOUR_GPUs_CC+PTX" export FORCE_CUDA=1 export CUDA_HOME=/usr/local/cuda-12.0

[ahernandezmiro]

same issue for me in windows 10, using miniconda3

[furyNerd]

hi, @kalyco, can you give some example for helping understanding what is "YOUR_GPUs_CC+PTX" ? thanks so much

[karellat]

Not sure, what @kalyco means. But it seems connected to https://pytorch.org/docs/stable/generated/torch.cuda.get_arch_list.html

[frederiknolte]

He meant the compute capability of the GPU. My GPU has compute capability 7.5, so I set export TORCH_CUDA_ARCH_LIST="7.5+PTX" and it worked