search

threestudio-project / threestudio

A unified framework for 3D content generation.
Apache License 2.0
5.98k stars 463 forks source link

NerfAcc: No CUDA toolkit found. NerfAcc will be disabled. #300

Open yuchenlichuck opened 10 months ago

yuchenlichuck commented 10 months ago

When I use Pycharm, I use the code, I got NerfAcc: No CUDA toolkit found. NerfAcc will be disabled.

"/home/dubaiprince/miniconda3/envs/threestudio/bin/python /home/dubaiprince/Projects/threestudio/launch.py --config configs/dreamfusion-if.yaml --train --gpu 1 system.prompt_processor.prompt=a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes 
Global seed set to 0
[INFO] ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.
[INFO] Using 16bit Automatic Mixed Precision (AMP)
[INFO] GPU available: True (cuda), used: True
[INFO] TPU available: False, using: 0 TPU cores
[INFO] IPU available: False, using: 0 IPUs
[INFO] HPU available: False, using: 0 HPUs
[INFO] You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
[INFO] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1]
[INFO] 
  | Name       | Type                           | Params
--------------------------------------------------------------
0 | geometry   | ImplicitVolume                 | 12.6 M
1 | material   | DiffuseWithPointLightMaterial  | 0     
2 | background | NeuralEnvironmentMapBackground | 448   
3 | renderer   | NeRFVolumeRenderer             | 0     
--------------------------------------------------------------
12.6 M    Trainable params
0         Non-trainable params
12.6 M    Total params
50.419    Total estimated model params size (MB)
[INFO] Validation results will be saved to outputs/dreamfusion-if/a_zoomed_out_DSLR_photo_of_a_baby_bunny_sitting_on_top_of_a_stack_of_pancakes@20230906-210951/save
[INFO] Using prompt [a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes] and negative prompt []
[INFO] Using view-dependent prompts [side]:[a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes, side view] [front]:[a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes, front view] [back]:[a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes, back view] [overhead]:[a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes, overhead view]
[INFO] Loading Deep Floyd ...

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/model.fp16-00002-of-00002.safetensors, unet/diffusion_pytorch_model.fp16.safetensors, safety_checker/model.fp16.safetensors, text_encoder/model.fp16-00001-of-00002.safetensors]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.safetensors
If this behavior is not expected, please check your folder structure.
Loading pipeline components...: 100%|██████████| 3/3 [00:00<00:00, 19.64it/s]
[INFO] Loaded Deep Floyd!
/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:442: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 24 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:442: PossibleUserWarning: The dataloader, val_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 24 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
Epoch 0: : 0it [00:00, ?it/s]NerfAcc: No CUDA toolkit found. NerfAcc will be disabled.
Traceback (most recent call last):
  File "/home/dubaiprince/Projects/threestudio/launch.py", line 237, in <module>
    main(args, extras)
  File "/home/dubaiprince/Projects/threestudio/launch.py", line 180, in main
    trainer.fit(system, datamodule=dm, ckpt_path=cfg.resume)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 980, in _run
    results = self._run_stage()
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1023, in _run_stage
    self.fit_loop.run()
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
    self.advance()
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 355, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 133, in run
    self.advance(data_fetcher)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 219, in advance
    batch_output = self.automatic_optimization.run(trainer.optimizers[0], kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 188, in run
    self._optimizer_step(kwargs.get("batch_idx", 0), closure)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 266, in _optimizer_step
    call._call_lightning_module_hook(
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 146, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/core/module.py", line 1270, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 161, in step
    step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py", line 231, in optimizer_step
    return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/amp.py", line 76, in optimizer_step
    closure_result = closure()
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 142, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 128, in closure
    step_output = self._step_fn()
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 315, in _training_step
    training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 294, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py", line 380, in training_step
    return self.model.training_step(*args, **kwargs)
  File "/home/dubaiprince/Projects/threestudio/threestudio/systems/dreamfusion.py", line 38, in training_step
    out = self(batch)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/dubaiprince/Projects/threestudio/threestudio/systems/dreamfusion.py", line 24, in forward
    render_out = self.renderer(**batch)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/dubaiprince/Projects/threestudio/threestudio/models/renderers/nerf_volume_renderer.py", line 170, in forward
    ray_indices, t_starts_, t_ends_ = self.estimator.sampling(
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/nerfacc/estimators/occ_grid.py", line 164, in sampling
    intervals, samples = traverse_grids(
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/nerfacc/grid.py", line 128, in traverse_grids
    t_mins, t_maxs, hits = ray_aabb_intersect(rays_o, rays_d, aabbs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/nerfacc/grid.py", line 43, in ray_aabb_intersect
    t_mins, t_maxs, hits = _C.ray_aabb_intersect(
  File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/nerfacc/cuda/__init__.py", line 13, in call_cuda
    return getattr(_C, name)(*args, **kwargs)
AttributeError: 'NoneType' object has no attribute 'ray_aabb_intersect'"
bennyguo commented 10 months ago

Please make sure that path/to/cuda/bin is in $PATH and path/to/cuda/lib64 is in $LD_LIBRARY_PATH. path/to/cuda often points to /usr/local/cuda

bbbbubble commented 5 months ago

For me, instead of pip install nerface, installing from source using pip install git+https://github.com/nerfstudio-project/nerfacc.git solved the problem. Hope this helps.