search

nerfstudio-project / nerfstudio

A collaboration friendly studio for NeRFs
https://docs.nerf.studio
Apache License 2.0
9.55k stars 1.3k forks source link

when i use instant-ngp for train, it will new bug #2134

Open fanzz1208 opened 1 year ago

fanzz1208 commented 1 year ago

Describe the bug File "train.py", line 190, in launch [0/1821] main_func(local_rank=0, world_size=world_size, config=config) File "train.py", line 101, in train_loop trainer.train() File "/opt/conda/lib/python3.7/site-packages/nerfstudio/engine/trainer.py", line 239, in train loss, loss_dict, metrics_dict = self.train_iteration(step) File "/opt/conda/lib/python3.7/site-packages/nerfstudio/utils/profiler.py", line 93, in inner out = func(*args, kwargs) File "/opt/conda/lib/python3.7/site-packages/nerfstudio/engine/trainer.py", line 433, in trainiteration , loss_dict, metrics_dict = self.pipeline.get_train_loss_dict(step=step) File "/opt/conda/lib/python3.7/site-packages/nerfstudio/pipelines/dynamic_batch.py", line 80, in get_train_loss_dict model_outputs, loss_dict, metrics_dict = super().get_train_loss_dict(step) File "/opt/conda/lib/python3.7/site-packages/nerfstudio/utils/profiler.py", line 93, in inner out = func(*args, *kwargs) File "/opt/conda/lib/python3.7/site-packages/nerfstudio/pipelines/base_pipeline.py", line 278, in get_train_loss_dict model_outputs = self.model(ray_bundle) File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "/opt/conda/lib/python3.7/site-packages/nerfstudio/models/base_model.py", line 140, in forward return self.get_outputs(ray_bundle) File "/opt/conda/lib/python3.7/site-packages/nerfstudio/models/instant_ngp.py", line 186, in get_outputs cone_angle=self.config.cone_angle, File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "/opt/conda/lib/python3.7/site-packages/nerfstudio/model_components/ray_samplers.py", line 480, in forward alpha_thre=alpha_thre, File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, *kwargs) File "/opt/conda/lib/python3.7/site-packages/nerfacc/estimators/occ_grid.py", line 172, in sampling cone_angle=cone_angle, File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(args, kwargs) File "/opt/conda/lib/python3.7/site-packages/nerfacc/grid.py", line 128, in traverse_grids t_mins, t_maxs, hits = ray_aabb_intersect(rays_o, rays_d, aabbs) File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, *kwargs) File "/opt/conda/lib/python3.7/site-packages/nerfacc/grid.py", line 49, in ray_aabb_intersect miss_value, File "/opt/conda/lib/python3.7/site-packages/nerfacc/cuda/init.py", line 13, in call_cuda return getattr(_C, name)(args, **kwargs) AttributeError: 'NoneType' object has no attribute 'ray_aabb_intersect'

fanzz1208 commented 1 year ago

Traceback (most recent call last): File "train.py", line 270, in entrypoint() File "train.py", line 261, in entrypoint main( File "train.py", line 246, in main launch( File "train.py", line 189, in launch main_func(local_rank=0, world_size=world_size, config=config) File "train.py", line 100, in train_loop trainer.train() File "/workspace/nerfstudio2/nerfstudio/engine/trainer.py", line 250, in train callback.run_callback_at_location( File "/workspace/nerfstudio2/nerfstudio/engine/callbacks.py", line 112, in run_callback_at_location self.run_callback(step=step) File "/workspace/nerfstudio2/nerfstudio/engine/callbacks.py", line 97, in run_callback self.func(*self.args, self.kwargs, step=step) File "/workspace/nerfstudio2/nerfstudio/models/instant_ngp.py", line 154, in update_occupancy_grid self.occupancy_grid.update_every_n_steps( File "/opt/conda/envs/lll/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/opt/conda/envs/lll/lib/python3.8/site-packages/nerfacc/estimators/occ_grid.py", line 247, in update_every_n_steps self._update( File "/opt/conda/envs/lll/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/opt/conda/envs/lll/lib/python3.8/site-packages/nerfacc/estimators/occ_grid.py", line 305, in _update occ = occ_eval_fn(x).squeeze(-1) File "/workspace/nerfstudio2/nerfstudio/models/instant_ngp.py", line 156, in occ_eval_fn=lambda x: self.field.density_fn(x) self.config.render_step_size, File "/workspace/nerfstudio2/nerfstudio/fields/base_field.py", line 67, in densityfn density, = self.get_density(ray_samples) File "/workspace/nerfstudio2/nerfstudio/fields/nerfacto_field.py", line 216, in get_density h = self.mlp_base(positions_flat).view(ray_samples.frustums.shape, -1) File "/opt/conda/envs/lll/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/opt/conda/envs/lll/lib/python3.8/site-packages/torch/nn/modules/container.py", line 217, in forward input = module(input) File "/opt/conda/envs/lll/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, **kwargs) File "/workspace/nerfstudio2/nerfstudio/field_components/mlp.py", line 178, in forward return self.pytorch_fwd(in_tensor) File "/workspace/nerfstudio2/nerfstudio/field_components/mlp.py", line 164, in pytorch_fwd for i, layer in enumerate(self.layers): File "/opt/conda/envs/lll/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1614, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'MLP' object has no attribute 'layers'

tancik commented 1 year ago

The second issue is fixed in #2124

s-esposito commented 1 year ago

@fanzz1208 were you able to solve the first error?

Terry10086 commented 1 year ago

The second issue is fixed in #2124

I am facing the first problem too. Is there any solution?

brentyi commented 1 year ago

If you're seeing the first error, you can check that nvcc is in your PATH? Installing cuda-toolkit[^1] may fix this.

[^1]: For example, via conda: conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit

Related: https://github.com/KAIR-BAIR/nerfacc/issues/89

sinAshish commented 1 year ago

this change is not reflected in the package installed via pip. it'd be great if the latest pip package can be released with this change!