Open bchretien opened 7 months ago
# Important to allow xys grads to populate properly
if self.training:
try:
self.xys.retain_grad()
except Exception as e:
print(e)
can't retain_grad on Tensor that has requires_grad=False
Exception in thread Thread-7:
Traceback (most recent call last):
File "C:\Users\jyomu\scoop\persist\rye\py\cpython@3.10.13\install\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "E:\AI\nerfstudio\nerfstudio\viewer\render_state_machine.py", line 222, in run
outputs = self._render_img(action.camera_state)
File "E:\AI\nerfstudio\nerfstudio\viewer\render_state_machine.py", line 177, in _render_img
assert len(outputs["depth"].shape) == 3
AttributeError: 'NoneType' object has no attribute 'shape'
I'm facing the same issue when I run ns-train splatfacto. Is there any resolution for this?
@jyomu @bchretien Facing the same issue ... any leads on cause of this ?
@InduCherukuri / @Exception4U: as a simple workaround (but probably not solving the root cause):
# Important to allow xys grads to populate properly
if self.training and self.xys.requires_grad:
self.xys.retain_grad()
Do you have sample datasets that we can reproduce the error?
@jb-ye: it happened a bit at random, mostly when I interacted with the viewer and modified the settings (e.g. resolution of the rendering). Alas I cannot share the associated dataset.
I am experiencing the same issue. It appears that when the viewer renders the image, the model is not correctly set to eval mode.
Describe the bug
I randomly get this error during training with
splatfacto
(tested on the c491e3e1 commit in Docker):A check on
requires_grad
might be missing.To Reproduce Steps to reproduce the behavior:
splatfacto
training with for example:Expected behavior
No random error.
Screenshots N/A
Additional context N/A