nan raises pdb debugger with LLFF dataset

samhodge-aiml commented 11 months ago

see

[Epoch 3295] it=240500, loss=5.24928904, time=10.5139
  3295 ep: Train: ATE: 0.574 RPE_r: 27.093
  3295 ep: Train: PSNR: 17.051
> /media/sam/aimlwork/github/nope-nerf/model/losses.py(207)forward()
-> 'loss': loss,
(Pdb) exit
Traceback (most recent call last):
  File "/media/sam/aimlwork/github/nope-nerf/train.py", line 364, in <module>
    train(cfg=cfg)
  File "/media/sam/aimlwork/github/nope-nerf/train.py", line 210, in train
    loss_dict = trainer.train_step(batch, it, epoch_it, scheduling_start, render_path)
  File "/media/sam/aimlwork/github/nope-nerf/model/training.py", line 87, in train_step
    loss_dict = self.compute_loss(data, it=it, epoch=epoch, scheduling_start=scheduling_start, out_render_path=render_path)
  File "/media/sam/aimlwork/github/nope-nerf/model/training.py", line 371, in compute_loss
    loss_dict = self.loss(rendered_rgb, rgb_gt, rendered_depth, gt_depth, **kwargs)
  File "/media/sam/aimlwork/github/nope-nerf/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/sam/aimlwork/github/nope-nerf/model/losses.py", line 207, in forward
    'loss': loss,
  File "/media/sam/aimlwork/github/nope-nerf/model/losses.py", line 207, in forward
    'loss': loss,
  File "/media/sam/aimlwork/github/nope-nerf/env/lib/python3.10/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
  File "/media/sam/aimlwork/github/nope-nerf/env/lib/python3.10/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit

samhodge-aiml commented 11 months ago

Restarting seems fine.

bianwenjing commented 11 months ago

Hi, please let me know if this issue still exists

ActiveVisionLab / nope-nerf

nan raises pdb debugger with LLFF dataset #23