Closed saliteta closed 2 months ago
I solved this issue by adding the clip_grad_norm to contain the gradient of index_decoder
total_loss = dataset.reconstruction_loss_weight * loss \
+ dataset.semantic_loss_weight * semantic_loss \
+ dataset.uncertainty_loss_weight * uncertainty_loss \
+ dataset.xyzmlp_loss_weight * xyz_mlp_loss \
+ dataset.smooth_loss_weight * smooth_loss
total_loss.backward()
torch.nn.utils.clip_grad_norm_(index_decoder.parameters(), max_norm=1)
iter_end.record()
After training for several epochs, the semantic loss will be nan.