buaavrcg / LEGaussians

Pytorch Code for "LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding"
https://buaavrcg.github.io/LEGaussians/
MIT License
100 stars 14 forks source link

Nan value for semantic loss #12

Closed saliteta closed 2 months ago

saliteta commented 2 months ago

After training for several epochs, the semantic loss will be nan.

image

saliteta commented 2 months ago

I solved this issue by adding the clip_grad_norm to contain the gradient of index_decoder

        total_loss = dataset.reconstruction_loss_weight * loss \
                    + dataset.semantic_loss_weight * semantic_loss \
                    + dataset.uncertainty_loss_weight * uncertainty_loss \
                    + dataset.xyzmlp_loss_weight * xyz_mlp_loss \
                    + dataset.smooth_loss_weight * smooth_loss

        total_loss.backward()

        torch.nn.utils.clip_grad_norm_(index_decoder.parameters(), max_norm=1)

        iter_end.record()