search

NVlabs / EmerNeRF

PyTorch Implementation of EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision
https://emernerf.github.io/
Other
572 stars 44 forks source link

How to optimize EmerNeRF #21

Open nevergone123 opened 7 months ago

nevergone123 commented 7 months ago

frame_019

Thanks for your wonderful works!

I'm doing a project related to AD simulation. I ran emernerf on scene in which ego car move at a medium or high speed, like waymo scene id 2. But the decomposition result is not good.

Following the approach of DynIBAR, I tried to improve emernerf by adding 2D optical flow supervision to the training process. I merged the flow loss into the total pixel loss. However, the flow loss did not converge during training, and the overall training results became worse.

So the questions are:

  1. How can I optimize emernerf? For example, improve its decomposition ability.
  2. If I use additional 2D optical flow supervision, besides merging the flow loss into the total_pixel_loss, what else should I do? Will this approach improve the decomposition results?

Looking forward to your advice. Thank you once again!

ntaquan0125 commented 6 months ago

I think outliers come from lighting variations between views. And 2D flow supervision for fast ego-motion data should be handled with care.