search

NVlabs / ParallelInversion

Parallel Inversion of Neural Radiance Fields for Robust Pose Estimation (ICRA 2023)
https://pnerfp.github.io/
Other
29 stars 4 forks source link

Using per image losses instead of pixel-wise? #3

Closed maxfehrentz closed 8 months ago

maxfehrentz commented 8 months ago

Hi, I'm wondering what would be necessary to implement global losses for pose estimation like NCC (or at least approximations) instead of pixel-wise ones? I guess this would come with heavy punishment on performance but maybe an approximation instead of rendering a full image might be sufficient?

As far as I understand, all the currently available losses operate on individual RGB values rendered along individual rays. I would probably need to change the ray-casting regime in order to pass an (approximated) image to my own loss and gradient function. Not quite sure where and how to tackle this.

Do you have any insights on this?

Uio96 commented 8 months ago

Hi, I'm wondering what would be necessary to implement global losses for pose estimation like NCC (or at least approximations) instead of pixel-wise ones? I guess this would come with heavy punishment on performance but maybe an approximation instead of rendering a full image might be sufficient?

As far as I understand, all the currently available losses operate on individual RGB values rendered along individual rays. I would probably need to change the ray-casting regime in order to pass an (approximated) image to my own loss and gradient function. Not quite sure where and how to tackle this.

Do you have any insights on this?

Thanks for the interest in our work. I agree with your idea that it would require a lot more effort if you want to implement some global losses based on this codebase. It may also affect the speed performance as you may have to hack some implementation details.

Another way around this is to play around with some torch-based NGP codes. I have not had the time to go into their details though. Their speed may not be as good as this one but would be easier for you to implement your customized global loss.