Possibility to use OccupancyGrid for generalizable NeRF

[junzhezhang]

Hi,

Thanks for the amazing work!

I am quite curious if this framework can be used in a generalizable setting? If, let's say, we have prior knowledge of the occupancy of the N scenes in the train set and M scenes in the test set, in the form of 64x64x64 grids, do we have a way to maintain a new OccupancyGrid for each scene and manually store the prior occupancy into it?

Hope I made myself clear.

Thank you! Junzhe

[liruilong940607]

Hi I'm working on a new version of nerfacc which will support for generalizable nerfs. Current version unfortunately does not work with generalizable nerfs because the OccupancyGrid does not support taking in prior knowledge.

[junzhezhang]

Hi Ruilong,

Great! Do you have a rough timeline for this?

Btw, if you need any sort of "customer requirement" for generalizable and generative case, I am willing to discuss with you:)

Junzhe

[liruilong940607]

Open for suggestions!! You are welcome to left notes here on your user case (Pesudo code or whatever).

About the timeline, I'm coming to an end to finish the core library but haven't really built an example for a full test and demonstration of usage. Idea is to have a pixelnerf example but it might take me a few more weeks, as I'm not full time on this project.

[ttsesm]

@liruilong940607 any update on this?

[liruilong940607]

Hi I'm supporting the proposal network way from Mip-NeRF 360, which would support accelerating generalizable nerfs. Still working on it

[ttsesm]

I see, thanks.

[liruilong940607]

With the proposal network estimator supported in nerfacc>=0.5.0 this should be doable. Basically just create a cheaper version of your network as the proposal network, and feed it with whatever input you feed to your main nerf, and train them jointly.

We don't have a specific example for generalizable nerfs but the example here should be easily applicable to generalizable nerfs.