about Semantic Nerf (or autonerfacto) pretrained weights

[jeonggwanlee]

Hello! Thank you so much for sharing your incredible work! I have read the README.md and followed each step carefully.

I am interested in testing mesh generation (or NeRF inference), but to the best of my knowledge, I was unable to find the pretrained weights for Semantic NeRF (or autonerfacto).

Would it be possible for you to share those weights if available?

Thanks.

[PierreMarza]

Hi!

Thank you for your interest in our work! This code allows you to collect data with a modular policy (pre-trained weights of the most used policy in the paper are released), and then train a NeRF model (Semantic NeRF or autonerfacto) from the collected data. Have you been able to try this? Once training is done, you will be able to test the final NeRF model (mesh generation and/or NeRF inference).

A different NeRF model is trained for each policy rollout, so the pre-trained weights depend on the specific 3D scene, but also policy trajectory within such scene. I would thus suggest you train a NeRF for the specific scenario you are interested in. Let me know if you have any issues with training a NeRF from the code, or please provide more information about what you would like to do, and I’ll see what I can do to help!

[jeonggwanlee]

Thank you for kind answering!

Sure, I tried to train Semantic NeRF first following your instructions in README.md. I also got the mesh generation, too. (It looks nice!)

I'm curious about how to do NeRF Inference using trained Semantic NeRF model. In my best knowledge, I could not find how to inference NeRF (to check NeRF quality). I imagined that hopefully looking around the room and looking at novel view supported by NeRF.

I have trained Semantic Nerfacto, too. and maybe I can try BEV map generation, planning, Pose refinement. But, I'm not sure I can check NeRF quality.

Is there any way to check NeRF quality? (or just inference of NeRF). I will check your code more, too. I expected that mesh generation code might help to find NeRF inference part.

Again, Thank you for sharing your interesting work :)

[jeonggwanlee]

I read "mesh generation" code and realized that I misunderstand.

So, we can define cube (256, 256, 256) and trained NeRF model can predict each point's color (rgb) and density, too. Using density, we can create occupancy with a certain threshold. Finally we can generate mesh. Awesome!

If you think some parts I misunderstood, feel free to let me know! :)

[PierreMarza]

Doing inference with a NeRF model is about predicting density, color (and semantics in our case) for a given point in the considered 3D space. This is what we indeed use to build a mesh representation.

However, it looks like you are mostly interested in performing novel view synthesis, i.e. rendering (NeRF inference + volumetric rendering), to evaluate the ability of the NeRF model to render frames from novel camera views, right? Such evaluation (on a held-out set of test camera poses) is already performed during training. You can find some inference samples on the nerfstudio tensorboard (“IMAGES” section — when training autonerfacto) for example, and mean rendering metrics for all test images (“SCALARS/Eval Images Metrics Dict (all images)”). Let me know if you need more help.

[jeonggwanlee]

Yes, That is what I'd like to try. I'll check it! Thank you very much 👍 @PierreMarza