Not accurate results

[Kaivalya192]

https://github.com/ajzhai/NeRF2Physics/assets/120581640/bbd181b7-400b-4071-99cd-5c70909059aa

i used nerfstudio for recustruction i passed the video to it and extracted point cloud

the info.json

{ "idx_to_caption": "51", "caption": "a black office chair with a black seat", "candidate_materials_density": "(plastic: 900-1200 kg/m^3);(metal: 7000-8000 kg/m^3);(foam: 30-100 kg/m^3);(fabric: 200-500 kg/m^3);(wood: 500-900 kg/m^3)", "thickness": "(plastic: 0.1-0.5 cm);(metal: 0.2-1.0 cm);(foam: 2-5 cm);(fabric: 0.1-0.3 cm);(wood: 0.5-1.5 cm)" }

what is the issue why it is pridicting it wood? and clip feature fusion are also not upto the mark... what can be the issue if it is regarding point cloud generation so please specify steps to generate point cloud from nerfstudio...

[ajzhai]

Hi @Kaivalya192 , the point cloud seems reasonable to me. If you want better reconstruction, you can try to play with different settings in nerfstudio, or capture more views near the bottom of the chair.

The reason that the LLM predicts wood is because it is predicting possible materials for the chair, not the exact set of materials. Then, CLIP is used to select within the possible ones.

I'm not sure what you mean by the clip feature fusion being not up to the mark. Can you elaborate?

[Kaivalya192]

Because in the clip feature image ... pink part is majorly covered in that part where the metal and leather part is not distinguishable.

I think the clip feature should be different from up to down in chair.

At what bases kernel regression chooses material for clip feature?

And also one thing... Why is the weight pridiction zero? (Density is provided correctly )

[ajzhai]

Yea, something is definitely weird with your results, although I am not sure which visualization you were looking at in your image 2 and 3. The weight prediction should never be zero, unless you somehow have zero points in the integral. What does this part print for you? https://github.com/ajzhai/NeRF2Physics/blob/6d81c093ed05434f31c0c735fdad1e51355bde86/predict_property.py#L132

[Kaivalya192]

This is output of feature fusion.

and this is the output of pridict_property.py

~/NeRF2Physics$ /home/iitgn-robotics/miniconda3/envs/n2p/bin/python /home/iitgn-robotics/NeRF2Physics/predict_property.py

scene: chair_lab

num. dense points: 16681 caption: a black office chair with a black seat candidate materials: plastic: 900.0 - 1200.0 kg/m^3, 0.1 - 0.5 cm metal: 7000.0 - 8000.0 kg/m^3, 0.2 - 1.0 cm foam: 30.0 - 100.0 kg/m^3, 2.0 - 5.0 cm fabric: 200.0 - 500.0 kg/m^3, 0.1 - 0.3 cm wood: 500.0 - 900.0 kg/m^3, 0.5 - 1.5 cm surface cell size: 1.9565 cm predicted total mass: [0.0000 - 0.0000 kg]

[ajzhai]

Can you check here and see if any of mat_cell_products, dense_pred_products, or bound_volume are zero?

https://github.com/ajzhai/NeRF2Physics/blob/6d81c093ed05434f31c0c735fdad1e51355bde86/predict_property.py#L103