search

Harry-Zhi / semantic_nerf

The implementation of "In-Place Scene Labelling and Understanding with Implicit Scene Representation" [ICCV 2021].
Other
426 stars 56 forks source link

bad result of office scene #13

Closed zhouzhenghong-gt closed 2 years ago

zhouzhenghong-gt commented 2 years ago

Hello! Thanks for your great work!! I am very interested in your semantic-nerf. And I encountered a problem when I apply other model on your pre-rendered replica dataset. The same parameters performed well on room0, but performed very poorly on scene data like office4. Have you encountered a similar problem? Are the hyperparameters of each scene basically the same?

This is novel view synthesis result of room0: room0

This is novel view synthesis result of office4: office4

Harry-Zhi commented 2 years ago

Hi @zhouzhenghong-gt,

Thanks for your insterest in Semantic-NeRF. As far as I know. the sequneces in office4 are quite challenging (severe occlusions, drastic motion, glass materials ), leading to relatively worse recontruction compared to that of rooms in replica, and sometimes leads to unstable training. In your case of office4, are the photometric loss and semantic loss well converged? Will a re-training help to avoid this case?

Also, you could try sequence_2 from Office4 which has less severe motions compared to sequence_1 and see if the reconstruction/view synthesis could be improved.

Please let me know if these helps or not.

zhouzhenghong-gt commented 2 years ago

Thanks! @Harry-Zhi Thank you for your reply, you are right that the data of office4 is indeed challenging. This issue gives the result of reconstruction using only rgb. Later, after I tried to use MSI(multi-sphere image) to consider the background information, the reconstruction can get stable results.