search

KevinXu02 / splatfacto-w

Apache License 2.0
73 stars 2 forks source link

Background model may not be rendered correctly by the ns-render command #13

Closed kstoneriv3 closed 2 weeks ago

kstoneriv3 commented 3 weeks ago

Hi, thanks for making this useful GS implementation for wild available! I see significant improvement in the quality of GS for my dataset!!

Today, I found that ns-render command does not properly render the background model, probably because it only looks at Model.get_outputs() when rendering.

I am not 100% confident that is the case but just wanted to let you know this.

KevinXu02 commented 3 weeks ago

Hi! Could you please try adding --camera_idx [idx] to the end of your rendering command?

kstoneriv3 commented 3 weeks ago

Hi, thanks for the quick reply! I am using ns-render camera-path command like below so I am not sure what argument I should provide to --camera-idx.

ns-render camera-path --load-config outputs/nerfstudio/splatfacto-w-light/2024-08-16_111204/config.yml --camera-path-filename /var/data/360/fps05/nerfstudio/camera_paths/2024-08-16-08-54-17.json --output-path renders/nerfstudio/splatfacto-w3.mp4
KevinXu02 commented 3 weeks ago

You can see the current camera idx in CLI when you click in the viewer. And then you can decide which idx to pass in.

kstoneriv3 commented 3 weeks ago

I am trying to generate a video from novel viewpoints. Would it be possible with camera-idx parameter? It looks like it only supports generation of image from training camera's perspective, if I understand correctly.

kstoneriv3 commented 3 weeks ago

Sorry, I tried setting a --camera_idx and and it worked out.

I was mistakenly assuming that --camera_idx specifies the position of the camera from which an image is rendered, but I realized that it is used to choose the camera which the foreground and background model (i.e., appearance embedding) corresponds to. (please correct me if I am wrong!)

Anyways, thanks for the following up.

kstoneriv3 commented 3 weeks ago

Just a random question, but if --camera_idx specifies the appearance embedding, how does it know which appearance embedding to use for the foreground? As I mentioned above, it renders only the foreground (but not the background) without knowing which --camera_idx to use.

KevinXu02 commented 2 weeks ago

In the training process, an optimizable appearance embedding is assigned to each training image (where the embedding is derived from the camera index). This embedding fits both the foreground and background. So given the idx, it will use the appearance of that training image to render the whole scene.

kstoneriv3 commented 2 weeks ago

Thank you for clarification!!