spacetime-only and render-only and test video

[Spphire]

i find that result from render-only or spacetime-only is much more better than test video created during training.

is that normal?

step150000:

https://user-images.githubusercontent.com/56157591/229069828-b8f614e5-2cef-4f5c-ab37-3ed9a997812f.mp4

render-only:

https://user-images.githubusercontent.com/56157591/229069888-7757f54b-8f65-4211-bd2d-1968b88dcd6f.mp4

(ps. im testing to only use 2 views(2 origin video) to train)

[Giodiro]

That's weird! Of course the "during training" video is from a specific pose, but the colorful artifacts you're seeing are strange. One possibility that comes to mind is the tuning of near-far bounds that we do for "render-only" videos, nothing fancy see https://github.com/sarafridov/K-Planes/blob/e7cd380c97f5e2abae3f7e17a7c02cab6c65bad7/plenoxels/datasets/video_datasets.py#L75 If that's the culprit then you simply have a ton of weirdly colorful floaters

[sarafridov]

I agree, and I can make two additional suggestions in case it's useful, given that you are training with only 2 viewpoints. One is that you may see improvement from increasing the weight on the spatial total variation regularization, which might help remove some of the floaters (at the cost of losing some detail). The other is that you can try pretraining just the weights of the color and density decoders on the full set of training views (or using our pretrained checkpoint for those weights), and then only optimizing the plane values on the two training views. Possibly the weird colors could be coming from directions of the view-dependent color decoder that were not supervised.