search

sarafridov / K-Planes

Other
480 stars 46 forks source link

Temporal difference (IST) strategy #40

Closed xuyaojian123 closed 11 months ago

xuyaojian123 commented 11 months ago

image

could you explain why temporal difference (IST) strategy cannot be used for monocular videos?

sarafridov commented 11 months ago

The temporal difference strategy works like this: Given a fixed-position camera, compare pixel values over time to detect motion, and then prioritize training with those pixels that change value. If the camera is moving between different timesteps, then you can't discover and prioritize motion by doing pixel-wise temporal differencing because all the pixels will exhibit camera motion.

xuyaojian123 commented 11 months ago

Thanks your reply. You may understand me wrong, I understand how IST is calculated, I want to ask why IST does not apply to monocular videos (only one fixed camera)?

sarafridov commented 11 months ago

Ah, usually when people say "monocular videos" in the context of radiance field modeling, they mean a single camera that moves around over time. If there is only one camera and it doesn't move, then you don't get enough information for a volumetric reconstruction (at least, not without relying on very strong depth priors).

xuyaojian123 commented 11 months ago

Thank you very much! By the way, help me see see issue https://github.com/sarafridov/K-Planes/issues/38#issue-2002214935.