search

google-research / kubric

A data generation pipeline for creating semi-realistic synthetic multi-object videos with rich annotations such as instance segmentation masks, depth maps, and optical flow.
Apache License 2.0
2.34k stars 233 forks source link

Inconsistency between optical flow and depth/pose/camera intrinsics #301

Open mmbannert opened 1 year ago

mmbannert commented 1 year ago

Hi,

We discovered a problem with the optic flow, depth, and/or camera pose data that Kubric produces. The problem along with the code to reproduce it is visualized on here.

We are trying to decompose the optical flow from Kubric videos generated with movi_def_worker.py into rigid and nonrigid optical flow (roughly similar to http://arxiv.org/abs/1805.09806). This decomposition requires that we combine camera pose & intrinsics as well as depth information to compute the optical flow. We noticed, however, that the computation of optical flow based on those data is inconsistent with the ground-truth (GT) optical flow that is provided by Kubric. To check that our analysis is indeed correct, we applied it also to Sintel, Monkaa, and Spring as a sanity check. Here, our computation of optical flow is indeed consistent with the ground-truth optical flow that they provide.

Could the inconsistency between reported GT optical flow and depth/pose-based optical flow indicate that something is off about Kubric's GT optical flow, depth, or pose information? Alternatively, are we perhaps misinterpreting some of the data in Kubric, or using them incorrectly?

Help would be greatly appreciated!

Best, Michael

mmbannert commented 11 months ago

To everyone facing the same questions: I made a PR that includes the kind of documentation that I would have found helpful back then. https://github.com/google-research/kubric/pull/307