Advice on how to run the code on custom videos

[antoan]

Hi, thank you for making this public.

I wanted to ask if I can run this on video frames in the wild or if there are any additional input constraints to satisfy, like resolution.

Is a 360 degree camera trajectory required?

My aim is to exact unknown camera intrinsic and camera pose of an ceiling mounted uncalibrated PTZ camera inside an industrial hanger.

Thanks in advance.

[dcharatan]

In-the-wild videos should work as long as they don't have very unusual focal lengths. For an arbitrary video, I would follow these steps:

• Crop the video to the segment that highlights the object/part of the scene you're interested in.
• Dump the video's frames into a folder (e.g., using ffmpeg).
• Use python3 -m flowmap.subsample +in_path=<path to frames> +out_path=<output path> to subsample the video. This script will try to evenly distribute the optical flow across the video frames. If you want something that runs quickly, I would aim for ~50 frames. We ran up to ~150 frames for the paper, but this takes a while (~20 mins on an A100). You could probably go up to ~300 with a large GPU like an A100, but this would be slower.
• Confirm that the subsampled video is likely to work well with optical flow tracking. In other words, make sure there are no extremely large jumps between frames.
• Run FlowMap. Make sure you've downloaded the initialization and use it!

You don't necessarily need a 360-degree sequence. We tried both 360-degree sequences and forward-facing sequences. You could try sequences with multiple loops, but be warned that FlowMap doesn't have a loop closure mechanism (this would be an exciting direction for future work!).

Note that FlowMap assumes a constant focal length across the video. Also, if your video has a focal length that's very zoomed in or zoomed out, you might have to tweak the focal length candidates here.

[leo-frank]

@antoan btw, you should also assume there are no moving objects in the scene.

[dcharatan]

@leo-frank Makes a good point as well! Small moving objects (e.g., people walking in the background of some of the Tanks and Temples scenes) are usually fine, but large moving objects will confuse FlowMap.