Customized Run is not working.

[OwenHua666]

Great paper and thank you for making the source code public. The quick start code works for me in a conda environment, but it would be nice if the code can support virtual environment. I am running into an error when I want to try the customized run on a random internet video. The video is 1920x1080. I don't provide the camera model. The command I used is: python main.py --video_file ./data/videos/1943413.mp4 --path ./results/pexels_video1 --make_video --model_type "midas2" The error I got is: Traceback (most recent call last): File "main.py", line 13, in <module> dp.process(params) File "/home/owen/Dev/consistent_depth/process.py", line 117, in process return self.pipeline(params) File "/home/owen/Dev/consistent_depth/process.py", line 88, in pipeline ft.fine_tune(writer=self.writer) File "/home/owen/Dev/consistent_depth/depth_fine_tuning.py", line 257, in fine_tune validate(0, 0) File "/home/owen/Dev/consistent_depth/depth_fine_tuning.py", line 248, in validate criterion, val_data_loader, suffix(epoch, niters) File "/home/owen/Dev/consistent_depth/depth_fine_tuning.py", line 323, in eval_and_save for _, data in zip(range(N), data_loader): File "/home/owen/anaconda3/envs/consistent_depth/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ data = self._next_data() File "/home/owen/anaconda3/envs/consistent_depth/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data) File "/home/owen/anaconda3/envs/consistent_depth/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise() File "/home/owen/anaconda3/envs/consistent_depth/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise raise self.exc_type(msg) IndexError: Caught IndexError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/owen/anaconda3/envs/consistent_depth/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index) File "/home/owen/anaconda3/envs/consistent_depth/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/owen/anaconda3/envs/consistent_depth/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/owen/Dev/consistent_depth/loaders/video_dataset.py", line 172, in __getitem__ intrinsics = torch.stack([self.intrinsics[k] for k in pair], dim=0) File "/home/owen/Dev/consistent_depth/loaders/video_dataset.py", line 172, in <listcomp> intrinsics = torch.stack([self.intrinsics[k] for k in pair], dim=0) IndexError: index 64 is out of bounds for dimension 0 with size 13

I am suspecting that the camera parameter derived from colmap is not compatible with the dataloader.

[roxanneluo]

Can you share the full output of your results through dropbox, for instance? I suspect that not all frames have their cameras successfully registered.

[sergeygross]

the same problem with half of videos

here is the full log sample: https://drive.google.com/file/d/1LlafA0vQDXDCOr_znvrZu4lQy9q0kkdB/view?usp=sharing

[sergeygross]

Another part of video analisys ends with missing bins:

.... cameras = read_cameras_binary(os.path.join(path, "cameras" + ext)) File "/tmp/consistent_depth/third_party/colmap/scripts/python/read_write_model.py", line 135, in read_cameras_binary with open(path_to_model_file, "rb") as fid: FileNotFoundError: [Errno 2] No such file or directory: 'data/videos/out_144p_10s/s_lecture2_10s_144p/colmap_dense/sparse/0/cameras.bin'

[noobtoob4lyfe]

Same thing is happening to me. I was able to get a whole file completed the other day but I'm no longer able to do so. Seems almost random. "FileNotFoundError: [Errno 2] No such file or directory: 'results/GH017205/colmap_dense/sparse/0/cameras.bin'"

[roxanneluo]

the same problem with half of videos

here is the full log sample: https://drive.google.com/file/d/1LlafA0vQDXDCOr_znvrZu4lQy9q0kkdB/view?usp=sharing

Hi, it seems that COLMAP fails to register camera poses on your video. Only 15 frames have been successfully registered. It could be because the sequence is too challenging like not enough baseline between the cameras, the scene is too dynamic, not enough features can be detected in your sequences. I'd encourage you to download COLMAP gui and inspect your COLMAP reconstruction.

[roxanneluo]

Hi, cameras.bin missing also implies COLMAP fails to create a sparse model for your scene. It could be because the sequence is too challenging like not enough baseline between the cameras, the scene is too dynamic, not enough features can be detected in your sequences. I'd encourage you to download COLMAP gui and inspect your COLMAP reconstruction. It would be much easier for COLMAP to register the cameras if the camera intrinsics can be pre-calibrated.

[yearofthewhopper]

Same problem as above. no camera bin "colmap_dense/sparse/0/cameras.bi" no generated video. I think the last answer is a little strange though cus its a very simple video with one surface and slow movement. Has this been tested at all with iphone videos? On what camera was the demo video shot?

[mattbev]

to add to the above comment, I am currently getting this same error on a video clip that has previously ran to completion successfully. now, on every new run regardless of inputs/params I get this error

[mattbev]

to add to the above comment, I am currently getting this same error on a video clip that has previously ran to completion successfully. now, on every new run regardless of inputs/params I get this error

clarifying: many of the early steps in the pipeline are skipped and then it fails during feature extraction and matching because colmap_dense/database.db is never populated (afik). this holds true:

• regardless of the video input, from a similar level of motion to the demo down to low/no motion
• whether specifying a camera model or not

[semel1]

The similar problem as above. Trying to test COLMAP part using supplied ayush.mp4 and deleting results/ayush/colmap_dense and `results/ayush/depth_colmap_dense. Upon start "COLMAP reconstruction" it throws an error:"The procedure entry point could not be located ...". Clicking "Ok" three times cancel execution with an error "FileNotFoundError: [Errno 2] No such file or directory: 'results/GH017205/colmap_dense/sparse/0/cameras.bin'". I'm running Windows 10 with Installed all required prerequisites and "COLMAP-3.7-windows-cuda. All stanalone COLMAP's tests "RUN_TESTS.bat" and COLMAP.bat work just fine. Demo with "colmap_dense" and "depth_colmap_dense" inside "results/ayush/" works fine. Environment variables for COLMAP (lib and bin) specified .

[kayleeliyx]

Another part of video analisys ends with missing bins:

.... cameras = read_cameras_binary(os.path.join(path, "cameras" + ext)) File "/tmp/consistent_depth/third_party/colmap/scripts/python/read_write_model.py", line 135, in read_cameras_binary with open(path_to_model_file, "rb") as fid: FileNotFoundError: [Errno 2] No such file or directory: 'data/videos/out_144p_10s/s_lecture2_10s_144p/colmap_dense/sparse/0/cameras.bin'

I met the same problem. May I ask how did you solve it?

[semel1]

I have not. Just gave up, hoping to manage their next release https://github.com/facebookresearch/robust_cvd So far it works at Colab, but processing time took me (Colab) 3hr to process 142 frames 1440x1080. Output depth has 384x288 instead of input 1440x1080 ???

[kayleeliyx]

I have not. Just gave up, hoping to manage their next release https://github.com/facebookresearch/robust_cvd So far it works at Colab, but processing time took me (Colab) 3hr to process 142 frames 1440x1080. Output depth has 384x288 instead of input 1440x1080 ???

May I ask whether you use png file or raw file? I don't understand the data format in .raw file. What's the difference between png and raw file? Thanks a lot for your help! I'm happy to discuss the issues for this project!

[semel1]

As input for "Robust CVD" I've used ProRes HQ 1440x1080 *.mov video