Receiving error when runnning video_demo.py

[gigasurgeon]

I am receiving an error when runnning video_demo.py. I am not receiving error while running demo.py. I have extracted the frames. Does the video require any more inputs?

python3 video_demo.py SCENE_DIR="/vmdata/amitsingh/camera_tracking/fb_vggsfm/extracted_frames/Consumer 1 mop cropped.mp4" gr_visualize=True
Poselib is available
Poselib is available
/vmdata/amitsingh/camera_tracking/fb_vggsfm/vggsfm/video_demo.py:18: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path="cfgs/", config_name="video_demo")
/opt/conda/envs/vggsfm_tmp/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
Model Config: query_by_midpoint: true
shared_camera: true
camera_type: SIMPLE_PINHOLE
max_query_pts: 512
model_name: vggsfm_v2_0_0
seed: 0
img_size: 1024
debug: false
center_order: false
mixed_precision: fp16
extract_color: true
filter_invalid_frame: true
comple_nonvis: true
query_frame_num: 2
robust_refine: 2
BA_iters: 1
fine_tracking: true
load_gt: false
viz_visualize: false
gr_visualize: true
fmat_thres: 4.0
max_reproj_error: 4.0
init_max_reproj_error: 4.0
dense_depth: false
avg_pose: true
save_to_disk: true
SCENE_DIR: /vmdata/amitsingh/camera_tracking/fb_vggsfm/extracted_frames/Consumer 1
  mop cropped.mp4
resume_ckpt: ckpt/vggsfm_v2_0_0.bin
auto_download_ckpt: true
query_method: aliked
use_poselib: true
shift_point2d_to_original_res: false
make_reproj_video: false
visual_tracks: false
visual_query_points: false
visual_dense_point_cloud: false
query_by_interval: false
MODEL:
  _target_: vggsfm.models.VGGSfM
  TRACK:
    _target_: vggsfm.models.TrackerPredictor
    efficient_corr: false
    COARSE:
      stride: 4
      down_ratio: 2
      FEATURENET:
        _target_: vggsfm.models.BasicEncoder
      PREDICTOR:
        _target_: vggsfm.models.BaseTrackerPredictor
    FINE:
      FEATURENET:
        _target_: vggsfm.models.ShallowEncoder
      PREDICTOR:
        _target_: vggsfm.models.BaseTrackerPredictor
        depth: 4
        corr_levels: 3
        corr_radius: 3
        latent_dim: 32
        hidden_size: 256
        fine: true
        use_spaceatt: false
  CAMERA:
    _target_: vggsfm.models.CameraPredictor
  TRIANGULAE:
    _target_: vggsfm.models.Triangulator

Building VGGSfM
Using cache found in /home/thegi/.cache/torch/hub/facebookresearch_dinov2_main
/home/thegi/.cache/torch/hub/facebookresearch_dinov2_main/dinov2/layers/swiglu_ffn.py:51: UserWarning: xFormers is not available (SwiGLU)
  warnings.warn("xFormers is not available (SwiGLU)")
/home/thegi/.cache/torch/hub/facebookresearch_dinov2_main/dinov2/layers/attention.py:33: UserWarning: xFormers is not available (Attention)
  warnings.warn("xFormers is not available (Attention)")
/home/thegi/.cache/torch/hub/facebookresearch_dinov2_main/dinov2/layers/block.py:40: UserWarning: xFormers is not available (Block)
  warnings.warn("xFormers is not available (Block)")
[2024-09-04 01:40:31,256][dinov2][INFO] - using MLP layer as FFN
VGGSfM built successfully
Data size of Sequence: 1
['Consumer 1 mop cropped.mp4']
Run Sparse Reconstruction for Scene Consumer 1 mop cropped.mp4
Predicting tracks with query_index = 15
Predicting tracks with query_index = 0
I20240904 01:41:21.226734 140080810829632 misc.cc:198] 
==============================================================================
Global bundle adjustment
==============================================================================
I20240904 01:41:21.531737 140080810829632 timer.cc:91] Elapsed time: 0.005 [minutes]
I20240904 01:41:23.032067 140080810829632 misc.cc:198] 
==============================================================================
Global bundle adjustment
==============================================================================
I20240904 01:41:26.281797 140080810829632 timer.cc:91] Elapsed time: 0.054 [minutes]
I20240904 01:41:28.077859 140080810829632 misc.cc:198] 
==============================================================================
Global bundle adjustment
==============================================================================
I20240904 01:41:31.284615 140080810829632 timer.cc:91] Elapsed time: 0.053 [minutes]
I20240904 01:41:32.854869 140080810829632 misc.cc:198] 
==============================================================================
Global bundle adjustment
==============================================================================
I20240904 01:41:35.227464 140080810829632 timer.cc:91] Elapsed time: 0.040 [minutes]
Running iterative BA by 1 times
I20240904 01:41:36.004411 140080810829632 misc.cc:198] 
==============================================================================
Global bundle adjustment
==============================================================================
I20240904 01:41:40.224216 140080810829632 bundle_adjustment.cc:866] 
Bundle adjustment report:
    Residuals : 70236
   Parameters : 4077
   Iterations : 93
         Time : 4.19116 [s]
 Initial cost : 0.796396 [px]
   Final cost : 0.608271 [px]
  Termination : Convergence

I20240904 01:41:40.224265 140080810829632 timer.cc:91] Elapsed time: 0.070 [minutes]
Error executing job with overrides: ['SCENE_DIR=/vmdata/amitsingh/camera_tracking/fb_vggsfm/extracted_frames/Consumer 1 mop cropped.mp4', 'gr_visualize=True']
Traceback (most recent call last):
  File "/vmdata/amitsingh/camera_tracking/fb_vggsfm/vggsfm/video_demo.py", line 75, in demo_fn
    predictions = vggsfm_runner.run(
  File "/vmdata/amitsingh/camera_tracking/fb_vggsfm/vggsfm/vggsfm/runners/video_runner.py", line 144, in run
    self.extra_params = init_extra[0:1].clone()  # 1xnum_extra_params
TypeError: 'NoneType' object is not subscriptable

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[jytime]

Oh this means you are running video_runner with a simple_pinhole camera model. Let me make a quick fix to this.

[jytime]

Please check this:

https://github.com/facebookresearch/vggsfm/commit/5899a99ff263519c254110fde2036e5ce11f6874

[gigasurgeon]

Thanks. Its working now.