search

facebookresearch / OmnimatteRF

A matting method that combines dynamic 2D foreground layers and a 3D background model.
MIT License
120 stars 14 forks source link

Preprocess Data #2

Open l-dawei opened 11 months ago

l-dawei commented 11 months ago

Great project, thank you to the author for their excellent work. I encountered two issues during data preprocessing. One is the obtain COLMAP stage, and the other is the Depth Estimation stage. In the obtain COLMAP stage, only 10 of my images were pointed out; In the Depth Estimation stage, there is a problem of network structure mismatch in the weight dpt_beit_large_512.pt.

The first stage:

[2023-10-03 19:16:45,343][main][INFO] - Run: colmap feature_extractor --database_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/run_colmap/database.db --image_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/rgb_1x --SiftExtraction.use_gpu 0 --SiftExtraction.upright 0 --ImageReader.camera_model OPENCV --ImageReader.single_camera 1 [2023-10-03 19:16:45,418][main][INFO] - Run: colmap exhaustive_matcher --database_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/run_colmap/database.db --SiftMatching.use_gpu 0 [2023-10-03 19:16:45,543][main][INFO] - Run: colmap mapper --database_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/run_colmap/database.db --image_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/rgb_1x --output_path /data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/run_colmap/sparse --Mapper.ba_refine_principal_point 1 --Mapper.filter_max_reproj_error 2 --Mapper.tri_complete_max_reproj_error 2 --Mapper.min_num_matches 32 Post-colmap Cameras 5 hwf = 450 450 1505.2784385924213 Images # 10 Points (313, 3) Visibility (313, 10) Depth stats 0.013273805525082845 11.832586834420141 10.408927046544216 Done with imgs2poses [2023-10-03 19:28:29,752][main][ERROR] - Colmap only recovered 10 for 200 images

The last stage:

[2023-10-03 18:44:15,308][main][INFO] - Process 200 files /data_1/anaconda3/envs/omnimatte/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Error executing job with overrides: ['input=/data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/rgb_1x', 'output=/data_1/ldw_models/OmnimatteRF/data/matting/wild/obama/depth', 'model=/data_1/ldw_models/OmnimatteRF/data/pretrained/midas/dpt_beit_large_512.pt'] Traceback (most recent call last): File "/data_1/ldw_models/OmnimatteRF/preprocess/run_depth.py", line 51, in main model, transform, net_w, net_h = load_model(device, cfg.model, cfg.type, optimize=False) File "/data_1/ldw_models/OmnimatteRF/third_party/MiDaS/midas/model_loader.py", line 52, in load_model model = DPTDepthModel( File "/data_1/ldw_models/OmnimatteRF/third_party/MiDaS/midas/dpt_depth.py", line 165, in init self.load(path) File "/data_1/ldw_models/OmnimatteRF/third_party/MiDaS/midas/base_model.py", line 18, in load self.load_state_dict(parameters) File "/data_1/anaconda3/envs/omnimatte/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for DPTDepthModel: Unexpected key(s) in state_dict: "pretrained.model.blocks.0.attn.relative_position_index", "pretrained.model.blocks.1.attn.relative_position_index", "pretrained.model.blocks.2.attn.relative_position_index", "pretrained.model.blocks.3.attn.relative_position_index", "pretrained.model.blocks.4.attn.relative_position_index", "pretrained.model.blocks.5.attn.relative_position_index", "pretrained.model.blocks.6.attn.relative_position_index", "pretrained.model.blocks.7.attn.relative_position_index", "pretrained.model.blocks.8.attn.relative_position_index", "pretrained.model.blocks.9.attn.relative_position_index", "pretrained.model.blocks.10.attn.relative_position_index", "pretrained.model.blocks.11.attn.relative_position_index", "pretrained.model.blocks.12.attn.relative_position_index", "pretrained.model.blocks.13.attn.relative_position_index", "pretrained.model.blocks.14.attn.relative_position_index", "pretrained.model.blocks.15.attn.relative_position_index", "pretrained.model.blocks.16.attn.relative_position_index", "pretrained.model.blocks.17.attn.relative_position_index", "pretrained.model.blocks.18.attn.relative_position_index", "pretrained.model.blocks.19.attn.relative_position_index", "pretrained.model.blocks.20.attn.relative_position_index", "pretrained.model.blocks.21.attn.relative_position_index", "pretrained.model.blocks.22.attn.relative_position_index", "pretrained.model.blocks.23.attn.relative_position_index". Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Could you help me? Thanks a lot.

logchan commented 11 months ago

For COLMAP: unfortunately, if COLMAP fails, you may need to tweak its config to make it work. You can also try other SfM solutions like pixsfm. Or you can try if the included RoDynRF works.

For depths, based on this issue it seems that you need to downgrade a package:

pip install timm==0.6.7