search

dcharatan / pixelsplat

[CVPR 2024 Oral, Best Paper Runner-Up] Code for "pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction" by David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann
http://davidcharatan.com/pixelsplat/
MIT License
830 stars 56 forks source link

The pretrained model cannot be loaded #65

Closed hj-jung-01 closed 4 months ago

hj-jung-01 commented 4 months ago

Hi, I'm trying to run evaluation on re10k subset following your documentation. But loading pretrained model fails.

Error executing job with overrides: ['+experiment=re10k', 'mode=test', 'dataset/view_sampler=evaluation', 'dataset.view_sampler.index_path=re10k_subset/test/index.json', 'checkpointing.load=checkpoints/re10k.ckpt'] Traceback (most recent call last): File "/nasspace/samsung/pixelsplat/src/main.py", line 130, in train trainer.test( File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 754, in test return call._call_and_handle_interrupt( File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 44, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 794, in _test_impl results = self._run(model, ckpt_path=ckpt_path) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 956, in _run self._checkpoint_connector._restore_modules_and_callbacks(ckpt_path) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/checkpoint_connector.py", line 398, in _restore_modules_and_callbacks self.restore_model() File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/checkpoint_connector.py", line 275, in restore_model self.trainer.strategy.load_model_state_dict( File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 372, in load_model_state_dict self.lightning_module.load_state_dict(checkpoint["state_dict"], strict=strict) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for ModelWrapper: Unexpected key(s) in state_dict: "encoder.epipolar_transformer.transformer.layers.0.1.fn.layers.0.weight", "encoder.epipolar_transformer.transformer.layers.0.1.fn.layers.0.bias", "encoder.epipolar_transformer.transformer.layers.0.1.fn.layers.3.weight", "encoder.epipolar_transformer.transformer.layers.0.1.fn.layers.3.bias", "encoder.epipolar_transformer.transformer.layers.1.1.fn.layers.0.weight", "encoder.epipolar_transformer.transformer.layers.1.1.fn.layers.0.bias", "encoder.epipolar_transformer.transformer.layers.1.1.fn.layers.3.weight", "encoder.epipolar_transformer.transformer.layers.1.1.fn.layers.3.bias".

The excuted command is python3 -m src.main +experiment=re10k mode=test dataset/view_sampler=evaluation dataset.view_sampler.index_path=re10k_subset/test/index.json checkpointing.load=checkpoints/re10k.ckpt

I have traced the implementation of epipolar transformer, there is no variable named "layers" or use of "ModuleList" in attension.py. So there is completely missing 8 layers.

I think the pretrained model's network architecture and the implementation are not matched. Can you check out this error?

dcharatan commented 4 months ago

Make sure you're using the right combination of model and checkpoint. The original version of the model and the improved camera-ready version of the model are slightly different, so the checkpoints aren't compatible. I would recommend pulling the latest commit and using these checkpoints.