[demo.py] No such file or directory: 'hamer_training_data/vitpose_backbone.pth'

[zc-alexfan]

Thanks for the great work.

I was trying to run the demo code and encountered this:

Traceback (most recent call last):
  File "/home/zfan/hold-private/dataset_generator/hamer/demo.py", line 208, in <module>
    main()
  File "/home/zfan/hold-private/dataset_generator/hamer/demo.py", line 38, in main
    model, model_cfg = load_hamer(args.checkpoint)
  File "/home/zfan/hold-private/dataset_generator/hamer/hamer/models/__init__.py", line 47, in load_hamer
    model = HAMER.load_from_checkpoint(checkpoint_path, strict=False, cfg=model_cfg)
  File "/home/zfan/miniconda3/envs/hamer/lib/python3.10/site-packages/pytorch_lightning/utilities/model_helpers.py", line 125, in wrapper
    return self.method(cls, *args, **kwargs)
  File "/home/zfan/miniconda3/envs/hamer/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1581, in load_from_checkpoint
    loaded = _load_from_checkpoint(
  File "/home/zfan/miniconda3/envs/hamer/lib/python3.10/site-packages/pytorch_lightning/core/saving.py", line 91, in _load_from_checkpoint
    model = _load_state(cls, checkpoint, strict=strict, **kwargs)
  File "/home/zfan/miniconda3/envs/hamer/lib/python3.10/site-packages/pytorch_lightning/core/saving.py", line 158, in _load_state
    obj = cls(**_cls_kwargs)
  File "/home/zfan/hold-private/dataset_generator/hamer/hamer/models/hamer.py", line 36, in __init__
    self.backbone.load_state_dict(torch.load(cfg.MODEL.BACKBONE.PRETRAINED_WEIGHTS, map_location='cpu')['state_dict'])
  File "/home/zfan/miniconda3/envs/hamer/lib/python3.10/site-packages/torch/serialization.py", line 791, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/zfan/miniconda3/envs/hamer/lib/python3.10/site-packages/torch/serialization.py", line 271, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/zfan/miniconda3/envs/hamer/lib/python3.10/site-packages/torch/serialization.py", line 252, in __init__
    super().__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'hamer_training_data/vitpose_backbone.pth'

I ran fetch_demo_data.sh and it should have the data for demo purposes. Do I also need to run fetch_training_data.sh to get hamer_training_data/vitpose_backbone.pth? But the training data is huge.

Therefore, I commented out this and the code runs with the following warning in state_dict:

Use load_from_local loader                                                                                                   
The model and loaded state dict do not match exactly                                                                         

unexpected key in source state_dict: backbone.blocks.0.mlp.experts.0.weight, backbone.blocks.0.mlp.experts.0.bias, backbone.b
locks.0.mlp.experts.1.weight, backbone.blocks.0.mlp.experts.1.bias, backbone.blocks.0.mlp.experts.2.weight, backbone.blocks.0
.mlp.experts.2.bias, backbone.blocks.0.mlp.experts.3.weight, backbone.blocks.0.mlp.experts.3.bias, backbone.blocks.0.mlp.expe
rts.4.weight, backbone.blocks.0.mlp.experts.4.bias, backbone.blocks.0.mlp.experts.5.weight, backbone.blocks.0.mlp.experts.5.b
ias, backbone.blocks.1.mlp.experts.0.weight,
...

My current assumption is that: I still need to download the training data to get the pretrained backbone so that the state_dict warning disappear to obtain the full-performing model. Is this correct?

[zc-alexfan]

The file is indeed in the training data.

But the warning still appears. Does this affect the performance?

Use load_from_local loader                                                                                                   
The model and loaded state dict do not match exactly                                                                         

unexpected key in source state_dict: backbone.blocks.0.mlp.experts.0.weight, backbone.blocks.0.mlp.experts.0.bias, backbone.b
locks.0.mlp.experts.1.weight, backbone.blocks.0.mlp.experts.1.bias, backbone.blocks.0.mlp.experts.2.weight, backbone.blocks.0
.mlp.experts.2.bias, backbone.blocks.0.mlp.experts.3.weight, backbone.blocks.0.mlp.experts.3.bias, backbone.blocks.0.mlp.expe
rts.4.weight, backbone.blocks.0.mlp.experts.4.bias, backbone.blocks.0.mlp.experts.5.weight, backbone.blocks.0.mlp.experts.5.b
ias, backbone.blocks.1.mlp.experts.0.weight,
...

[Derek0907]

Hi, i am meeting the same issue, have you solved this problem?

[zc-alexfan]

No unfortunately

[geopavlakos]

I have removed the dependency on the vitpose_backbone.pth file for the demo code. Please pull again the latest version.

The key mismatch warning is on the side of the ViTPose model we use for localizing the hand in the image. This warning is expected, so it shouldn't worry you.