LocalEntryNotFoundError - Can not load correct pretrained twins model

[BotaoPeng98]

Hi, Dr. Junheum Park,

Thanks for releasing code of your fabulous work!

When I was debugging the code, it got an error.

huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I located where the error came from. it seems that the 'timm.creat_model('twins_svt_large', pretrained=pretrained)' failed to download the twins pre-trained model.

Then I searched on HF, I inferred you might use the model from here. Then, I download pytorch_model.bin from it, and modified the code timm.creat_model('twins_svt_large, pretrained=pretrained)' to timm.creat_model('twins_svt_large', pretrained=pretrained, pretrained_cfg_overlay=dict(file='path/to/checkpoint/pytorch_model.bin')), so that it can load the model that I have downloaded.

However, another error emerged.

import sys; print('Python %s on %s' % (sys.version, sys.platform))
/home/botao/miniconda3/bin/conda run -n BiFormer --no-capture-output python /home/botao/.pycharm_helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 127.0.0.1 --port 46109 --file /mnt/d/botao/BiFormer-main/run.py --first images/img1.png --second images/img2.png --output images/img1-img2.png 
Connected to pydev debugger (build 232.9559.58)
/home/botao/miniconda3/envs/BiFormer/lib/python3.9/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  /opt/conda/conda-bld/pytorch_1659484806139/work/aten/src/ATen/native/TensorShape.cpp:2894.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/mnt/d/botao/BiFormer-main/model/BiFormer.py:126: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
  nn.init.kaiming_normal(m.weight.data, mode='fan_in')
/mnt/d/botao/BiFormer-main/model/Upsampler.py:94: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
  nn.init.kaiming_normal(m.weight.data, mode='fan_in')
/mnt/d/botao/BiFormer-main/model/SynthesisNet.py:77: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
  nn.init.kaiming_normal(m.weight.data, mode='fan_in')
[BiFormer] Start to interpolate an intermediate frame
First input frame: images/img1.png
Second input frame: images/img2.png
Traceback (most recent call last):
  File "/mnt/d/botao/BiFormer-main/run.py", line 66, in <module>
    flow_fw = BiFormer(img1_prev, img3_prev)
  File "/home/botao/miniconda3/envs/BiFormer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/d/botao/BiFormer-main/model/BiFormer.py", line 186, in forward
    x = self.encoder(torch.cat([x1,x2], dim=0))
  File "/home/botao/miniconda3/envs/BiFormer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/d/botao/BiFormer-main/model/twins_encoder.py", line 24, in forward
    x, size = embed(x)
  File "/home/botao/miniconda3/envs/BiFormer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/botao/miniconda3/envs/BiFormer/lib/python3.9/site-packages/timm/models/twins.py", line 274, in forward
    x = self.proj(x).flatten(2).transpose(1, 2)
  File "/home/botao/miniconda3/envs/BiFormer/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/botao/miniconda3/envs/BiFormer/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/botao/miniconda3/envs/BiFormer/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [128, 3, 4, 4], expected input[2, 4, 256, 448] to have 3 channels, but got 4 channels instead

It seemed that I got the wrong pretrained model whose channels are mismatched.

Could you help me figure out what I did wrong? How can I solve this problem?

Thanks again.

Botao Peng

[JunHeum]

Thanks for your interest in our work!

I think below check list will help you solve the issue.

First, check the version of timm>=0.9.2.

Second, change pretrained=True in timm.create_model into pretrained=False.

timm.create_model('twins_svt_large',pretrained=True) first download the pretrained model from Hub and then load it from local cache.

But, it is not required to run our codes because provided BiFormer_Weights.zip contains the parameters of the encoder.

Hence, you can easily turn off the loading that option like below snippet. (I changed pretrained=True into pretrained=False) https://github.com/JunHeum/BiFormer/blob/91ddc9c29b1ab10eb91b5188317e7f4f4b3df981/model/BiFormer.py#L35-L41

[BotaoPeng98]

Thank you so much. May you be happy every day, and succeed in your work!