TimmBackbone.from_pretrained out_indices not working correctly (out_indices is sorted using strings blocks.11 before blocks.3)

f-fuchs commented 1 month ago

System Info

transformers version: 4.42.4
Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Python version: 3.10.14
Huggingface_hub version: 0.23.5
Safetensors version: 0.4.3
Accelerate version: not installed
Accelerate config: not found
PyTorch version (GPU?): 2.3.1+cu121 (False)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using distributed or parallel set-up in script?: no

Who can help?

@amyeroberts

Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

from transformers import TimmBackbone
backbone = TimmBackbone.from_pretrained("eva02_base_patch14_224.mim_in22k", out_indices=[3, 5, 7, 11], num_channels=1)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[20], [line 1](vscode-notebook-cell:?execution_count=20&line=1)
----> [1](vscode-notebook-cell:?execution_count=20&line=1) backbone = TimmBackbone.from_pretrained(
      [2](vscode-notebook-cell:?execution_count=20&line=2)     "eva02_base_patch14_224.mim_in22k",
      [3](vscode-notebook-cell:?execution_count=20&line=3)     out_indices=[3, 5, 7, 11],
      [4](vscode-notebook-cell:?execution_count=20&line=4)     num_channels=1,
      [5](vscode-notebook-cell:?execution_count=20&line=5)     # out_features=[["blocks.3", "blocks.5", "blocks.7", "blocks.11"]],
      [6](vscode-notebook-cell:?execution_count=20&line=6) )

File /mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:113, in TimmBackbone.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    [105](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:105) out_indices = kwargs.pop("out_indices", config.out_indices)
    [106](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:106) config = TimmBackboneConfig(
    [107](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:107)     backbone=pretrained_model_name_or_path,
    [108](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:108)     num_channels=num_channels,
   (...)
    [111](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:111)     out_indices=out_indices,
    [112](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:112) )
--> [113](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:113) return super()._from_config(config, **kwargs)

File /mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py:1428, in PreTrainedModel._from_config(cls, config, **kwargs)
   [1426](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py:1426)         model = cls(config, **kwargs)
   [1427](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py:1427) else:
-> [1428](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py:1428)     model = cls(config, **kwargs)
   [1430](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py:1430) # restore default dtype if it was modified
   [1431](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py:1431) if dtype_orig is not None:

File /mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:89, in TimmBackbone.__init__(self, config, **kwargs)
     [85](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:85) self._return_layers = {
     [86](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:86)     layer["module"]: str(layer["index"]) for layer in self._backbone.feature_info.get_dicts()
     [87](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:87) }
     [88](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:88) self._all_layers = {layer["module"]: str(i) for i, layer in enumerate(self._backbone.feature_info.info)}
---> [89](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/models/timm_backbone/modeling_timm_backbone.py:89) super()._init_backbone(config)

File /mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:190, in BackboneMixin._init_backbone(self, config)
    [187](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:187) self.backbone_type = BackboneType.TIMM if self.use_timm_backbone else BackboneType.TRANSFORMERS
    [189](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:189) if self.backbone_type == BackboneType.TIMM:
--> [190](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:190)     self._init_timm_backbone(config)
    [191](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:191) elif self.backbone_type == BackboneType.TRANSFORMERS:
    [192](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:192)     self._init_transformers_backbone(config)

File /mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:162, in BackboneMixin._init_timm_backbone(self, config)
    [159](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:159) out_features = self._backbone.feature_info.module_name()
    [161](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:161) # We verify the out indices and out features are valid
--> [162](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:162) verify_out_features_out_indices(
    [163](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:163)     out_features=out_features, out_indices=out_indices, stage_names=self.stage_names
    [164](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:164) )
    [165](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:165) self._out_features, self._out_indices = out_features, out_indices

File /mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:49, in verify_out_features_out_indices(out_features, out_indices, stage_names)
     [47](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:47)         raise ValueError(f"out_features must not contain any duplicates, got {out_features}")
     [48](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:48)     if out_features != (sorted_feats := [feat for feat in stage_names if feat in out_features]):
---> [49](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:49)         raise ValueError(
     [50](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:50)             f"out_features must be in the same order as stage_names, expected {sorted_feats} got {out_features}"
     [51](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:51)         )
     [53](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:53) if out_indices is not None:
     [54](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/Users/fuchsfa/Desktop/Git/foundation-models/.venv/lib/python3.10/site-packages/transformers/utils/backbone_utils.py:54)     if not isinstance(out_indices, list):

ValueError: out_features must be in the same order as stage_names, expected ['blocks.3', 'blocks.5', 'blocks.7', 'blocks.11'] got ['blocks.11', 'blocks.3', 'blocks.5', 'blocks.7']

Expected behavior

The could should load the model correctly.

I think this line if positive_indices != tuple(sorted(positive_indices)): (https://github.com/huggingface/transformers/blame/main/src/transformers/utils/backbone_utils.py#L64) is the problem and sorted should be replaced by human/natural sorting.

amyeroberts commented 1 month ago

Hi @f-fuchs, thanks for raising this!

Yes, this is definitely not something we want. Unfortunately, this seems to be coming from timm itself:

In [16]: import timm

In [17]: model = timm.create_model(
    ...:     'eva02_base_patch14_224.mim_in22k',
    ...:     pretrained=True,
    ...:     features_only=True,
    ...:     in_chans=1,
    ...:     out_indices=[3, 5, 7, 11]
    ...: )

In [18]: model.feature_info.out_indices
Out[18]: (11, 3, 5, 7)

@rwightman What's the best / recommended way to get the indices according to the order they're returned in timm?

rwightman commented 1 month ago

@amyeroberts @f-fuchs oh fudge, I see the issue, I used sets to resolve the indices and they don't keep the order, unlike dicts (really wish they'd updated sets when they updated dicts for consistent order).

Anyways, I'll fix this ASAP, but it's worth noting that ONLY the info is out of order, the features are returned in order of layer iteration so they will be correct...

github-actions[bot] commented 2 weeks ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

huggingface / transformers