search

unum-cloud / uform

Pocket-Sized Multimodal AI for content understanding and generation across multilingual texts, images, and 🔜 video, up to 5x faster than OpenAI CLIP and LLaVA 🖼️ & 🖋️
https://unum-cloud.github.io/uform/
Apache License 2.0
1.01k stars 60 forks source link

Bug: can't load unum-cloud/uform-vl-english #28

Closed beaugunderson closed 1 year ago

beaugunderson commented 1 year ago

uform==0.2.1

import uform

model = uform.get_model('unum-cloud/uform-vl-english')

result:


lib/python3.11/site-packages/timm/models/_factory.py:114: UserWarning: Mapping deprecated model name deit3_base_patch16_224_in21ft1k to current deit3_base_patch16_224.fb_in22k_ft_in1k.
  model = create_fn(
Traceback (most recent call last):
  File "t.py", line 5, in <module>
    model = uform.get_model('unum-cloud/uform-vl-english')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/uform.py", line 484, in get_model
    model.text_encoder.load_state_dict(state['text_encoder'])
  File "lib/python3.11/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for TextEncoder:
    Unexpected key(s) in state_dict: "backbone.embeddings.position_ids".```
ashvardanian commented 1 year ago

@VoVoR and @kimihailv have we seen such issues before? Can this be coming from HuggingFace?

ashvardanian commented 1 year ago

Thank you, @beaugunderson! There was a breaking change in HuggingFace API. We have now patched it. Please upgrade UForm to v0.3.0 🤗