Error loading model with device_map="auto" for AutoModelForVisualQuestionAnswering in visual-question-answering pipeline

[chakravarthik27]

System Info

• transformers version: 4.44.2
• Platform: Windows-10-10.0.22631-SP0
• Python version: 3.9.13
• Huggingface_hub version: 0.24.7
• Safetensors version: 0.4.5
• Accelerate version: 0.34.2
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1+cpu (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:

Who can help?

@Rocketknight1

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction

from transformers import pipeline

pipe = pipeline("visual-question-answering", model=path, device_map="auto")

Expected behavior

it should be support for all type of models under visual-question-answering pipeline

[chakravarthik27]

ValueError: Could not load model Salesforce/blip-vqa-base with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForVisualQuestionAnswering'>, <class 'transformers.models.blip.modeling_blip.BlipForQuestionAnswering'>). See the original errors:

while loading with AutoModelForVisualQuestionAnswering, an error is thrown:
Traceback (most recent call last):
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\pipelines\base.py", line 286, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\modeling_utils.py", line 3875, in from_pretrained
    no_split_modules = model._get_no_split_modules(device_map)
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\modeling_utils.py", line 1979, in _get_no_split_modules
    raise ValueError(
ValueError: BlipForQuestionAnswering does not support `device_map='auto'`. To implement support, the model class needs to implement the `_no_split_modules` attribute.

while loading with BlipForQuestionAnswering, an error is thrown:
Traceback (most recent call last):
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\pipelines\base.py", line 286, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\modeling_utils.py", line 3875, in from_pretrained
    no_split_modules = model._get_no_split_modules(device_map)
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\modeling_utils.py", line 1979, in _get_no_split_modules
    raise ValueError(
ValueError: BlipForQuestionAnswering does not support `device_map='auto'`. To implement support, the model class needs to implement the `_no_split_modules` attribute.