Error when loading multimodal extension

[RodriMora]

Describe the bug

When I toggle the option for multimodal the software crashes.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Turning on the "multimodal" toggle in the "Session" tab throws this error and crashes textgen webui.

Screenshot

Logs

╭───────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /home/ubuntu/text-generation-webui/server.py:263 in <module>                                                                                                                                                                        │
│                                                                                                                                                                                                                                     │
│   262                 time.sleep(0.5)                                                                                                                                                                                               │
│ ❱ 263                 create_interface()                                                                                                                                                                                            │
│   264                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                     │
│ /home/ubuntu/text-generation-webui/server.py:156 in create_interface                                                                                                                                                                │
│                                                                                                                                                                                                                                     │
│   155         extensions_module.create_extensions_tabs()  # Extensions tabs                                                                                                                                                         │
│ ❱ 156         extensions_module.create_extensions_block()  # Extensions block                                                                                                                                                       │
│   157                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                     │
│ /home/ubuntu/text-generation-webui/modules/extensions.py:199 in create_extensions_block                                                                                                                                             │
│                                                                                                                                                                                                                                     │
│   198                 extension, _ = row                                                                                                                                                                                            │
│ ❱ 199                 extension.ui()                                                                                                                                                                                                │
│   200                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                     │
│ /home/ubuntu/text-generation-webui/extensions/multimodal/script.py:99 in ui                                                                                                                                                         │
│                                                                                                                                                                                                                                     │
│    98     global multimodal_embedder                                                                                                                                                                                                │
│ ❱  99     multimodal_embedder = MultimodalEmbedder(params)                                                                                                                                                                          │
│   100     with gr.Column():                                                                                                                                                                                                         │
│                                                                                                                                                                                                                                     │
│ /home/ubuntu/text-generation-webui/extensions/multimodal/multimodal_embedder.py:27 in __init__                                                                                                                                      │
│                                                                                                                                                                                                                                     │
│    26     def __init__(self, params: dict):                                                                                                                                                                                         │
│ ❱  27         pipeline, source = load_pipeline(params)                                                                                                                                                                              │
│    28         self.pipeline = pipeline                                                                                                                                                                                              │
│                                                                                                                                                                                                                                     │
│ /home/ubuntu/text-generation-webui/extensions/multimodal/pipeline_loader.py:34 in load_pipeline                                                                                                                                     │
│                                                                                                                                                                                                                                     │
│   33     else:                                                                                                                                                                                                                      │
│ ❱ 34         model_name = shared.args.model.lower()                                                                                                                                                                                 │
│   35         for k in pipeline_modules:                                                                                                                                                                                             │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'NoneType' object has no attribute 'lower'

System Info

Running locally. Latest version as per this date, just run the update script with option A) and B)

CPU: EPYC 7402
RAM: 512GB
GPU: 4x 3090

Ubuntu 22.04.4

Nvidia driver 550.54.15

[Touch-Night]

You should run multimodal model with command flags, not just simply check it in Session tab.

[RodriMora]

You should run multimodal model with command flags, not just simply check it in Session tab.

you mean add --multimodal in the CMD_FLAGS.txt?

[Touch-Night]

Yes, but multimodal extension is now buggy. I failed to load wojtab_llava-7b-v0-4bit-128g model and TheBloke_vicuna-7B-1.1-GPTQ doesn't seems to work.

[raffaeler]

@Touch-Night When I add --multimodal in CMD_FLAGS.txt, I get the following error:

server.py: error: argument --multimodal-pipeline: expected one argument

[Touch-Night]

It seems broken right now, I cannot load multimodal models either.

[raffaeler]

Thank you for confirming @Touch-Night

[daveposh]

@Touch-Night When I add --multimodal in CMD_FLAGS.txt, I get the following error:

server.py: error: argument --multimodal-pipeline: expected one argument

you need to load a pipeline with the CMD FLAG example: --multimodal-pipeline llava-7b

Available pipelines ['llava-7b','llava-13b', 'llava-llama-2-13b', 'llava-v1.5-13b', 'llava-v1.5-7b']

[raffaeler]

Available pipelines ['llava-7b','llava-13b', 'llava-llama-2-13b', 'llava-v1.5-13b', 'llava-v1.5-7b']

what about phi-3-vision instead?

TIA

[daveposh]

Available pipelines ['llava-7b','llava-13b', 'llava-llama-2-13b', 'llava-v1.5-13b', 'llava-v1.5-7b']

@Touch-Night what about phi-3-vision instead?

TIA

Install them under extensions/multimodal/pipelines

Default available are: ['llava-7b','llava-13b', 'llava-llama-2-13b', 'llava-v1.5-13b', 'llava-v1.5-7b']

[daveposh]

Multimodal Support

[Touch-Night]

https://github.com/oobabooga/text-generation-webui/pull/6089