Can't load mosaicml/mpt-7b-instruct model

[almagest21]

Hi, thank you for this great ui. Today I tried to use mosaicml/mpt-7b-instruct model. downloading from webui is successufully finished but it can't be loaded. Here is error message:

Traceback (most recent call last): File "D:\dl\webui\text-generation-webui\server.py", line 917, in shared.model, shared.tokenizer = load_model(shared.model_name) File "D:\dl\webui\text-generation-webui\modules\models.py", line 74, in load_model shared.model_type = find_model_type(model_name) File "D:\dl\webui\text-generation-webui\modules\models.py", line 62, in find_model_type config = AutoConfig.from_pretrained(Path(f'{shared.args.model_dir}/{model_name}'), trust_remote_code=shared.args.trust_remote_code) File "D:\dl\webui\installer_files\env\lib\site-packages\transformers\models\auto\configuration_auto.py", line 919, in from_pretrained raise ValueError( ValueError: Loading models\mosaicml_mpt-7b-instruct requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error.

Could you please give me the file and location to add this trust_remote_code=True option?

[oobabooga]

Try --trust-remote-code

[almagest21]

@oobabooga Thanks. Could you teach me where to add this argument on start_windows.bat or other file?

[jepjoo]

INSTRUCTIONS.TXT tells you how to do this:

Adding flags like --chat, --notebook, etc

Edit the "webui.py" script using a text editor and add the desired flags to the line that says

run_cmd("python server.py --chat --model-menu")

For instance, to add the --notebook flag, change it to

run_cmd("python server.py --notebook --model-menu")

[hwamil]

There's no such flag named --trust-remote-code

edit: I was wrong. But even when enabled I get the same message still.

[SrVill]

There's no such flag named --trust-remote-code

edit: I was wrong. But even when enabled I get the same message still.

Same problem. After setting the flag (which can be enabled directly from the interface), an error will appear related to the lack of support for this model. Asked a question here: https://github.com/oobabooga/text-generation-webui/issues/1856

[SharkWipf]

Works fine for me (Fedora), just needed to pip install einops and launch with --trust-remote-code.

[Slug-Cat]

I've tried launching with --trust-remote-code and pip install einops and it still gives the same error: Traceback (most recent call last): File “E:\oobabooga\text-generation-webui[server.py](http://server.py/)”, line 103, in load_model_wrapper shared.model, shared.tokenizer = load_model(shared.model_name) File “E:\oobabooga\text-generation-webui\modules[models.py](http://models.py/)”, line 207, in load_model config = AutoConfig.from_pretrained(checkpoint) File “E:\oobabooga\installer_files\env\lib\site-packages\transformers\models\auto\configuration_auto.py”, line 919, in from_pretrained raise ValueError( ValueError: Loading models\mosaicml_mpt-7b-storywriter requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error.

[SharkWipf]

You might need to manually download/clone (make sure you have git-lfs enabled if you clone) the huggingface model repo to the models folder instead of auto-downloading it, that's what I did.

[NicolasMejiaPetit]

I've been running into the same issues. I tried the --trust_remote_code and pip install einops, and i get the same error as shown at the top of this thread.

[oobabooga]

Try updating the web ui, if trust-remote-code doesn't work then you are probably using an older version of the UI

[almagest21]

Sorry to confusing you, but when I wrote the --trust-remote-code argument on webui.py as jepjoo taught me, it was successfully proceeded. But I got another error about the missing einops module. Therefore, I installed it by adding "call pip install einops" before "call python webui.py" line. And tried again and got the following error:

C:\Users\alma/.cache\huggingface\modules\transformers_modules\mosaicml_mpt-7b-instruct\attention.py:148: UserWarning: Using attn_impl: torch. If your model does not use alibi or prefix_lm we recommend using attn_impl: flash otherwise we recommend using attn_impl: triton. warnings.warn('Using attn_impl: torch. If your model does not use alibi or ' + 'prefix_lm we recommend using attn_impl: flash otherwise ' + 'we recommend using attn_impl: triton.') Traceback (most recent call last): File "D:\dl\webui\text-generation-webui\server.py", line 872, in shared.model, shared.tokenizer = load_model(shared.model_name) File "D:\dl\webui\text-generation-webui\modules\models.py", line 219, in load_model model = LoaderClass.from_pretrained(checkpoint, **params) File "D:\dl\webui\installer_files\env\lib\site-packages\transformers\models\auto\auto_factory.py", line 466, in from_pretrained return model_class.from_pretrained( File "D:\dl\webui\installer_files\env\lib\site-packages\transformers\modeling_utils.py", line 2703, in from_pretrained raise ValueError(f"{model.class.name} does not support device_map='{device_map}' yet.") ValueError: MPTForCausalLM does not support device_map='auto' yet.

If anyone knows of this problem, I would appreciate any help.

[Slug-Cat]

I get this error when trying to run the model with --trust-remote-code at 16bit precision, and get the other error when trying to run the model at 8bit precision. I'm on the latest version of the UI on windows and have confirmed that the model folder matches the huggingface page of the model.

Sorry to confusing you, but when I wrote the --trust-remote-code argument on webui.py as jepjoo taught me, it was successfully proceeded. But I got another error about the missing einops module. Therefore, I installed it by adding "call pip install einops" before "call python webui.py" line. And tried again and got the following error:

C:\Users\alma/.cache\huggingface\modules\transformers_modules\mosaicml_mpt-7b-instruct\attention.py:148: UserWarning: Using attn_impl: torch. If your model does not use alibi or prefix_lm we recommend using attn_impl: flash otherwise we recommend using attn_impl: triton. warnings.warn('Using attn_impl: torch. If your model does not use alibi or ' + 'prefix_lm we recommend using attn_impl: flash otherwise ' + 'we recommend using attn_impl: triton.') Traceback (most recent call last): File "D:\dl\webui\text-generation-webui\server.py", line 872, in shared.model, shared.tokenizer = load_model(shared.model_name) File "D:\dl\webui\text-generation-webui\modules\models.py", line 219, in load_model model = LoaderClass.from_pretrained(checkpoint, params) File "D:\dl\webui\installer_files\env\lib\site-packages\transformers\models\auto\auto_factory.py", line 466, in from_pretrained return model_class.from_pretrained( File "D:\dl\webui\installer_files\env\lib\site-packages\transformers\modeling_utils.py", line 2703, in from_pretrained raise ValueError(f"{model.class.name**} does not support device_map='{device_map}' yet.") ValueError: MPTForCausalLM does not support device_map='auto' yet.

If anyone knows of this problem, I would appreciate any help.

[CarlKenner]

Once those errors are solved, you will also need instruction-following characters and prompts for mpt-instruct and mpt-chat, and for them to be automatically recognised, which I added to my pull request #1596.

In the meantime, for the mpt-7b-instruct model, you should scroll down in the chat window and set it to instruct then choose Alpaca from the dropdown menu.

But for the mpt-7b-chat model, it uses a variant of ChatML, so it won't work with the Alpaca option.

[Lesani]

WHERE is that webui.py? I can't find it. I got the remote code flag in the bat file, I can see trust_remote_code enabled in the "interface mode" tab, but still get the ", then set the option trust_remote_code=True to remove this error." error. I cannot find a "webui.py" file anywhere in the repo ?!?

[jepjoo]

WHERE is that webui.py? I can't find it. I got the remote code flag in the bat file, I can see trust_remote_code enabled in the "interface mode" tab, but still get the ", then set the option trust_remote_code=True to remove this error." error. I cannot find a "webui.py" file anywhere in the repo ?!?

It's included in the 1-click installers.

https://github.com/oobabooga/text-generation-webui/releases/tag/installers

I'm unsure if it's relevant to your method installation though. almagest21 mentioned start_windows.bat so I assumed they are using the 1-click installer in my reply to them.

[camenduru]

https://github.com/oobabooga/text-generation-webui/pull/1891

[DocShotgun]

It looks like people have managed to get the full versions of the model running, but has anyone had any luck with the 4bit quantized version? https://huggingface.co/OccamRazor/mpt-7b-storywriter-4bit-128g

It looks like occam made some edits to GPTQ to support mpt in his repo: https://github.com/0cc4m/GPTQ-for-LLaMa

Not a coder myself, but wondering if it would be possible to adapt those changes to a webui-compatible version of GPTQ-for-LLaMa?

[SrVill]

It looks like people have managed to get the full versions of the model running...

How did they do it? So far I'm stuck on a bug:

Traceback (most recent call last): File “E:\oobabooga_windows\text-generation-webui[server.py](http://server.py/)”, line 60, in load_model_wrapper shared.model, shared.tokenizer = load_model(shared.model_name) File “E:\oobabooga_windows\text-generation-webui\modules[models.py](http://models.py/)”, line 219, in load_model model = LoaderClass.from_pretrained(checkpoint, **params) File “E:\oobabooga_windows\installer_files\env\lib\site-packages\transformers\models\auto\auto_factory.py”, line 466, in from_pretrained return model_class.from_pretrained( File “E:\oobabooga_windows\installer_files\env\lib\site-packages\transformers\modeling_utils.py”, line 2703, in from_pretrained raise ValueError(f"{model.class.name} does not support device_map='{device_map}' yet.") ValueError: MPTForCausalLM does not support device_map='auto' yet.

[Lesani]

Uncheck the ‚auto devices‘ checkbox in the model area

[Supercabb]

@SrVill I solved the ValueError: MPTForCausalLM does not support device_map='auto' yet. I add a specific device_map before loading the model in the file modules/models.py

I can load all MPT-7b models with 8 bits in a RTX 3080 with this change.

           ....
            model.tie_weights()
            params['device_map'] = infer_auto_device_map(
                model,
                dtype=torch.int8,
                max_memory=params['max_memory'],
                no_split_module_classes=model._no_split_modules
            )

        params['device_map'] = {
    "transformer.blocks.0.attn.Wqkv.weight": 0,
    "transformer.blocks.0.attn.out_proj.weight": 0,
    "transformer.blocks.0.ffn.down_proj.weight": 0,
    "transformer.blocks.0.ffn.up_proj.weight": 0,
    "transformer.blocks.0.norm_1.weight": 0,
    "transformer.blocks.0.norm_2.weight": 0,
    "transformer.blocks.1.attn.Wqkv.weight": 0,
    "transformer.blocks.1.attn.out_proj.weight": 0,
    "transformer.blocks.1.ffn.down_proj.weight": 0,
    "transformer.blocks.1.ffn.up_proj.weight": 0,
    "transformer.blocks.1.norm_1.weight": 0,
    "transformer.blocks.1.norm_2.weight": 0,
    "transformer.blocks.10.attn.Wqkv.weight": 0,
    "transformer.blocks.10.attn.out_proj.weight": 0,
    "transformer.blocks.10.ffn.down_proj.weight": 0,
    "transformer.blocks.10.ffn.up_proj.weight": 0,
    "transformer.blocks.10.norm_1.weight": 0,
    "transformer.blocks.10.norm_2.weight": 0,
    "transformer.blocks.11.attn.Wqkv.weight": 0,
    "transformer.blocks.11.attn.out_proj.weight": 0,
    "transformer.blocks.11.ffn.down_proj.weight": 0,
    "transformer.blocks.11.ffn.up_proj.weight": 0,
    "transformer.blocks.11.norm_1.weight": 0,
    "transformer.blocks.11.norm_2.weight": 0,
    "transformer.blocks.12.attn.Wqkv.weight": 0,
    "transformer.blocks.12.attn.out_proj.weight": 0,
    "transformer.blocks.12.ffn.down_proj.weight": 0,
    "transformer.blocks.12.ffn.up_proj.weight": 0,
    "transformer.blocks.12.norm_1.weight": 0,
    "transformer.blocks.12.norm_2.weight": 0,
    "transformer.blocks.13.attn.Wqkv.weight": 0,
    "transformer.blocks.13.attn.out_proj.weight": 0,
    "transformer.blocks.13.ffn.down_proj.weight": 0,
    "transformer.blocks.13.ffn.up_proj.weight": 0,
    "transformer.blocks.13.norm_1.weight": 0,
    "transformer.blocks.13.norm_2.weight": 0,
    "transformer.blocks.14.attn.Wqkv.weight": 0,
    "transformer.blocks.14.attn.out_proj.weight": 0,
    "transformer.blocks.14.ffn.down_proj.weight": 0,
    "transformer.blocks.14.ffn.up_proj.weight": 0,
    "transformer.blocks.14.norm_1.weight": 0,
    "transformer.blocks.14.norm_2.weight": 0,
    "transformer.blocks.15.attn.Wqkv.weight": 0,
    "transformer.blocks.15.attn.out_proj.weight": 0,
    "transformer.blocks.15.ffn.down_proj.weight": 0,
    "transformer.blocks.15.ffn.up_proj.weight": 0,
    "transformer.blocks.15.norm_1.weight": 0,
    "transformer.blocks.15.norm_2.weight": 0,
    "transformer.blocks.16.attn.Wqkv.weight": 0,
    "transformer.blocks.16.attn.out_proj.weight": 0,
    "transformer.blocks.16.ffn.down_proj.weight": 0,
    "transformer.blocks.16.ffn.up_proj.weight": 0,
    "transformer.blocks.16.norm_1.weight": 0,
    "transformer.blocks.16.norm_2.weight": 0,
    "transformer.blocks.17.attn.Wqkv.weight": 0,
    "transformer.blocks.17.attn.out_proj.weight": 0,
    "transformer.blocks.17.ffn.down_proj.weight": 0,
    "transformer.blocks.17.ffn.up_proj.weight": 0,
    "transformer.blocks.17.norm_1.weight": 0,
    "transformer.blocks.17.norm_2.weight": 0,
    "transformer.blocks.18.attn.Wqkv.weight": 0,
    "transformer.blocks.18.attn.out_proj.weight": 0,
    "transformer.blocks.18.ffn.down_proj.weight": 0,
    "transformer.blocks.18.ffn.up_proj.weight": 0,
    "transformer.blocks.18.norm_1.weight": 0,
    "transformer.blocks.18.norm_2.weight": 0,
    "transformer.blocks.19.attn.Wqkv.weight": 0,
    "transformer.blocks.19.attn.out_proj.weight": 0,
    "transformer.blocks.19.ffn.down_proj.weight": 0,
    "transformer.blocks.19.ffn.up_proj.weight": 0,
    "transformer.blocks.19.norm_1.weight": 0,
    "transformer.blocks.19.norm_2.weight": 0,
    "transformer.blocks.2.attn.Wqkv.weight": 0,
    "transformer.blocks.2.attn.out_proj.weight": 0,
    "transformer.blocks.2.ffn.down_proj.weight": 0,
    "transformer.blocks.2.ffn.up_proj.weight": 0,
    "transformer.blocks.2.norm_1.weight": 0,
    "transformer.blocks.2.norm_2.weight": 0,
    "transformer.blocks.20.attn.Wqkv.weight": 0,
    "transformer.blocks.20.attn.out_proj.weight": 0,
    "transformer.blocks.20.ffn.down_proj.weight": 0,
    "transformer.blocks.20.ffn.up_proj.weight": 0,
    "transformer.blocks.20.norm_1.weight": 0,
    "transformer.blocks.20.norm_2.weight": 0,
    "transformer.blocks.21.attn.Wqkv.weight": 0,
    "transformer.blocks.21.attn.out_proj.weight": 0,
    "transformer.blocks.21.ffn.down_proj.weight": 0,
    "transformer.blocks.21.ffn.up_proj.weight": 0,
    "transformer.blocks.21.norm_1.weight": 0,
    "transformer.blocks.21.norm_2.weight": 0,
    "transformer.blocks.22.attn.Wqkv.weight": 0,
    "transformer.blocks.22.attn.out_proj.weight": 0,
    "transformer.blocks.22.ffn.down_proj.weight": 0,
    "transformer.blocks.22.ffn.up_proj.weight": 0,
    "transformer.blocks.22.norm_1.weight": 0,
    "transformer.blocks.22.norm_2.weight": 0,
    "transformer.blocks.23.attn.Wqkv.weight": 0,
    "transformer.blocks.23.attn.out_proj.weight": 0,
    "transformer.blocks.23.ffn.down_proj.weight": 0,
    "transformer.blocks.23.ffn.up_proj.weight": 0,
    "transformer.blocks.23.norm_1.weight": 0,
    "transformer.blocks.23.norm_2.weight": 0,
    "transformer.blocks.24.attn.Wqkv.weight": 0,
    "transformer.blocks.24.attn.out_proj.weight": 0,
    "transformer.blocks.24.ffn.down_proj.weight": 0,
    "transformer.blocks.24.ffn.up_proj.weight": 0,
    "transformer.blocks.24.norm_1.weight": 0,
    "transformer.blocks.24.norm_2.weight": 0,
    "transformer.blocks.25.attn.Wqkv.weight": 0,
    "transformer.blocks.25.attn.out_proj.weight": 0,
    "transformer.blocks.25.ffn.down_proj.weight": 0,
    "transformer.blocks.25.ffn.up_proj.weight": 0,
    "transformer.blocks.25.norm_1.weight": 0,
    "transformer.blocks.25.norm_2.weight": 0,
    "transformer.blocks.26.attn.Wqkv.weight": 0,
    "transformer.blocks.26.attn.out_proj.weight": 0,
    "transformer.blocks.26.ffn.down_proj.weight": 0,
    "transformer.blocks.26.ffn.up_proj.weight": 0,
    "transformer.blocks.26.norm_1.weight": 0,
    "transformer.blocks.26.norm_2.weight": 0,
    "transformer.blocks.27.attn.Wqkv.weight": 0,
    "transformer.blocks.27.attn.out_proj.weight": 0,
    "transformer.blocks.27.ffn.down_proj.weight": 0,
    "transformer.blocks.27.ffn.up_proj.weight": 0,
    "transformer.blocks.27.norm_1.weight": 0,
    "transformer.blocks.27.norm_2.weight": 0,
    "transformer.blocks.28.attn.Wqkv.weight": 0,
    "transformer.blocks.28.attn.out_proj.weight": 0,
    "transformer.blocks.28.ffn.down_proj.weight": 0,
    "transformer.blocks.28.ffn.up_proj.weight": 0,
    "transformer.blocks.28.norm_1.weight": 0,
    "transformer.blocks.28.norm_2.weight": 0,
    "transformer.blocks.29.attn.Wqkv.weight": 0,
    "transformer.blocks.29.attn.out_proj.weight": 0,
    "transformer.blocks.29.ffn.down_proj.weight": 0,
    "transformer.blocks.29.ffn.up_proj.weight": 0,
    "transformer.blocks.29.norm_1.weight": 0,
    "transformer.blocks.29.norm_2.weight": 0,
    "transformer.blocks.3.attn.Wqkv.weight": 0,
    "transformer.blocks.3.attn.out_proj.weight": 0,
    "transformer.blocks.3.ffn.down_proj.weight": 0,
    "transformer.blocks.3.ffn.up_proj.weight": 0,
    "transformer.blocks.3.norm_1.weight": 0,
    "transformer.blocks.3.norm_2.weight": 0,
    "transformer.blocks.30.attn.Wqkv.weight": 0,
    "transformer.blocks.30.attn.out_proj.weight": 0,
    "transformer.blocks.30.ffn.down_proj.weight": 0,
    "transformer.blocks.30.ffn.up_proj.weight": 0,
    "transformer.blocks.30.norm_1.weight": 0,
    "transformer.blocks.30.norm_2.weight": 0,
    "transformer.blocks.31.attn.Wqkv.weight": 0,
    "transformer.blocks.31.attn.out_proj.weight": 0,
    "transformer.blocks.31.ffn.down_proj.weight": 0,
    "transformer.blocks.31.ffn.up_proj.weight": 0,
    "transformer.blocks.31.norm_1.weight": 0,
    "transformer.blocks.31.norm_2.weight": 0,
    "transformer.blocks.4.attn.Wqkv.weight": 0,
    "transformer.blocks.4.attn.out_proj.weight": 0,
    "transformer.blocks.4.ffn.down_proj.weight": 0,
    "transformer.blocks.4.ffn.up_proj.weight": 0,
    "transformer.blocks.4.norm_1.weight": 0,
    "transformer.blocks.4.norm_2.weight": 0,
    "transformer.blocks.5.attn.Wqkv.weight": 0,
    "transformer.blocks.5.attn.out_proj.weight": 0,
    "transformer.blocks.5.ffn.down_proj.weight": 0,
    "transformer.blocks.5.ffn.up_proj.weight": 0,
    "transformer.blocks.5.norm_1.weight": 0,
    "transformer.blocks.5.norm_2.weight": 0,
    "transformer.blocks.6.attn.Wqkv.weight": 0,
    "transformer.blocks.6.attn.out_proj.weight": 0,
    "transformer.blocks.6.ffn.down_proj.weight": 0,
    "transformer.blocks.6.ffn.up_proj.weight": 0,
    "transformer.blocks.6.norm_1.weight": 0,
    "transformer.blocks.6.norm_2.weight": 0,
    "transformer.blocks.7.attn.Wqkv.weight": 0,
    "transformer.blocks.7.attn.out_proj.weight": 0,
    "transformer.blocks.7.ffn.down_proj.weight": 0,
    "transformer.blocks.7.ffn.up_proj.weight": 0,
    "transformer.blocks.7.norm_1.weight": 0,
    "transformer.blocks.7.norm_2.weight": 0,
    "transformer.blocks.8.attn.Wqkv.weight": 0,
    "transformer.blocks.8.attn.out_proj.weight": 0,
    "transformer.blocks.8.ffn.down_proj.weight": 0,
    "transformer.blocks.8.ffn.up_proj.weight": 0,
    "transformer.blocks.8.norm_1.weight": 0,
    "transformer.blocks.8.norm_2.weight": 0,
    "transformer.blocks.9.attn.Wqkv.weight": 0,
    "transformer.blocks.9.attn.out_proj.weight": 0,
    "transformer.blocks.9.ffn.down_proj.weight": 0,
    "transformer.blocks.9.ffn.up_proj.weight": 0,
    "transformer.blocks.9.norm_1.weight": 0,
    "transformer.blocks.9.norm_2.weight": 0,
    "transformer.norm_f.weight": 0,
    "transformer.wte.weight": 0,
    "transformer.blocks.0.attn.Wqkv.SCB": 0,
    "transformer.blocks.0.attn.out_proj.SCB": 0,
    "transformer.blocks.0.ffn.down_proj.SCB": 0,
    "transformer.blocks.0.ffn.up_proj.SCB": 0,
    "transformer.blocks.0.norm_1.SCB": 0,
    "transformer.blocks.0.norm_2.SCB": 0,
    "transformer.blocks.1.attn.Wqkv.SCB": 0,
    "transformer.blocks.1.attn.out_proj.SCB": 0,
    "transformer.blocks.1.ffn.down_proj.SCB": 0,
    "transformer.blocks.1.ffn.up_proj.SCB": 0,
    "transformer.blocks.1.norm_1.SCB": 0,
    "transformer.blocks.1.norm_2.SCB": 0,
    "transformer.blocks.10.attn.Wqkv.SCB": 0,
    "transformer.blocks.10.attn.out_proj.SCB": 0,
    "transformer.blocks.10.ffn.down_proj.SCB": 0,
    "transformer.blocks.10.ffn.up_proj.SCB": 0,
    "transformer.blocks.10.norm_1.SCB": 0,
    "transformer.blocks.10.norm_2.SCB": 0,
    "transformer.blocks.11.attn.Wqkv.SCB": 0,
    "transformer.blocks.11.attn.out_proj.SCB": 0,
    "transformer.blocks.11.ffn.down_proj.SCB": 0,
    "transformer.blocks.11.ffn.up_proj.SCB": 0,
    "transformer.blocks.11.norm_1.SCB": 0,
    "transformer.blocks.11.norm_2.SCB": 0,
    "transformer.blocks.12.attn.Wqkv.SCB": 0,
    "transformer.blocks.12.attn.out_proj.SCB": 0,
    "transformer.blocks.12.ffn.down_proj.SCB": 0,
    "transformer.blocks.12.ffn.up_proj.SCB": 0,
    "transformer.blocks.12.norm_1.SCB": 0,
    "transformer.blocks.12.norm_2.SCB": 0,
    "transformer.blocks.13.attn.Wqkv.SCB": 0,
    "transformer.blocks.13.attn.out_proj.SCB": 0,
    "transformer.blocks.13.ffn.down_proj.SCB": 0,
    "transformer.blocks.13.ffn.up_proj.SCB": 0,
    "transformer.blocks.13.norm_1.SCB": 0,
    "transformer.blocks.13.norm_2.SCB": 0,
    "transformer.blocks.14.attn.Wqkv.SCB": 0,
    "transformer.blocks.14.attn.out_proj.SCB": 0,
    "transformer.blocks.14.ffn.down_proj.SCB": 0,
    "transformer.blocks.14.ffn.up_proj.SCB": 0,
    "transformer.blocks.14.norm_1.SCB": 0,
    "transformer.blocks.14.norm_2.SCB": 0,
    "transformer.blocks.15.attn.Wqkv.SCB": 0,
    "transformer.blocks.15.attn.out_proj.SCB": 0,
    "transformer.blocks.15.ffn.down_proj.SCB": 0,
    "transformer.blocks.15.ffn.up_proj.SCB": 0,
    "transformer.blocks.15.norm_1.SCB": 0,
    "transformer.blocks.15.norm_2.SCB": 0,
    "transformer.blocks.16.attn.Wqkv.SCB": 0,
    "transformer.blocks.16.attn.out_proj.SCB": 0,
    "transformer.blocks.16.ffn.down_proj.SCB": 0,
    "transformer.blocks.16.ffn.up_proj.SCB": 0,
    "transformer.blocks.16.norm_1.SCB": 0,
    "transformer.blocks.16.norm_2.SCB": 0,
    "transformer.blocks.17.attn.Wqkv.SCB": 0,
    "transformer.blocks.17.attn.out_proj.SCB": 0,
    "transformer.blocks.17.ffn.down_proj.SCB": 0,
    "transformer.blocks.17.ffn.up_proj.SCB": 0,
    "transformer.blocks.17.norm_1.SCB": 0,
    "transformer.blocks.17.norm_2.SCB": 0,
    "transformer.blocks.18.attn.Wqkv.SCB": 0,
    "transformer.blocks.18.attn.out_proj.SCB": 0,
    "transformer.blocks.18.ffn.down_proj.SCB": 0,
    "transformer.blocks.18.ffn.up_proj.SCB": 0,
    "transformer.blocks.18.norm_1.SCB": 0,
    "transformer.blocks.18.norm_2.SCB": 0,
    "transformer.blocks.19.attn.Wqkv.SCB": 0,
    "transformer.blocks.19.attn.out_proj.SCB": 0,
    "transformer.blocks.19.ffn.down_proj.SCB": 0,
    "transformer.blocks.19.ffn.up_proj.SCB": 0,
    "transformer.blocks.19.norm_1.SCB": 0,
    "transformer.blocks.19.norm_2.SCB": 0,
    "transformer.blocks.2.attn.Wqkv.SCB": 0,
    "transformer.blocks.2.attn.out_proj.SCB": 0,
    "transformer.blocks.2.ffn.down_proj.SCB": 0,
    "transformer.blocks.2.ffn.up_proj.SCB": 0,
    "transformer.blocks.2.norm_1.SCB": 0,
    "transformer.blocks.2.norm_2.SCB": 0,
    "transformer.blocks.20.attn.Wqkv.SCB": 0,
    "transformer.blocks.20.attn.out_proj.SCB": 0,
    "transformer.blocks.20.ffn.down_proj.SCB": 0,
    "transformer.blocks.20.ffn.up_proj.SCB": 0,
    "transformer.blocks.20.norm_1.SCB": 0,
    "transformer.blocks.20.norm_2.SCB": 0,
    "transformer.blocks.21.attn.Wqkv.SCB": 0,
    "transformer.blocks.21.attn.out_proj.SCB": 0,
    "transformer.blocks.21.ffn.down_proj.SCB": 0,
    "transformer.blocks.21.ffn.up_proj.SCB": 0,
    "transformer.blocks.21.norm_1.SCB": 0,
    "transformer.blocks.21.norm_2.SCB": 0,
    "transformer.blocks.22.attn.Wqkv.SCB": 0,
    "transformer.blocks.22.attn.out_proj.SCB": 0,
    "transformer.blocks.22.ffn.down_proj.SCB": 0,
    "transformer.blocks.22.ffn.up_proj.SCB": 0,
    "transformer.blocks.22.norm_1.SCB": 0,
    "transformer.blocks.22.norm_2.SCB": 0,
    "transformer.blocks.23.attn.Wqkv.SCB": 0,
    "transformer.blocks.23.attn.out_proj.SCB": 0,
    "transformer.blocks.23.ffn.down_proj.SCB": 0,
    "transformer.blocks.23.ffn.up_proj.SCB": 0,
    "transformer.blocks.23.norm_1.SCB": 0,
    "transformer.blocks.23.norm_2.SCB": 0,
    "transformer.blocks.24.attn.Wqkv.SCB": 0,
    "transformer.blocks.24.attn.out_proj.SCB": 0,
    "transformer.blocks.24.ffn.down_proj.SCB": 0,
    "transformer.blocks.24.ffn.up_proj.SCB": 0,
    "transformer.blocks.24.norm_1.SCB": 0,
    "transformer.blocks.24.norm_2.SCB": 0,
    "transformer.blocks.25.attn.Wqkv.SCB": 0,
    "transformer.blocks.25.attn.out_proj.SCB": 0,
    "transformer.blocks.25.ffn.down_proj.SCB": 0,
    "transformer.blocks.25.ffn.up_proj.SCB": 0,
    "transformer.blocks.25.norm_1.SCB": 0,
    "transformer.blocks.25.norm_2.SCB": 0,
    "transformer.blocks.26.attn.Wqkv.SCB": 0,
    "transformer.blocks.26.attn.out_proj.SCB": 0,
    "transformer.blocks.26.ffn.down_proj.SCB": 0,
    "transformer.blocks.26.ffn.up_proj.SCB": 0,
    "transformer.blocks.26.norm_1.SCB": 0,
    "transformer.blocks.26.norm_2.SCB": 0,
    "transformer.blocks.27.attn.Wqkv.SCB": 0,
    "transformer.blocks.27.attn.out_proj.SCB": 0,
    "transformer.blocks.27.ffn.down_proj.SCB": 0,
    "transformer.blocks.27.ffn.up_proj.SCB": 0,
    "transformer.blocks.27.norm_1.SCB": 0,
    "transformer.blocks.27.norm_2.SCB": 0,
    "transformer.blocks.28.attn.Wqkv.SCB": 0,
    "transformer.blocks.28.attn.out_proj.SCB": 0,
    "transformer.blocks.28.ffn.down_proj.SCB": 0,
    "transformer.blocks.28.ffn.up_proj.SCB": 0,
    "transformer.blocks.28.norm_1.SCB": 0,
    "transformer.blocks.28.norm_2.SCB": 0,
    "transformer.blocks.29.attn.Wqkv.SCB": 0,
    "transformer.blocks.29.attn.out_proj.SCB": 0,
    "transformer.blocks.29.ffn.down_proj.SCB": 0,
    "transformer.blocks.29.ffn.up_proj.SCB": 0,
    "transformer.blocks.29.norm_1.SCB": 0,
    "transformer.blocks.29.norm_2.SCB": 0,
    "transformer.blocks.3.attn.Wqkv.SCB": 0,
    "transformer.blocks.3.attn.out_proj.SCB": 0,
    "transformer.blocks.3.ffn.down_proj.SCB": 0,
    "transformer.blocks.3.ffn.up_proj.SCB": 0,
    "transformer.blocks.3.norm_1.SCB": 0,
    "transformer.blocks.3.norm_2.SCB": 0,
    "transformer.blocks.30.attn.Wqkv.SCB": 0,
    "transformer.blocks.30.attn.out_proj.SCB": 0,
    "transformer.blocks.30.ffn.down_proj.SCB": 0,
    "transformer.blocks.30.ffn.up_proj.SCB": 0,
    "transformer.blocks.30.norm_1.SCB": 0,
    "transformer.blocks.30.norm_2.SCB": 0,
    "transformer.blocks.31.attn.Wqkv.SCB": 0,
    "transformer.blocks.31.attn.out_proj.SCB": 0,
    "transformer.blocks.31.ffn.down_proj.SCB": 0,
    "transformer.blocks.31.ffn.up_proj.SCB": 0,
    "transformer.blocks.31.norm_1.SCB": 0,
    "transformer.blocks.31.norm_2.SCB": 0,
    "transformer.blocks.4.attn.Wqkv.SCB": 0,
    "transformer.blocks.4.attn.out_proj.SCB": 0,
    "transformer.blocks.4.ffn.down_proj.SCB": 0,
    "transformer.blocks.4.ffn.up_proj.SCB": 0,
    "transformer.blocks.4.norm_1.SCB": 0,
    "transformer.blocks.4.norm_2.SCB": 0,
    "transformer.blocks.5.attn.Wqkv.SCB": 0,
    "transformer.blocks.5.attn.out_proj.SCB": 0,
    "transformer.blocks.5.ffn.down_proj.SCB": 0,
    "transformer.blocks.5.ffn.up_proj.SCB": 0,
    "transformer.blocks.5.norm_1.SCB": 0,
    "transformer.blocks.5.norm_2.SCB": 0,
    "transformer.blocks.6.attn.Wqkv.SCB": 0,
    "transformer.blocks.6.attn.out_proj.SCB": 0,
    "transformer.blocks.6.ffn.down_proj.SCB": 0,
    "transformer.blocks.6.ffn.up_proj.SCB": 0,
    "transformer.blocks.6.norm_1.SCB": 0,
    "transformer.blocks.6.norm_2.SCB": 0,
    "transformer.blocks.7.attn.Wqkv.SCB": 0,
    "transformer.blocks.7.attn.out_proj.SCB": 0,
    "transformer.blocks.7.ffn.down_proj.SCB": 0,
    "transformer.blocks.7.ffn.up_proj.SCB": 0,
    "transformer.blocks.7.norm_1.SCB": 0,
    "transformer.blocks.7.norm_2.SCB": 0,
    "transformer.blocks.8.attn.Wqkv.SCB": 0,
    "transformer.blocks.8.attn.out_proj.SCB": 0,
    "transformer.blocks.8.ffn.down_proj.SCB": 0,
    "transformer.blocks.8.ffn.up_proj.SCB": 0,
    "transformer.blocks.8.norm_1.SCB": 0,
    "transformer.blocks.8.norm_2.SCB": 0,
    "transformer.blocks.9.attn.Wqkv.SCB": 0,
    "transformer.blocks.9.attn.out_proj.SCB": 0,
    "transformer.blocks.9.ffn.down_proj.SCB": 0,
    "transformer.blocks.9.ffn.up_proj.SCB": 0,
    "transformer.blocks.9.norm_1.SCB": 0,
    "transformer.blocks.9.norm_2.SCB": 0,
    "transformer.norm_f.SCB": 0,
    "transformer.wte.SCB": 0
  }

        model = LoaderClass.from_pretrained(checkpoint, **params)
        ....

[ajallooeian]

I think I found the issue:

This line should be copied before if/else: https://github.com/oobabooga/text-generation-webui/blob/main/modules/models.py#LL172C43-L172C60

so if you use cpu it would still work! Let me know if a PR is needed.

[YoussefMP]

I added the flag "--trust-remote-code" under webui.py l.164, it fixed it for me.

[anuragupadhyaya]

This works for me.

base_model = AutoModelForCausalLM.from_pretrained("mosaicml/mpt-7b-instruct",
                                                        offload_folder="model/",
                                                        trust_remote_code=True,
                                                        device_map="auto")

[franciscolemos]

--trust-remote-code

[ChrisNonyminus]

--trust-remote-code

Not working.

[eizoxx]

I can't start the web ui for some reason. I always get in the cmd_windows.bat this error:

(D:\AI\LLM\oobabooga_windows\installer_files\env) D:\AI\LLM\oobabooga_windows\text-generation-webui>python server.py --model mosaicml_mpt-7b-storywriter --listen --trust-remote-code --bf16 2023-08-23 00:24:58 WARNING:trust_remote_code is enabled. This is dangerous. Traceback (most recent call last): File "D:\AI\LLM\oobabooga_windows\text-generation-webui\server.py", line 28, in from modules import ( File "D:\AI\LLM\oobabooga_windows\text-generation-webui\modules\chat.py", line 17, in from modules.text_generation import ( File "D:\AI\LLM\oobabooga_windows\text-generation-webui\modules\text_generation.py", line 23, in from modules.models import clear_torch_cache, local_rank File "D:\AI\LLM\oobabooga_windows\text-generation-webui\modules\models.py", line 148

• config = transformers.AutoConfig.from_pretrained( ^ IndentationError: unindent does not match any outer indentation level

I followed every step from: https://youtu.be/EvM0A6d_KSA https://youtu.be/QVVb6Md6huA

But I can't get it to run. Can anyone help?

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.

[Eddcapone]

This is still relevant. Please reopen.

[EvgeneKuklin]

Use CMD_FLAGS.TXT