Failed to load language model file

[postsun]

Hello, I'm developing chatbot models with Unreal Engine. You made a great plugin and I'm trying to use it, but I can't load the language model. I'm using it in Windows environment.

In the 'LlamaComponent.cpp', 'LlamaComponent.h files', I changed the pathToModel variable to: pathToModel="D:\unreal_project\PlayfabMenu_retire_3\Plugins\UELlama\Content\llama-2-13b-chat.ggmlv3.q4_0.bin";

Obviously, the file exists in the folder, but in the logs I get the error message "LogTemp: Error: 0000043995E7C000 unable to load model". What can I do to resolve this issue?

[mika314]

I think @matiasgql updated the code to the newer format, and you need to download a different type of the model; instead of ggml, you need gguf type.

@matiasgql, correct me if I am wrong.

[mika314]

Alternatively, you can use the previous commit.

[matiasgql]

In the newer versions of llama.cpp, GGML format is replaced by GGUF v2, fortunately you can update your model with a python script from the llama.cpp repo called "convert-llama-ggml-to-gguf.py" or downgrade the llama.cpp version with some changes in the code of the plugin, also you can use a previous commit as Mika said.

[postsun]

I think @matiasgql updated the code to the newer format, and you need to download a different type of the model; instead of ggml, you need gguf type.

@matiasgql, correct me if I am wrong.

That fixed it, thanks for making a great plugin.

[postsun]

In the newer versions of llama.cpp, GGML format is replaced by GGUF v2, fortunately you can update your model with a python script from the llama.cpp repo called "convert-llama-ggml-to-gguf.py" or downgrade the llama.cpp version with some changes in the code of the plugin, also you can use a previous commit as Mika said.

Thanks to your detailed explanation, I was able to find the .py file and convert it just fine. Thanks.

[oivio]

Now I did come across same issue. Maybe I should start from question if model should be placed inside "Content" Plugin folder? As recommended to use gguf so I did try to run "llama-2-13b-chat.Q8_0.gguf" from: https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/tree/main and placed inside "Plugins\UELlama\Content\models\" also set on component:

Getting error: LogTemp: Error: 00000A3F2E958600 unable to load model

I did also convert llama-2-13b-chat.ggmlv3.q4_0.bin with convert-llama-ggml-to-gguf.py but same issue.

[mika314]

Just provide the full path to the model.

[oivio]

Just provide the full path to the model.

Thank you, that actually did work.

Just out of topic if can I ask. Does anyone know how to stop auto-complete? So AI wont speak for yourself?

[mika314]

You can use stopSequences.

[postsun]

Yes, I understand, and I'll try to implement your suggestions.

Thank you for letting me know, I've asked you several times.

-----Original Message----- From: "Mika @.> To: @.>; Cc: @.>; @.>; Sent: 2023-10-10 (화) 01:32:21 (GMT+09:00) Subject: Re: [mika314/UELlama] Failed to load language model file (Issue #6)

You can use stopSequences. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[shuffleops]

You can use stopSequences.

Hey @mika314, How do you use the stopSequences?

[mika314]

A stop sequence is the text that will instruct the model to stop generating as soon as the model generates the stop sequence. An example of a stop sequence could be: '\nUser:'