ValidationError: 1 validation error for LlamaCpp n_gpu_layers extra fields not permitted (type=value_error.extra)

[neeewwww]

.env

Generic

TEXT_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2 TEXT_EMBEDDINGS_MODEL_TYPE=HF # LlamaCpp or HF USE_MLOCK=true

Ingestion

PERSIST_DIRECTORY=db DOCUMENTS_DIRECTORY=source_documents INGEST_CHUNK_SIZE=500 INGEST_CHUNK_OVERLAP=50

Generation

MODEL_TYPE=LlamaCpp # GPT4All or LlamaCpp MODEL_PATH=eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin MODEL_TEMP=0.8 MODEL_N_CTX=1024 # Max total size of prompt+answer MODEL_MAX_TOKENS=256 # Max size of answer MODEL_STOP=[STOP] CHAIN_TYPE=betterstuff N_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db N_FORWARD_DOCUMENTS=100 # How many documents to forward to the LLM, chosen among those retrieved N_GPU_LAYERS=4

Python version

3.11.3

System

Windows 11

CASALIOY version

Main

Information

• [X] The official example scripts
• [ ] My own modified scripts

Related Components

• [ ] Document ingestion
• [ ] GUI
• [X] Prompt answering

Reproduction

/

D:\120hz\CASALIOY>python casalioy/ingest.py found local model dir at models\sentence-transformers\all-MiniLM-L6-v2 found local model file at models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin

Delete current database?(Y/N): y Deleting db... Scanning files Processing ABRACEEL_process_230519.pdf Processing 89 chunks Creating a new collection, size=384 Saving 89 chunks Saved, the collection now holds 89 documents. Processed ABRACEEL_process_230519.pdf 100.0% [=======================================================================================>] 1/ 1 eta [00:00] Done

D:\120hz\CASALIOY>python casalioy/startLLM.py found local model dir at models\sentence-transformers\all-MiniLM-L6-v2 found local model file at models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin llama.cpp: loading model from models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin llama_model_load_internal: format = ggjt v2 (latest) llama_model_load_internal: n_vocab = 32000 llama_model_load_internal: n_ctx = 1024 llama_model_load_internal: n_embd = 4096 llama_model_load_internal: n_mult = 256 llama_model_load_internal: n_head = 32 llama_model_load_internal: n_layer = 32 llama_model_load_internal: n_rot = 128 llama_model_load_internal: ftype = 9 (mostly Q5_1) llama_model_load_internal: n_ff = 11008 llama_model_load_internal: n_parts = 1 llama_model_load_internal: model size = 7B llama_model_load_internal: ggml ctx size = 72.75 KB llama_model_load_internal: mem required = 6612.59 MB (+ 1026.00 MB per state) .................................................................................................... llama_init_from_file: kv self size = 512.00 MB AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ D:\120hz\CASALIOY\casalioy\startLLM.py:135 in │ │ │ │ 132 │ │ 133 │ │ 134 if name == "main": │ │ ❱ 135 │ main() │ │ 136 │ │ │ │ D:\120hz\CASALIOY\casalioy\startLLM.py:123 in main │ │ │ │ 120 # noinspection PyMissingOrEmptyDocstring │ │ 121 def main() -> None: │ │ 122 │ session = PromptSession(auto_suggest=AutoSuggestFromHistory()) │ │ ❱ 123 │ qa_system = QASystem(get_embedding_model()[0], persist_directory, modelpath, model │ │ 124 │ while True: │ │ 125 │ │ query = prompt_HTML(session, "\nEnter a query: ").strip() │ │ 126 │ │ if query == "exit": │ │ │ │ D:\120hz\CASALIOY\casalioy\startLLM.py:57 in init │ │ │ │ 54 │ │ │ case "LlamaCpp": │ │ 55 │ │ │ │ from langchain.llms import LlamaCpp │ │ 56 │ │ │ │ │ │ ❱ 57 │ │ │ │ llm = LlamaCpp( │ │ 58 │ │ │ │ │ model_path=model_path, │ │ 59 │ │ │ │ │ n_ctx=n_ctx, │ │ 60 │ │ │ │ │ temperature=model_temp, │ │ │ │ in pydantic.main.BaseModel.init:341 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ ValidationError: 1 validation error for LlamaCpp n_gpu_layers extra fields not permitted (type=value_error.extra)

Expected behavior

Question Prompt.

[hippalectryon-0]

CASALIOY version Main

Can you confirm that you're on commit f29182b ? It sounds like you're on a previous release that didn't include the n_gpu_layers arg

[neeewwww]

Yep, I'm using the latest files. Just confirmed.

[hippalectryon-0]

Did you also rerun poetry install ? What's the output of poetry show ?

[neeewwww]

Rerun the "Build from Source" procedures solves the problem. Question: Each new version need to re-run the poetry?

[hippalectryon-0]

Each time we update the requirements, you need to run "poetry install" to update them.

We'll add a section to the README.