Closed neeewwww closed 1 year ago
CASALIOY version Main
Can you confirm that you're on commit f29182b ? It sounds like you're on a previous release that didn't include the n_gpu_layers
arg
Yep, I'm using the latest files. Just confirmed.
Did you also rerun poetry install
? What's the output of poetry show
?
Rerun the "Build from Source" procedures solves the problem. Question: Each new version need to re-run the poetry?
Each time we update the requirements, you need to run "poetry install" to update them.
We'll add a section to the README.
.env
Generic
TEXT_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2 TEXT_EMBEDDINGS_MODEL_TYPE=HF # LlamaCpp or HF USE_MLOCK=true
Ingestion
PERSIST_DIRECTORY=db DOCUMENTS_DIRECTORY=source_documents INGEST_CHUNK_SIZE=500 INGEST_CHUNK_OVERLAP=50
Generation
MODEL_TYPE=LlamaCpp # GPT4All or LlamaCpp MODEL_PATH=eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin MODEL_TEMP=0.8 MODEL_N_CTX=1024 # Max total size of prompt+answer MODEL_MAX_TOKENS=256 # Max size of answer MODEL_STOP=[STOP] CHAIN_TYPE=betterstuff N_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db N_FORWARD_DOCUMENTS=100 # How many documents to forward to the LLM, chosen among those retrieved N_GPU_LAYERS=4
Python version
3.11.3
System
Windows 11
CASALIOY version
Main
Information
Related Components
Reproduction
D:\120hz\CASALIOY>python casalioy/ingest.py found local model dir at models\sentence-transformers\all-MiniLM-L6-v2 found local model file at models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin
Delete current database?(Y/N): y Deleting db... Scanning files Processing ABRACEEL_process_230519.pdf Processing 89 chunks Creating a new collection, size=384 Saving 89 chunks Saved, the collection now holds 89 documents. Processed ABRACEEL_process_230519.pdf 100.0% [=======================================================================================>] 1/ 1 eta [00:00] Done
D:\120hz\CASALIOY>python casalioy/startLLM.py found local model dir at models\sentence-transformers\all-MiniLM-L6-v2 found local model file at models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin llama.cpp: loading model from models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin llama_model_load_internal: format = ggjt v2 (latest) llama_model_load_internal: n_vocab = 32000 llama_model_load_internal: n_ctx = 1024 llama_model_load_internal: n_embd = 4096 llama_model_load_internal: n_mult = 256 llama_model_load_internal: n_head = 32 llama_model_load_internal: n_layer = 32 llama_model_load_internal: n_rot = 128 llama_model_load_internal: ftype = 9 (mostly Q5_1) llama_model_load_internal: n_ff = 11008 llama_model_load_internal: n_parts = 1 llama_model_load_internal: model size = 7B llama_model_load_internal: ggml ctx size = 72.75 KB llama_model_load_internal: mem required = 6612.59 MB (+ 1026.00 MB per state) .................................................................................................... llama_init_from_file: kv self size = 512.00 MB AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ D:\120hz\CASALIOY\casalioy\startLLM.py:135 in │
│ │
│ 132 │
│ 133 │
│ 134 if name == "main": │
│ ❱ 135 │ main() │
│ 136 │
│ │
│ D:\120hz\CASALIOY\casalioy\startLLM.py:123 in main │
│ │
│ 120 # noinspection PyMissingOrEmptyDocstring │
│ 121 def main() -> None: │
│ 122 │ session = PromptSession(auto_suggest=AutoSuggestFromHistory()) │
│ ❱ 123 │ qa_system = QASystem(get_embedding_model()[0], persist_directory, modelpath, model │
│ 124 │ while True: │
│ 125 │ │ query = prompt_HTML(session, "\nEnter a query: ").strip() │
│ 126 │ │ if query == "exit": │
│ │
│ D:\120hz\CASALIOY\casalioy\startLLM.py:57 in init │
│ │
│ 54 │ │ │ case "LlamaCpp": │
│ 55 │ │ │ │ from langchain.llms import LlamaCpp │
│ 56 │ │ │ │ │
│ ❱ 57 │ │ │ │ llm = LlamaCpp( │
│ 58 │ │ │ │ │ model_path=model_path, │
│ 59 │ │ │ │ │ n_ctx=n_ctx, │
│ 60 │ │ │ │ │ temperature=model_temp, │
│ │
│ in pydantic.main.BaseModel.init:341 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: 1 validation error for LlamaCpp
n_gpu_layers
extra fields not permitted (type=value_error.extra)
Expected behavior
Question Prompt.