search

su77ungr / CASALIOY

♾️ toolkit for air-gapped LLMs on consumer-grade hardware
Apache License 2.0
230 stars 31 forks source link

running gpt4 all models #114

Open abcnow opened 1 year ago

abcnow commented 1 year ago

.env

Generic

TEXT_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2 TEXT_EMBEDDINGS_MODEL_TYPE=HF # LlamaCpp or HF USE_MLOCK=false

Ingestion

PERSIST_DIRECTORY=db DOCUMENTS_DIRECTORY=source_documents INGEST_CHUNK_SIZE=500 INGEST_CHUNK_OVERLAP=50 INGEST_N_THREADS=4

Generation

MODEL_TYPE=GPT4All # GPT4All or LlamaCpp MODEL_PATH=thebloke/ggml-gpt4all-j-v1.3-groovy.bin MODEL_TEMP=0.8 MODEL_N_CTX=1024 # Max total size of prompt+answer MODEL_MAX_TOKENS=256 # Max size of answer MODEL_STOP=[STOP] CHAIN_TYPE=betterstuff N_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db N_FORWARD_DOCUMENTS=100 # How many documents to forward to the LLM, chosen among those retrieved N_GPU_LAYERS=4

Python version

3.11.3

System

OS Monterey version 12.6.8

CASALIOY version

latest

Information

Related Components

Reproduction

I switched to gpt4all model since the vicuna's answer were below average (with the docs you have there by default).. and it is giving me this error: declare -x PWD="/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main" declare -x SDKMAN_CANDIDATES_API="https://api.sdkman.io/2" declare -x SDKMAN_CANDIDATES_DIR="/Users/ac/.sdkman/candidates" declare -x SDKMAN_DIR="/Users/ac/.sdkman" declare -x SDKMAN_PLATFORM="Darwin" declare -x SDKMAN_VERSION="5.7.2+323" declare -x SHELL="/bin/bash" declare -x SHLVL="2" declare -x SSH_AUTH_SOCK="/private/tmp/com.apple.launchd.OyMPfNIIOU/Listeners" declare -x TERM="xterm-256color" declare -x TERM_PROGRAM="vscode" declare -x TERM_PROGRAM_VERSION="1.81.1" declare -x TMPDIR="/var/folders/bt/wyx4_3gx33q2cj3b_frrbz780000gn/T/" declare -x USER="ac" declare -x VSCODE_GIT_ASKPASS_EXTRA_ARGS="--ms-enable-electron-run-as-node" declare -x VSCODE_GIT_ASKPASS_MAIN="/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/git/dist/askpass-main.js" declare -x VSCODE_GIT_ASKPASS_NODE="/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)" declare -x VSCODE_GIT_IPC_HANDLE="/var/folders/bt/wyx4_3gx33q2cj3b_frrbz780000gn/T/vscode-git-8d97e97cde.sock" declare -x VSCODE_INJECTION="1" declare -x VSCODE_NONCE="890b781c-7725-4872-be2f-ec2096aa5f39" declare -x VSCODE_SHELL_LOGIN="1" declare -x XPC_FLAGS="0x0" declare -x XPC_SERVICE_NAME="application.com.microsoft.VSCode.13059730707.13059730714" declare -x _CE_CONDA="" declare -x _CE_M="" declare -x CFBundleIdentifier="com.microsoft.VSCode" declare -x CF_USER_TEXT_ENCODING="0x1F5:0x0:0x0"

The default interactive shell is now zsh. To update your account to use zsh, please run chsh -s /bin/zsh. For more details, please visit https://support.apple.com/kb/HT208050. (base) ac at acs-iMac in ~/Documents/software/ai/casalioy/test0/CASALIOY-main $ /Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/.venv/bin/python -u "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py" found local model dir at models/sentence-transformers/all-MiniLM-L6-v2 found local model file at models/thebloke/ggml-gpt4all-j-v1.3-groovy.bin gptj_model_load: loading model from 'models/thebloke/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ... gptj_model_load: n_vocab = 50400 gptj_model_load: n_ctx = 2048 gptj_model_load: n_embd = 4096 gptj_model_load: n_head = 16 gptj_model_load: n_layer = 28 gptj_model_load: n_rot = 64 gptj_model_load: f16 = 2 gptj_model_load: ggml ctx size = 4505.45 MB gptj_model_load: memory_size = 896.00 MB, n_mem = 57344 gptj_model_load: ................................... done gptj_model_load: model size = 3609.38 MB / num tensors = 285

Enter a query: source /Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/.venv/bi n/activate Traceback (most recent call last): File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 135, in main() File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 131, in main qa_system.prompt_once(query) File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 105, in prompt_once res = self.qa(query) ^^^^^^^^^^^^^^ File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/CustomChains.py", line 80, in call

model_n_ctx - self.llm.dict()["max_tokens"]


KeyError: 'max_tokens'
(base) 
---
I am not sure if you have a particular model in mind that could work for gpt4all implementation.. thanks in advance!

Expected behavior

program it is stopping right there...it should answer the question.. it is just stopping the process...

su77ungr commented 1 year ago

I would recommend sticking with llamacpp. The ReadMe states a model that's over six months in age and was just a test model. The quality of models improved incredible since this repo was created. I have to check compatibility of the LlamaCpp version to run Llama2 models

Nonetheless the last time I tested it, it was running with ggml-gpt4all-j-v1.3-groovy.bin