Open abcnow opened 1 year ago
I would recommend sticking with llamacpp. The ReadMe states a model that's over six months in age and was just a test model. The quality of models improved incredible since this repo was created. I have to check compatibility of the LlamaCpp version to run Llama2 models
Nonetheless the last time I tested it, it was running with ggml-gpt4all-j-v1.3-groovy.bin
.env
Generic
TEXT_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2 TEXT_EMBEDDINGS_MODEL_TYPE=HF # LlamaCpp or HF USE_MLOCK=false
Ingestion
PERSIST_DIRECTORY=db DOCUMENTS_DIRECTORY=source_documents INGEST_CHUNK_SIZE=500 INGEST_CHUNK_OVERLAP=50 INGEST_N_THREADS=4
Generation
MODEL_TYPE=GPT4All # GPT4All or LlamaCpp MODEL_PATH=thebloke/ggml-gpt4all-j-v1.3-groovy.bin MODEL_TEMP=0.8 MODEL_N_CTX=1024 # Max total size of prompt+answer MODEL_MAX_TOKENS=256 # Max size of answer MODEL_STOP=[STOP] CHAIN_TYPE=betterstuff N_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db N_FORWARD_DOCUMENTS=100 # How many documents to forward to the LLM, chosen among those retrieved N_GPU_LAYERS=4
Python version
3.11.3
System
OS Monterey version 12.6.8
CASALIOY version
latest
Information
Related Components
Reproduction
I switched to gpt4all model since the vicuna's answer were below average (with the docs you have there by default).. and it is giving me this error: declare -x PWD="/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main" declare -x SDKMAN_CANDIDATES_API="https://api.sdkman.io/2" declare -x SDKMAN_CANDIDATES_DIR="/Users/ac/.sdkman/candidates" declare -x SDKMAN_DIR="/Users/ac/.sdkman" declare -x SDKMAN_PLATFORM="Darwin" declare -x SDKMAN_VERSION="5.7.2+323" declare -x SHELL="/bin/bash" declare -x SHLVL="2" declare -x SSH_AUTH_SOCK="/private/tmp/com.apple.launchd.OyMPfNIIOU/Listeners" declare -x TERM="xterm-256color" declare -x TERM_PROGRAM="vscode" declare -x TERM_PROGRAM_VERSION="1.81.1" declare -x TMPDIR="/var/folders/bt/wyx4_3gx33q2cj3b_frrbz780000gn/T/" declare -x USER="ac" declare -x VSCODE_GIT_ASKPASS_EXTRA_ARGS="--ms-enable-electron-run-as-node" declare -x VSCODE_GIT_ASKPASS_MAIN="/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/git/dist/askpass-main.js" declare -x VSCODE_GIT_ASKPASS_NODE="/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)" declare -x VSCODE_GIT_IPC_HANDLE="/var/folders/bt/wyx4_3gx33q2cj3b_frrbz780000gn/T/vscode-git-8d97e97cde.sock" declare -x VSCODE_INJECTION="1" declare -x VSCODE_NONCE="890b781c-7725-4872-be2f-ec2096aa5f39" declare -x VSCODE_SHELL_LOGIN="1" declare -x XPC_FLAGS="0x0" declare -x XPC_SERVICE_NAME="application.com.microsoft.VSCode.13059730707.13059730714" declare -x _CE_CONDA="" declare -x _CE_M="" declare -x CFBundleIdentifier="com.microsoft.VSCode" declare -x CF_USER_TEXT_ENCODING="0x1F5:0x0:0x0"
The default interactive shell is now zsh. To update your account to use zsh, please run
chsh -s /bin/zsh
. For more details, please visit https://support.apple.com/kb/HT208050. (base) ac at acs-iMac in ~/Documents/software/ai/casalioy/test0/CASALIOY-main $ /Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/.venv/bin/python -u "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py" found local model dir at models/sentence-transformers/all-MiniLM-L6-v2 found local model file at models/thebloke/ggml-gpt4all-j-v1.3-groovy.bin gptj_model_load: loading model from 'models/thebloke/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ... gptj_model_load: n_vocab = 50400 gptj_model_load: n_ctx = 2048 gptj_model_load: n_embd = 4096 gptj_model_load: n_head = 16 gptj_model_load: n_layer = 28 gptj_model_load: n_rot = 64 gptj_model_load: f16 = 2 gptj_model_load: ggml ctx size = 4505.45 MB gptj_model_load: memory_size = 896.00 MB, n_mem = 57344 gptj_model_load: ................................... done gptj_model_load: model size = 3609.38 MB / num tensors = 285Enter a query: source /Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/.venv/bi n/activate Traceback (most recent call last): File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 135, in
main()
File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 131, in main
qa_system.prompt_once(query)
File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 105, in prompt_once
res = self.qa(query)
^^^^^^^^^^^^^^
File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/CustomChains.py", line 80, in call
Expected behavior
program it is stopping right there...it should answer the question.. it is just stopping the process...