kuvaus
commented
1 year ago Thanks for this!
I had a silly bug in json loading where "load_template" "save_log" and "load_log" were all pointing to params.load_template because I had forgotten to change the names after copy/pasting. Oops. With command line loading they were working correctly.
Fixed that in v.0.2.4. I hope this removes the issue.
Good thing you found this by testing. :)
I put some options into a json file, and run the chat program by "chat.exe -j chat.json", then an error occurred:
LlamaGPTJ-chat (v. 0.2.3) Your computer supports AVX2 LlamaGPTJ-chat: parsing options from json: chat.json LlamaGPTJ-chat: loading models\ggml-mpt-7b-chat.bin mpt_model_load: loading model from 'models\ggml-mpt-7b-chat.bin' - please wait ... mpt_model_load: n_vocab = 50432 mpt_model_load: n_ctx = 2048 mpt_model_load: n_embd = 4096 mpt_model_load: n_head = 32 mpt_model_load: n_layer = 32 mpt_model_load: alibi_bias_max = 8.000000 mpt_model_load: clip_qkv = 0.000000 mpt_model_load: ftype = 2 mpt_model_load: ggml ctx size = 5653.09 MB mpt_model_load: kv self size = 1024.00 MB mpt_model_load: .............................. done mpt_model_load: model size = 4629.02 MB / num tensors = 194 LlamaGPTJ-chat: done loading!
terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc
The content of chat.json is:
{ "top_p": 0.9, "top_k": 40, "temp": 0.9, "threads": 4, "save_log": "chat.log", "load_template": "prompt_template.txt", "model": "models\ggml-mpt-7b-chat.bin" }
While, I use the same options by means of a Windows batch file, it runs OK.
The batch file contains:
@echo off chat.exe --top_p 0.9 --top_k 40 --temp 0.9 --threads 4 --save_log chat.log --load_template prompt_template.txt --model models\ggml-mpt-7b-chat.bin