main: seed = 1687068338
starcoder_model_load: loading model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'
starcoder_model_load: n_vocab = 49280
starcoder_model_load: n_ctx = 2048
starcoder_model_load: n_embd = 2048
starcoder_model_load: n_head = 16
starcoder_model_load: n_layer = 24
starcoder_model_load: ftype = 1003
starcoder_model_load: qntvr = 1
starcoder_model_load: ggml ctx size = 1794.97 MB
starcoder_model_load: memory size = 768.00 MB, n_mem = 49152
starcoder_model_load: unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file
main: failed to load model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'
Notable differences from the sample output:
starcoder_model_load: ftype = 1 in my output vs starcoder_model_load: ftype = 3
(quanitzed models were produced with ./quantize models/bigcode/gpt_bigcode-santacoder-ggml.bin models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin 3; non-quanitzed model fails with a similar error)
starcoder_model_load: qntvr = 1 in my output vs. no info on qntvr in the sample output
Other notes:
this is running on a 2019 Intel MBP, not an M1
conda list is reproduced below in case I'm somehow missing a dependency
I'm getting the following error in the final step of the quickstart:
unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file
Input line:
./main -m models/bigcode/gpt_bigcode-santacoder-ggml.bin -p "def fibonnaci(" --top_k 0 --top_p 0.95 --temp 0.2
Output:
Notable differences from the sample output:
starcoder_model_load: ftype = 1
in my output vsstarcoder_model_load: ftype = 3
(quanitzed models were produced with./quantize models/bigcode/gpt_bigcode-santacoder-ggml.bin models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin 3
; non-quanitzed model fails with a similar error)starcoder_model_load: qntvr = 1
in my output vs. no info onqntvr
in the sample outputOther notes:
conda list
is reproduced below in case I'm somehow missing a dependency