Starcoder conversion and quantization instructions

[aseok]

Hi. Pls provide conversion and quantization instructions of the main starcoder model files.

[NouamaneTazi]

You can find the instructions here: https://github.com/bigcode-project/starcoder.cpp#quick-start

[aseok]

I have downloaded model files separately and skip downloading them in convert-hf-to-ggml.py. my problem is in quantization and probably running inference, how to pass the model files in quantization command? Should I rename them?

[asingleoat]

how to pass the model files in quantization command?

for the sharded model conversion don't pass the filenames, pass the directory: $ python convert-hf-to-ggml.py ./starcoder

then quantization is as in the README example: $ ./quantize starcoder-ggml.bin starcoder-ggml-q4_1.bin 3

[NouamaneTazi]

Does that fix your issue @aseok?