Open prof-schacht opened 1 year ago
That is not worked like that you need to use interface to convert your promt from string to int.
Try doing this instead.
Move your codegen model to lookup path: mv converters/codegen-16b/codgen16b-q4.bin ~/.cformers/models/Salesforce/codegen-16B-mono/int4_fixed_zero
, you may need to make the directories before moving though.
Run following code in python:
from interface import AutoInference as AI
ai = AI('Salesforce/codegen-16B-mono')
x = ai.generate('def palindrom(word):', num_tokens_to_generate=500)
print(x['token_str'])
Hi,
I converted the codgen-16b model by using the following code:
python3 convert_gptj_to_ggml.py sourceforge/codgen-16b ./codgen-16b 0
'./quantize_gptj ./codgen-16b/cogen-16b.bin 1'Inference I used the following command:
./main gptj -m converters/codegen-16b/codgen16b-q4.bin --prompt "def palindrom(word):" -t 8
But I got the following error:
gptj_model_load: loading model from 'converters/codegen-16b/codgen16b-q4.bin' - please wait ... gptj_model_load: valid model file 'converters/codegen-16b/codgen16b-q4.bin' (good magic) gptj_model_load: n_vocab = 51200 gptj_model_load: n_ctx = 512 gptj_model_load: n_embd = 6144 gptj_model_load: n_head = 24 gptj_model_load: n_layer = 34 gptj_model_load: n_rot = 64 gptj_model_load: f16 = 2 gptj_model_load: ggml ctx size = 10376.90 MB gptj_model_load: memory_size = 816.00 MB, n_mem = 17408 gptj_model_load: ........................................... done gptj_model_load: model size = 9560.82 MB / num tensors = 345 libc++abi: terminating with uncaught exception of type std::invalid_argument: stoi: no conversion zsh: abort ./main gptj -m converters/codegen-16b/codgen16b-q4.bin --prompt -t 8
Any Ideas?