Place model file, vocab.json, added_tokens.json, config.json from togethercomputer/GPT-JT-6B-v1 in a directory <modelDirectory>.
Convert model from torch to a serialized format: Run cformers/cformers/cpp/converters/convert_gptj_to_ggml.py with python3 convert_gptj_to_ggml.py <modelDirectory> 1 --- for bloom/GPTNeoX before <modelDirectory> you must add Huggingface Model Card (for example EleutherAI/pythia-6.9b-deduped)
Convert serialized model to quantized format: cformers/cformers/cpp/quantize_gptj.cpp as cd cformers/cformers/cpp && make && ./quantize_gptj <modelDirectory>/ggml-gptj-6b-model.bin <modelDirectory>/int4_fixed_zero.bin 2
Following steps would be required:
Place model file, vocab.json, added_tokens.json, config.json from togethercomputer/GPT-JT-6B-v1 in a directory
<modelDirectory>.Convert model from torch to a serialized format: Run cformers/cformers/cpp/converters/convert_gptj_to_ggml.py with
python3 convert_gptj_to_ggml.py <modelDirectory> 1--- for bloom/GPTNeoX before<modelDirectory>you must add Huggingface Model Card (for exampleEleutherAI/pythia-6.9b-deduped)Convert serialized model to quantized format: cformers/cformers/cpp/quantize_gptj.cpp as
cd cformers/cformers/cpp && make && ./quantize_gptj <modelDirectory>/ggml-gptj-6b-model.bin <modelDirectory>/int4_fixed_zero.bin 2Update Readme and cformers/cformers/interface.py with links and mention GPT-JT is supported
Test it, share how well it works and shoot a PR.