search

mit-han-lab / TinyChatEngine

TinyChatEngine: On-Device LLM Inference Library
https://mit-han-lab.github.io/TinyChatEngine/
MIT License
756 stars 73 forks source link

StarCoder model and AWQ file formats #80

Closed 167rgc911 closed 1 year ago

167rgc911 commented 1 year ago

Thanks for sharing all of your work which allows people like me (no ML/AI training) to play with it.

Recently you guys merged #78, is there a way to get the StarCoder model via the scripts (llm/tools/download_model.py)?

Are the files in the MIT-HAN-Lab HF (https://huggingface.co/datasets/mit-han-lab/awq-model-zoo/tree/main) page compatible?

Can TinyChatEngine use the file outputs of AutoAWQ?

167rgc911 commented 1 year ago

It seems I need to get the HF model and just use the provided Python scripts.