Is it possible to add support for other models?

efeslab / Atom

[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

277 stars 24 forks source link

Is it possible to add support for other models? #21

Closed wlll123456 closed 3 months ago

wlll123456 commented 3 months ago

Hi, this is great work. I found under /Atom/model/main.py file that it seems to only support llama, opt, mixtral models. If I need to add support for Qwen model, which files should I change?

happierpig commented 3 months ago

Hi @wlll123456 ,

Thanks for your interest! Sure it is possible to add support for models with human efforts. Basically, you need to add wrapper functions corresponding to the model architecture. You can follow this commit(7dafad02d6837ca1436d1c2bbec9c9edfd475e0e) where we add support for Mixtral models.