search

Preemo-Inc / text-generation-inference

Apache License 2.0
201 stars 22 forks source link

Mistral Models ct2 support #16

Open manishiitg opened 1 year ago

manishiitg commented 1 year ago

Feature request

ctranslate2 doesn't support mistral model conversion out of the box https://github.com/OpenNMT/CTranslate2/issues/1501

currently when using ct2 it automatically converts the model on the fly.

can we allow support to convert model to ct2 offline? that way i can convert mistral model offline and then use it with TGI.

Motivation

mistral support

Your contribution

-

michaelfeil commented 11 months ago

This issue should be solved upstream in ctranslate2. Once ctranslate2 supports mistral, tgi can bump the version.

Allowing a pre-converted model is not really on the roadmap, tgi really wants to download .safetensors in pytorch format. in that way, ct2 quantization, just quick-fixes this issue.