search

0cc4m / KoboldAI

GNU Affero General Public License v3.0
150 stars 30 forks source link

Request for T5 gptq model support. #53

Open sigmareaver opened 1 year ago

sigmareaver commented 1 year ago

I attempted to load up flan-ul2 4-bit 128g gptq, but it looks like T5ForConditionalGeneration isn't supported, or perhaps Encoder/Decoder type LLMs in general. In particular, https://github.com/qwopqwop200/transformers-t5 would also likely be needed to provide support for quantized T5.

sigmareaver commented 1 year ago

If possible, I could do a pull request rather than burden you with a feature request. If you could let me know what files/functions I should look at to add support for a new model type, since I'm not familiar with KoboldAI's codebase, that should be enough to get me started.