[Feature]: Support inference with the Q4_k_m quantization gguf models - Githubissues

SkywardAI / kirin

APIs aggregator for inference, fine-tuning and build models.

https://skywardai.github.io/skywardai.io/

Apache License 2.0

5 stars 7 forks source link

[Feature]: Support inference with the Q4_k_m quantization gguf models #162

Closed Aisuko closed 2 weeks ago

Aisuko commented 3 weeks ago

Contact Details(optional)

No response

What feature are you requesting?

More detail see discussion:

And the issues related to ML lib