search

kvcache-ai / ktransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations
Apache License 2.0
746 stars 39 forks source link

GPU support without fp16. Multi gpu support #15

Closed AlexBefest closed 4 months ago

AlexBefest commented 4 months ago

Hello, I am very excited about this project. I wanted to ask if ktransformers supports or will support video cards that do not work with fp16? Is there multi gpu support like llama cpp?

james0zan commented 4 months ago

We are working on multi-gpu suppport. But the support for non-fp16 card may not come soon, because we do not have such devices for experiemnting and testing.