Closed sidharthiimc closed 1 year ago
I am able to use quantized llama model over gpu using Llama cpp and langchain. But for MPT, I am able to do inference with only llm-rs. And there is no GPU support for this.
Gpu support is planned, but i have to wait until we implement it in rustformers/llm via https://github.com/rustformers/llm/pull/282.
I am able to use quantized llama model over gpu using Llama cpp and langchain. But for MPT, I am able to do inference with only llm-rs. And there is no GPU support for this.