search

LLukas22 / llm-rs-python

Unofficial python bindings for the rust llm library. 🐍❤️🦀
MIT License
71 stars 4 forks source link

GPU support - Feature Request #19

Closed sidharthiimc closed 1 year ago

sidharthiimc commented 1 year ago

I am able to use quantized llama model over gpu using Llama cpp and langchain. But for MPT, I am able to do inference with only llm-rs. And there is no GPU support for this.

LLukas22 commented 1 year ago

Gpu support is planned, but i have to wait until we implement it in rustformers/llm via https://github.com/rustformers/llm/pull/282.