search

ZeldaHuang / rwkv-cpp-server

Easily deploy your rwkv model
18 stars 0 forks source link

Add linux, 8bit and gpu support #1

Open harrisonvanderbyl opened 1 year ago

harrisonvanderbyl commented 1 year ago

https://github.com/harrisonvanderbyl/rwkv-cpp-cuda

Includes c++ sampler and tokenizer, as well as 8bit cuda support, with no libtorch dependency

ZeldaHuang commented 1 year ago

https://github.com/harrisonvanderbyl/rwkv-cpp-cuda

Includes c++ sampler and tokenizer, as well as 8bit cuda support, with no libtorch dependency

I'm considering supporting gpu inference, could this project support packing cuda dll dependency to release file? so that user can run the app without install cuda toolkit.

harrisonvanderbyl commented 1 year ago

The project links cudart_static, it should run without cuda being installed

ZeldaHuang commented 1 year ago

got it, I'll run the benchmark between libtorch implementation and your cuda implementation.