search

RWKV / rwkv.cpp

INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
MIT License
1.37k stars 90 forks source link

Add capability for extra binaries to be built with rwkv.cpp #87

Closed LoganDark closed 1 year ago

LoganDark commented 1 year ago

This also adds a quantizer that works without python. in the future, we might be able to convert from pytorch as well, without python.

it is short sighted to say that rwkv.cpp will always depend on python for support infrastructure

LoganDark commented 1 year ago

"Examples" is just a blanket term for "executables that use the library, that are provided with the distribution".

In reality examples can be used more like utilities here, where they offer access to some of the library's functionality without needing Python for FFI.

Here, we can get cpu info (good for debugging builds) and quantize model files without Python.

In the future, we may be able to load directly from PyTorch model files, or tokenize/sample without Python. Both of those will want examples or utilities.

LoganDark commented 1 year ago

wait why is the whole fucking world blowing up this built fine on my local and wsl I'm going to die

saharNooby commented 1 year ago

I welcome making rwkv.cpp Python-independent. It will, for example, make the library more easily embeddable into game engines.

A couple of formatting changes and good to go :)

LoganDark commented 1 year ago

sorry for making this so stressful

saharNooby commented 1 year ago

\nn\nAvailable formats

Wait, but double-n is still here. Maybe it is intended?

LoganDark commented 1 year ago

\nn\nAvailable formats

Wait, but double-n is still here. Maybe it is intended?

yes it is here, if you want to go with windows format, image

LoganDark commented 1 year ago

oh

LoganDark commented 1 year ago

sorry I'm dumb probably on drugs