V2 update plan - Githubissues

PicoCreator commented 1 year ago

Latest version of https://github.com/saharNooby/rwkv.cpp has new quantization format (breaking change?) and GPU offload (!!!) Since this might be potentially breaking changes, its gonna be a v2 update.

update to the newer version, which has breaking change on the model (might be backwards compat)
(to confirm if not backwards compat) create a new set of ".bin" files for the new version
make changes to the API, to add support for GPU offload (its a param now in the new version, on how many layers you want to offload to the GPU)
for input inference, update to the new batch mode API (10x faster)
(stretch goal) change to async API
support for world model / world tokenizer (we can detect this using the token count)

PicoCreator commented 1 year ago

<= 67.2 ms on 7B with partial GPU offload (3060 Ti 8G) is a huge win win

cahya-wirawan commented 1 year ago

Hi, I converted a pth file to bin format, unfortunately it crashed with following error message:

% rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin 
--------------------------------------
Starting RWKV chat mode
--------------------------------------
Loading model from ./rwkv-7b-369-Q5_1.bin ...
Unsupported file version 101
/Users/eugene/Desktop/RWKV/rwkv.cpp/rwkv.cpp:211: version == RWKV_FILE_VERSION
zsh: segmentation fault  rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin

Is it the compatibility issue mentioned here? Thanks (Looking forward to the world version :-) )

PicoCreator commented 1 year ago

Yup, the new rwkv.cpp - is now merged into v2 (publishing now)

PicoCreator commented 1 year ago

Merged in : 74655dee0d3824162d7926882ae95d1371a11fa2

This resolves all issues, EXCEPT support for world model tokenizer (needs a new JS tokenizer)

cgisky1980 commented 1 year ago

Yup, the new rwkv.cpp - is now merged into v2 (publishing now)

update the docs please

cgisky1980 commented 1 year ago

Merged in : 74655de

This resolves all issues, EXCEPT support for world model tokenizer (needs a new JS tokenizer)

waiting for it ~~

cahya-wirawan commented 1 year ago

Hi, I reinstalled the package, but when I run it, I get only following result:

% rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin
--------------------------------------
Starting RWKV chat mode
--------------------------------------
Loading model with {"path":"./rwkv-7b-369-Q5_1.bin","threads":4,"gpuOffload":0} ...
The following is a conversation between the User and the Bot ...
--------------------------------------
? User:  Hi how are you
Bot: <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

Did I convert the model wrong?

cgisky1980 commented 1 year ago

Hi, I reinstalled the package, but when I run it, I get only following result:

% rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin
--------------------------------------
Starting RWKV chat mode
--------------------------------------
Loading model with {"path":"./rwkv-7b-369-Q5_1.bin","threads":4,"gpuOffload":0} ...
The following is a conversation between the User and the Bot ...
--------------------------------------
? User:  Hi how are you
Bot: <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

Did I convert the model wrong?

world models not support yet

cahya-wirawan commented 1 year ago

Hi, I reinstalled the package, but when I run it, I get only following result:

% rwkv-cpp-node --modelPath ./rwkv-7b-369-Q5_1.bin
--------------------------------------
Starting RWKV chat mode
--------------------------------------
Loading model with {"path":"./rwkv-7b-369-Q5_1.bin","threads":4,"gpuOffload":0} ...
The following is a conversation between the User and the Bot ...
--------------------------------------
? User:  Hi how are you
Bot: <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

Did I convert the model wrong?

world models not support yet

It is just a normal finetuned raven model

RWKV / RWKV-cpp-node

V2 update plan #2