search

Atome-FE / llama-node

Believe in AI democratization. llama for nodejs backed by llama-rs, llama.cpp and rwkv.cpp, work locally on your laptop CPU. support llama/alpaca/gpt4all/vicuna/rwkv model.
https://llama-node.vercel.app/
Apache License 2.0
865 stars 63 forks source link

@llama-node/rwkv-cpp only contains rwkv-cpp.darwin-arm64.node #44

Closed yorkzero831 closed 1 year ago

yorkzero831 commented 1 year ago

https://www.npmjs.com/package/@llama-node/rwkv-cpp?activeTab=code

yorkzero831 commented 1 year ago

@hlhr202
got such error when manually copy rwkv-cpp.win32-x64-msvc.node into dependencies

Process finished with exit code -1073741819 (0xC0000005)
hlhr202 commented 1 year ago

@hlhr202 got such error when manually copy rwkv-cpp.win32-x64-msvc.node into dependencies

Process finished with exit code -1073741819 (0xC0000005)

just give me some time, fixing this.

yorkzero831 commented 1 year ago

@hlhr202 No worry, you are always so cool

hlhr202 commented 1 year ago

@hlhr202 No worry, you are always so cool

https://www.npmjs.com/package/@llama-node/rwkv-cpp?activeTab=code published with v0.0.35. but I have not ever tested on windows. you can have a try

hlhr202 commented 1 year ago

@hlhr202 got such error when manually copy rwkv-cpp.win32-x64-msvc.node into dependencies

Process finished with exit code -1073741819 (0xC0000005)

just give me some time, fixing this.

emmm, so this means you have compiled successfully on windows but cannot run it?

yorkzero831 commented 1 year ago

@hlhr202 got such error when manually copy rwkv-cpp.win32-x64-msvc.node into dependencies

Process finished with exit code -1073741819 (0xC0000005)

just give me some time, fixing this.

emmm, so this means you have compiled successfully on windows but cannot run it?

Yes, I thought it was caused by binary mismatch or something, but after trying on 0.0.35, the problem still existing, I think there should be other problems on windows.

hlhr202 commented 1 year ago

ok got this, will figure out what's wrong with win build. I can load the model and tokenizer, but failed while doing inference.

yorkzero831 commented 1 year ago

ok got this, will figure out what's wrong with win build. I can load the model and tokenizer, but failed while doing inference.

seems like the problem happens because pointer passing between rust and rwkv

yorkzero831 commented 1 year ago

@hlhr202 I have done some dirty changes, it works on my windows pc, and I found the end_token in rwkv-raven v11 looks like 187 so I set it as default https://github.com/yorkzero831/llama-node/commit/72413be2964ed6b24fb5e425893d845d0aeb5ad0

yorkzero831 commented 1 year ago

the log will look like

[2023-05-05T17:28:24Z INFO  rwkv_node_cpp::context] AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
1
token: 1533
 world

token: 15
.

token: 187

test test1 ... ok
hlhr202 commented 1 year ago

@yorkzero831 awesome. would you mind create pr for this? we warmly welcome contribution.

yorkzero831 commented 1 year ago

@hlhr202 sure working on it

yorkzero831 commented 1 year ago

@hlhr202 could you please have a look at this https://github.com/Atome-FE/llama-node/pull/46