search

RWKV / rwkv.cpp

INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
MIT License
1.37k stars 90 forks source link

Fix "'NoneType' object has no attribute 'cast'" error when model is freed #117

Closed mczk77 closed 1 year ago

mczk77 commented 1 year ago

This patch fixes the error by storing the null pointer in the constructor, which is used to set ctx.ptr when the model is freed.

saharNooby commented 1 year ago

I don't really understand how this works. Is it possible to provide a minimal snippet that reproduces the exception and is fixed by this PR?

mczk77 commented 1 year ago

This error happens on all of the examples in my environment. The simplest way I'm able to reproduce it is by modifying the generate_completions.py script to have it read one token and generate one token. The error it produces is:

Loading 20B tokenizer
System info: AVX=1 AVX2=1 AVX512=0 FMA=1 NEON=0 ARM_FMA=0 F16C=1 FP16_VA=0 WASM_SIMD=0 BLAS=0 SSE3=1 VSX=0
Loading RWKV model
1 tokens in prompt

--- Generation 0 ---

Hello[,]

Took 0.151 sec, 151 ms per token
Exception ignored in: <function RWKVModel.__del__ at 0x7fa783d80ee0>
Traceback (most recent call last):
  File "./rwkv.cpp/rwkv/rwkv_cpp_model.py", line 198, in __del__
  File "./rwkv.cpp/rwkv/rwkv_cpp_model.py", line 193, in free
  File "./rwkv.cpp/rwkv/rwkv_cpp_shared_library.py", line 235, in rwkv_free
AttributeError: 'NoneType' object has no attribute 'cast'

I'm running it on Python 3.10.6.