search

harrisonvanderbyl / rwkv-cpp-accelerated

A torchless, c++ rwkv implementation using 8bit quantization, written in cuda/hip/vulkan for maximum compatibility and minimum dependencies
MIT License
306 stars 19 forks source link

Not working on 7B & 14B models | Torch Binding #10

Closed nenkoru closed 1 year ago

nenkoru commented 1 year ago

Testing torch bindings and code doesn't work on large models.

Models are converted using a converter version against current master. The issue is not occuring while using 1b5 and 3b models.

The self.output after running interop.forward method stays the same(nAn). On the other hand, state is being changed. So there is some problem with output setting with CPP code.

Attached a jupyter notebook to reproduce, but with .md extension. So make sure to rename it back to .ipynb. (GH doesn't allow uploading ipynb for some reason) untitled_1.md

nenkoru commented 1 year ago

Fully resolved after ed3ecc300ffdc374ecce443cebbc6b0c5889c941 and #12 PR.