Cannot compile on Raspberry Pi

[nmeln]

README mentions Raspberry Pi: https://github.com/mit-han-lab/TinyChatEngine#step-by-step-to-deploy-llama2-7b-chat-with-tinychatengine

make chat -j fails with this error:

/usr/bin/ld: build/transformer/src/ops/linear.o: in function `Linear_FP::forward(Matrix3D<float> const&, Matrix3D<float>&)':
linear.cc:(.text+0xf8): undefined reference to `matmul::MatmulOperator::mat_mul_accelerator_transposed_fastover_column_bias(matmul_params const*)'
/usr/bin/ld: build/transformer/src/ops/linear.o: in function `Linear_FP_int4::forward(Matrix3D<float> const&, Matrix3D<float>&)':
linear.cc:(.text+0x564): undefined reference to `matmul::MatmulOperator::mat_mul_accelerator_int4_fast_no_offset(matmul_params const*)'
collect2: error: ld returned 1 exit status
make: *** [Makefile:195: chat] Error 1

OS: Raspbian 10

[RaymondWang0]

Hi @nmeln, thank you for your interest in our work. What version of Raspberry Pi are you using? We tested it on our Raspberry Pi 4 Model B with 8GB RAM and it works great.

[nmeln]

Here is part of the output of neofetch:

OS: Raspbian GNU/Linux 10 (buster) armv7l

Host: Raspberry Pi 4 Model B Rev 1.1

Kernel: 5.10.103-v7l+

CPU: BCM2711 (4) @ 1.500GHZ

Memory: 529MiB / 3597MiB

So my board is Pi4B (4GB)

[RaymondWang0]

Thanks for sharing the details. Here's the output of neofetch on our board for your reference:

OS: Debian GNU/Linux 10 (buster) aarch64
Host: Raspberry Pi 4 Model B Rev 1.4
Kernel: 5.10.103-v8+

Regarding the undefined reference errors you pointed out, they arose because our initial support was limited to aarch64 for Raspberry Pi. We've updated Makefile to include support for armv6, armv7, and armv8. But we don’t have these versions on hand and thus haven't tested them ourselves. We'd appreciate it if you could test it and share any feedback or issues you come across, and feel free to help us fix it.

On the other hand, please note that we recommend using the board with 8GB RAM. You may encounter a memory shortage issue when using 4GB RAM. We've updated README to make this clearer for users.

[RaymondWang0]

I'll close this issue due to inactivity. Please feel free to reopen it anytime.