RWKV example: likely transposition error for accelerate or metal

[TomArbuckle]

Clone the repo and try the RWKV example using different feature flags on an Apple M1 Max running the current release of macOS. Compare results of different feature flags.

1. Try without features -> OK

cargo run --example rwkv --release -- --prompt "An empty prompt" Finished release [optimized] target(s) in 0.43s Running target/release/examples/rwkv --prompt 'An empty prompt' avx: false, neon: true, simd128: false, f16c: false temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64 retrieved the files in 1.575708ms Running on CPU, to run on GPU(metal), build this example with --features metal loaded the model in 9.372851s An empty prompt box is displayed. The user enters the name of the file to be created. ^C

2. Try with --features accelerate -> Problem

cargo run --example rwkv --release --features accelerate -- --prompt "An empty prompt" Finished release [optimized] target(s) in 0.46s Running target/release/examples/rwkv --prompt 'An empty prompt' avx: false, neon: true, simd128: false, f16c: false temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64 retrieved the files in 639.542µs Running on CPU, to run on GPU(metal), build this example with --features metal loaded the model in 1.171418791s Error: MatMulUnexpectedStriding { lhs_l: Layout { shape: [1, 32, 64, 1], stride: [2048, 64, 1, 2048], start_offset: 0 }, rhs_l: Layout { shape: [1, 32, 1, 64], stride: [2048, 64, 2048, 1], start_offset: 0 }, bmnk: (32, 64, 64, 1), msg: "non-contiguous rhs" }

3. Try with --features metal -> Problem

cargo run --example rwkv --release --features metal -- --prompt "An empty prompt" Finished release [optimized] target(s) in 0.21s Running target/release/examples/rwkv --prompt 'An empty prompt' avx: false, neon: true, simd128: false, f16c: false temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64 retrieved the files in 327.208µs loaded the model in 1.27796575s Error: Metal error Invalid matmul arguments [2048, 64, 1, 2048] [2048, 64, 2048, 1] (64, 64, 1)

Caused by: Invalid matmul arguments [2048, 64, 1, 2048] [2048, 64, 2048, 1] (64, 64, 1)

I suspect indices have been transposed somewhere. HTH.

[LaurentMazare]

I've just merged #1719 which should hopefully fix this issue by making the tensor contiguous (the problem seems to be more of a striding issue than a wrong transposition).

[TomArbuckle]

Just to confirm that https://github.com/huggingface/candle/pull/1719 solves the issue. Thanks for the quick fix.

[hylarucoder]

Hi @LaurentMazare, I'm a big fan of your work!

I've encountered a recurring issue. the error that was fixed in #1719 has resurfaced.

When running the command:

cargo run --example rwkv --release --features metal -- --prompt "An empty prompt"

The metal feature was functioning correctly before #1943, as evidenced by commit 4523ecfb.

However, I've started to receive "Invalid matmul arguments" errors using latest code:

whisper metal feature appears to be broken as well

cargo run --example whisper --release --features metal --features="symphonia" -- --input=/Users/hylarucoder/output.wav --model=tiny

I hope this information is helpful. Please let me know if you need further details or clarification, as English is not my first language.Best regards.

[LaurentMazare]

Thanks for reporting this, I can confirm that this has been broken by the commit you've mentioned, I've just minted #1955 that should hopefully help with this (I've only tested on cuda but this should be equivalent to metal).