search

ikawrakow / ik_llama.cpp

llama.cpp clone with additional SOTA quants and improved CPU performance
MIT License
57 stars 4 forks source link

BF16 support on Metal #56

Closed ikawrakow closed 2 days ago

ikawrakow commented 3 days ago

It is slightly slower than fp16, but definitely a massive improvement compared to not having bf16 support at al. ~Didn't put any effort into optimizing the matrix x vector kernel, so it is likely one can improve bf16 TG performance~.

model size params backend ngl test t/s
llama 8B BF16 14.96 GiB 8.03 B Metal 100 pp512 538.84 ± 0.26
llama 8B F16 14.96 GiB 8.03 B Metal 100 pp512 587.26 ± 0.39
llama 8B BF16 14.96 GiB 8.03 B Metal 100 tg128 21.64 ± 0.05
llama 8B F16 14.96 GiB 8.03 B Metal 100 tg128 21.77 ± 0.03