Metal

[zodiac1214]

I tried to

cmake -DGGML_METAL=ON  ..
cmake --build . --config Release

but it is still only using CPU instead of Mac GPU

[PABannier]

Hello @zodiac1214 !

You are right; there is a mistake in the implementation, which makes it impossible for now to run Bark.cpp on Metal. I'll fix it in the next few days.

[ochafik]

FWIW, tried to wire the -ngl params here and hit a wall with:

ggml_metal_graph_compute_block_invoke: error: node 5, op = SET not implemented

(a ggml sync might help?)

git remote add ochafik https://github.com/ochafik/bark.cpp
git fetch ochafik
rm -fR build && \
  cmake -B build . -DGGML_METAL=1 -DCMAKE_BUILD_TYPE=Release && \
  cmake --build build && \
  cp build/bin/ggml-metal.metal build/encodec.cpp/ggml/src

./build/examples/main/main -m ./models/bark/ggml_weights.bin -p "Test" -t 4 -o out2.wav

show full output

``` ggml_metal_init: loaded kernel_silu 0x13ff44a50 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_relu 0x13ff44c80 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_gelu 0x13ff44eb0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_soft_max 0x13ff450e0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_soft_max_4 0x13ff45310 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_diag_mask_inf 0x13ff45540 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_diag_mask_inf_8 0x13ff45770 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_f32 0x13ff459a0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_f16 0x13ff45bd0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q4_0 0x13ff45e00 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q4_1 0x13ff46030 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q8_0 0x13ff46260 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q2_K 0x13ff46490 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q3_K 0x13ff466c0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q4_K 0x13ff468f0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q5_K 0x13ff46b20 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_get_rows_q6_K 0x13ff46d50 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_rms_norm 0x13ff46f80 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_norm 0x13ff471b0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_f32_f32 0x13ff473e0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_f16_f32 0x13ff47610 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_f16_f32_1row 0x13ff47840 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_f16_f32_l4 0x140125080 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_q4_0_f32 0x1401252b0 | th_max = 896 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_q4_1_f32 0x1401254e0 | th_max = 896 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_q8_0_f32 0x140125710 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_q2_K_f32 0x140125940 | th_max = 640 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_q3_K_f32 0x140125b70 | th_max = 576 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_q4_K_f32 0x140125da0 | th_max = 576 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_q5_K_f32 0x140125fd0 | th_max = 576 | th_width = 32 ggml_metal_init: loaded kernel_mul_mv_q6_K_f32 0x140126200 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_f32_f32 0x140126430 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_f16_f32 0x140126660 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q4_0_f32 0x140126890 | th_max = 704 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q8_0_f32 0x140126ac0 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q4_1_f32 0x140126cf0 | th_max = 704 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q2_K_f32 0x140126f20 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q3_K_f32 0x140127150 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q4_K_f32 0x140127380 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q5_K_f32 0x1401275b0 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_mul_mm_q6_K_f32 0x1401277e0 | th_max = 768 | th_width = 32 ggml_metal_init: loaded kernel_rope_f32 0x140127a10 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_rope_f16 0x140127c40 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_alibi_f32 0x140127e70 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_cpy_f32_f16 0x1401280a0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_cpy_f32_f32 0x1401282d0 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_cpy_f16_f16 0x140128500 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_concat 0x140128730 | th_max = 1024 | th_width = 32 ggml_metal_init: loaded kernel_sqr 0x140128960 | th_max = 1024 | th_width = 32 ggml_metal_init: GPU name: Apple M1 Ultra ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007) ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 98304.00 MB ggml_metal_init: maxTransferRate = built-in GPU ggml_metal_add_buffer: allocated 'backend ' buffer, size = 54.36 MB, ( 2561.59 / 98304.00) encodec_load_model_weights: model size = 44.36 MB encodec_load_model: n_q = 32 # bctx->n_gpu_layers = 99 bark_tokenize_input: prompt: 'I really love using llama.cpp and its ecosystem. They make me happy' bark_tokenize_input: number of tokens in prompt = 513, first 8 tokens: 10194 40229 26186 23430 57329 10167 10219 26635 ggml_metal_add_buffer: allocated 'backend ' buffer, size = 10.06 MB, ( 2571.66 / 98304.00) ggml_metal_graph_compute_block_invoke: error: node 5, op = SET not implemented GGML_ASSERT: /Users/ochafik/github/bark.cpp/encodec.cpp/ggml/src/ggml-metal.m:1428: false ```

[PABannier]

@ochafik Thanks for trying it!

Yes, we'll need to sync with the latest version of ggml. However, we'll have to implement additional operations to ggml and to write Metal kernels (e.g. sigmoid, pad_reflec_1).