Closed hswlab closed 8 months ago
I tested this, the execution breaks - bark_forward_coarse_encoder: mem per token = 8.51 MB bark_forward_coarse_encoder: sample time = 11.67 ms bark_forward_coarse_encoder: predict time = 197814.42 ms / 610.54 ms per token bark_forward_coarse_encoder: total time = 198112.20 ms
Hi @hswlab !
Thanks for trying on Google Colab.
I tested the prompt "this is an audio" on multiple machines and it works for me. Maybe, the CPU provided by Google Colab is not powerful enough even to run bark.cpp
. A few suggestions:
--seed
Hi. @PABannier Sorry for asking here since I don't know how to contact you, but I want to know why bark_forward_fine_encoder tries to allocate 30GB of memory? None of the weights are at that size. Also using the original bark model(non ggml), it runs well.
fine_gpt_eval: failed to allocate 32031611289 bytes
bark_forward_fine_encoder: ggml_aligned_malloc: insufficient memory (attempted to allocate 30547.73 MB)
GGML_ASSERT: ggml.c:4408: ctx->mem_buffer != NULL
Hi @akarshanbiswas ! Thanks for reaching out on this problem. Are you able to track back which operations cause this surge in memory? Also, which prompt did you input to the model?
I followed the same instructions the OP has in his colab notebook. Additionally I tested with quantize weights using the scripts available in the repo.
I also found out that the weight with codec
in its name is not being quantized and the program crashes in the process.
I am yet to check the coredumps that I got which I will do in a few hours for now. (currently afk). 🙂
Yes! Codec weights are not currently quantized because it does not provide any significant speed-ups (the forward pass is already fast), but degrades the audio quality. I'm currently investigating other problems, so your input on this memory problem is very much welcome ;)
I moved this discussion to a separate issue.
@PABannier thank you, I tried to quantize the weights and could successfully generate an output.wav Here's the whole script for GooGle Colab, in case anyone else wants to try it out.
# install cmake
!apt update
!apt install -y cmake
# Clone bark c++ from github
%cd /content
!git clone https://github.com/PABannier/bark.cpp.git
# Build
%cd /content/bark.cpp
!mkdir build
%cd ./build
!cmake ..
!cmake --build . --config Release
# install Python dependencies
%cd /content/bark.cpp
!python3 -m pip install -r requirements.txt
# obtain the original bark and encodec weights and place them in ./models
!python3 download_weights.py --download-dir ./models
# convert the model to ggml format
!python3 convert.py \
--dir-model ./models \
--codec-path ./models \
--vocab-path ./ggml_weights/ \
--out-dir ./ggml_weights/
# Quantize weights
!mkdir ggml_weights_q4
!cp /content/bark.cpp/ggml_weights/ggml_vocab.bin /content/bark.cpp/ggml_weights_q4
!cp /content/bark.cpp/ggml_weights/ggml_weights_codec.bin /content/bark.cpp/ggml_weights_q4
!cp /content/bark.cpp/ggml_weights/vocab.txt /content/bark.cpp/ggml_weights_q4
!/content/bark.cpp/build/bin/quantize /content/bark.cpp/ggml_weights/ggml_weights_text.bin /content/bark.cpp/ggml_weights_q4/ggml_weights_text.bin q4_0
!/content/bark.cpp/build/bin/quantize /content/bark.cpp/ggml_weights/ggml_weights_coarse.bin /content/bark.cpp/ggml_weights_q4/ggml_weights_coarse.bin q4_0
!/content/bark.cpp/build/bin/quantize /content/bark.cpp/ggml_weights/ggml_weights_fine.bin /content/bark.cpp/ggml_weights_q4/ggml_weights_fine.bin q4_0
# run the inference
!/content/bark.cpp/build/bin/main -m ./ggml_weights_q4 -p "Hi this is a test with quantized weights"
# ZIP Files
!zip -r bark.zip /content/bark.cpp/build/bin /content/bark.cpp/ggml_weights /content/bark.cpp/ggml_weights_q4
# Move files to Google Drive for download
from google.colab import drive
drive.mount('/content/drive')
!cp /content/bark.cpp/bark.zip '/content/drive/My Drive/'
it would be great to have https://github.com/PABannier/bark.cpp/issues/95#issuecomment-1697916151 linked or incorporated into the README
@khimaros I added it to the README. Thank you @hswlab !
Can anyone show a working example on Google Colab where a concrete audio file is generated? In my attempts, execution strangely breaks after these lines.
Here is the link to my attempt on Google Colab: https://colab.research.google.com/drive/1JVtJ6CDwxtKfFmEd8J4FGY2lzdL0d0jT?usp=sharing