search

bnosac / audio.whisper

Transcribe audio files using the "Whisper" Automatic Speech Recognition model from R
Other
102 stars 11 forks source link

Integrate CoreML #35

Open jwijffels opened 5 months ago

jwijffels commented 5 months ago

Should be enabled with this commit: https://github.com/bnosac/audio.whisper/commit/f81a8179870c786afadc9ce3b221733d0a6ba8e4

Install with

Sys.setenv(WHISPER_COREML = "1")
remotes::install_github("bnosac/audio.whisper", force = TRUE)

As shown at https://github.com/ggerganov/whisper.cpp?tab=readme-ov-file#core-ml-support Get one of these coreml models from https://huggingface.co/ggerganov/whisper.cpp/tree/d15393806e24a74f60827e23e986f0c10750b358 unzip and put it at the same path as the non-coreml model such that you have these in your working directory

ggml-base.en-encoder.mlmodelc/...
ggml-base.en.bin

provide the path to non-coreml model model <- whisper("ggml-base.en.bin")

From the README at whisper.cpp:

The first run on a device is slow, since the ANE service compiles the Core ML model to some device-specific format. Next runs are faster.

jmgirard commented 5 months ago

I can look into testing this today.

jmgirard commented 5 months ago

I ran it twice on rant1.mp3 with the medium model.

# No CoreML
remotes::install_github("bnosac/audio.whisper", force = TRUE)
model <- audio.whisper::whisper("medium")
trans <- audio.whisper::predict(model, newdata = "output.wav", language = "en", n_threads = 1)
trans$timing
#> 35.22 min

# CoreML
Sys.setenv(WHISPER_COREML = "1")
remotes::install_github("bnosac/audio.whisper", force = TRUE)
model <- audio.whisper::whisper("ggml-medium.bin")
trans1 <- audio.whisper::predict(model, newdata = "output.wav", language = "en", n_threads = 1)
trans1$timing
#> 11.53 min
trans2 <- audio.whisper::predict(model, newdata = "output.wav", language = "en", n_threads = 1)
trans2$timing
#> 10.59 min
jwijffels commented 5 months ago

That speedup on 1 thread seems to correspond to the 3x speedup mentionned in https://github.com/ggerganov/whisper.cpp?tab=readme-ov-file#core-ml-support The 1min speed gain between the 2 runs is smaller than I thougt after reading that comment.

jwijffels commented 5 months ago

I would be interested to understand if this now also gives an extra speedup in addition to Metal?

jmgirard commented 5 months ago
Trace ```r > Sys.setenv(WHISPER_ACCELERATE = "1") > Sys.setenv(WHISPER_METAL = "1") > Sys.setenv(WHISPER_COREML = "1") > remotes::install_github("bnosac/audio.whisper", force = TRUE) Downloading GitHub repo bnosac/audio.whisper@HEAD ── R CMD build ────────────────────────────────────────────────── ✔ checking for file ‘/private/var/folders/kr/tx86v16n5bx_djz_z2cpvfkc0000gq/T/RtmplIhCFa/remotes12843580b7cbe/bnosac-audio.whisper-ecdb06d/DESCRIPTION’ ... ─ preparing ‘audio.whisper’: ✔ checking DESCRIPTION meta-information ─ cleaning src ─ checking for LF line-endings in source and make files and shell scripts ─ checking for empty or unneeded directories ─ building ‘audio.whisper_0.3.2.tar.gz’ Installing package into ‘/opt/homebrew/lib/R/4.3/site-library’ (as ‘lib’ is unspecified) * installing *source* package ‘audio.whisper’ ... ** using staged installation ** libs using C++ compiler: ‘Apple clang version 15.0.0 (clang-1500.1.0.2.5)’ using C++11 using SDK: ‘MacOSX14.2.sdk’ I whisper.cpp build info: I UNAME_S: Darwin I UNAME_P: arm I UNAME_M: arm64 I PKG_CFLAGS: -DGGML_USE_ACCELERATE -DGGML_USE_METAL -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread I PKG_CPPFLAGS: -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread I PKG_LIBS: -framework Accelerate -framework Foundation -framework CoreML -framework Foundation -framework Metal -framework MetalKit clang -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -DGGML_USE_ACCELERATE -DGGML_USE_METAL -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -fPIC -g -O2 -c whisper_cpp/ggml-quants.c -o whisper_cpp/ggml-quants.o clang -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -DGGML_USE_ACCELERATE -DGGML_USE_METAL -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -fPIC -g -O2 -c whisper_cpp/ggml-backend.c -o whisper_cpp/ggml-backend.o clang -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -DGGML_USE_ACCELERATE -DGGML_USE_METAL -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -fPIC -g -O2 -c whisper_cpp/ggml-alloc.c -o whisper_cpp/ggml-alloc.o clang -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -DGGML_USE_ACCELERATE -DGGML_USE_METAL -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -fPIC -g -O2 -c whisper_cpp/ggml.c -o whisper_cpp/ggml.o whisper_cpp/ggml.c:9828:17: warning: 'cblas_sgemm' is deprecated: first deprecated in macOS 13.3 - An updated CBLAS interface supporting ILP64 is available. Please compile with -DACCELERATE_NEW_LAPACK to access the new headers and -DACCELERATE_LAPACK_ILP64 for ILP64 support. [-Wdeprecated-declarations] cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/vecLib.framework/Headers/cblas.h:610:6: note: 'cblas_sgemm' has been explicitly marked deprecated here void cblas_sgemm(const enum CBLAS_ORDER __Order, ^ whisper_cpp/ggml.c:10238:9: warning: 'cblas_sgemm' is deprecated: first deprecated in macOS 13.3 - An updated CBLAS interface supporting ILP64 is available. Please compile with -DACCELERATE_NEW_LAPACK to access the new headers and -DACCELERATE_LAPACK_ILP64 for ILP64 support. [-Wdeprecated-declarations] cblas_sgemm(CblasRowMajor, transposeA, CblasNoTrans, m, n, k, 1.0, a, lda, b, n, 0.0, c, n); ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/vecLib.framework/Headers/cblas.h:610:6: note: 'cblas_sgemm' has been explicitly marked deprecated here void cblas_sgemm(const enum CBLAS_ORDER __Order, ^ 2 warnings generated. clang++ -std=gnu++11 -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -fPIC -g -O2 -c whisper_cpp/whisper.cpp -o whisper_cpp/whisper.o clang++ -std=gnu++11 -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -fPIC -g -O2 -c whisper_cpp/common-ggml.cpp -o whisper_cpp/common-ggml.o clang++ -std=gnu++11 -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -fPIC -g -O2 -c whisper_cpp/common.cpp -o whisper_cpp/common.o clang++ -std=gnu++11 -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -fPIC -g -O2 -c rcpp_whisper.cpp -o rcpp_whisper.o clang++ -std=gnu++11 -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -fPIC -g -O2 -c RcppExports.cpp -o RcppExports.o clang -I"/opt/homebrew/Cellar/r/4.3.2/lib/R/include" -DNDEBUG -DWHISPER_USE_COREML -fobjc-arc -DGGML_USE_METAL -DSTRICT_R_HEADERS -I./dr_libs -I./whisper_cpp -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -I'/opt/homebrew/lib/R/4.3/site-library/Rcpp/include' -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/xz/include -I/opt/homebrew/include -fPIC -g -O2 -fobjc-exceptions -c whisper_cpp/ggml-metal.m -o whisper_cpp/ggml-metal.o whisper_cpp/ggml-metal.m:479:5: error: 'release' is unavailable: not available in automatic reference counting mode GGML_METAL_DEL_KERNEL(add); ^ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:479:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(add); ^~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:479:5: error: 'release' is unavailable: not available in automatic reference counting mode whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:479:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(add); ^~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:480:5: error: 'release' is unavailable: not available in automatic reference counting mode GGML_METAL_DEL_KERNEL(add_row); ^ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:480:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(add_row); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:480:5: error: 'release' is unavailable: not available in automatic reference counting mode whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:480:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(add_row); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:481:5: error: 'release' is unavailable: not available in automatic reference counting mode GGML_METAL_DEL_KERNEL(mul); ^ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:481:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(mul); ^~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:481:5: error: 'release' is unavailable: not available in automatic reference counting mode whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:481:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(mul); ^~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:482:5: error: 'release' is unavailable: not available in automatic reference counting mode GGML_METAL_DEL_KERNEL(mul_row); ^ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:482:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(mul_row); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:482:5: error: 'release' is unavailable: not available in automatic reference counting mode whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:482:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(mul_row); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:483:5: error: 'release' is unavailable: not available in automatic reference counting mode GGML_METAL_DEL_KERNEL(div); ^ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ whisper_cpp/ggml-metal.m:483:5: error: ARC forbids explicit message send of 'release' GGML_METAL_DEL_KERNEL(div); ^~~~~~~~~~~~~~~~~~~~~~~~~~ whisper_cpp/ggml-metal.m:476:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->function_##name release]; \ ~~~~~~~~~~~~~~~~~~~~ ^ whisper_cpp/ggml-metal.m:483:5: error: 'release' is unavailable: not available in automatic reference counting mode whisper_cpp/ggml-metal.m:477:27: note: expanded from macro 'GGML_METAL_DEL_KERNEL' [ctx->pipeline_##name release]; ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/objc/NSObject.h:37:1: note: 'release' has been explicitly marked unavailable here - (oneway void)release OBJC_ARC_UNAVAILABLE; ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated. make: *** [whisper_cpp/ggml-metal.o] Error 1 ERROR: compilation failed for package ‘audio.whisper’ * removing ‘/opt/homebrew/lib/R/4.3/site-library/audio.whisper’ * restoring previous ‘/opt/homebrew/lib/R/4.3/site-library/audio.whisper’ Warning message: In i.p(...) : installation of package ‘/var/folders/kr/tx86v16n5bx_djz_z2cpvfkc0000gq/T//RtmplIhCFa/file1284310bd00dd/audio.whisper_0.3.2.tar.gz’ had non-zero exit status ```
jwijffels commented 5 months ago

Looks like the PKG_CPPFLAGS += -fobjc-arc at https://github.com/bnosac/audio.whisper/blob/master/src/Makevars#L117 needed for compiling CoreML is incompatible with the compilation of metal.m That means I'll have to split the targets in the Makevars to allow Metal together with CoreML.