CoreML not working on Mac

[jettoblack]

I'm trying to run the CoreML example project with no changes in VS Code. I have 2 ARM Macbook Pros: M1 Pro running latest macOS Sonoma 14.1.1, and a M2 Max running Ventura 13.6. Both machines can run the Simple example just fine, but both fail to load the native library for the CoreML example:

Unhandled exception. System.Exception: Failed to load native whisper library. Error: Unknown error at Whisper.net.WhisperFactory..ctor(IWhisperProcessorModelLoader loader, Boolean delayInit, String libraryPath, Boolean bypassLoading) at Whisper.net.WhisperFactory.FromPath(String path, Boolean delayInitialization, String libraryPath, Boolean bypassLoading) at Program.Main(String[] args) in /Users/jasonl/Downloads/whisper.net/examples/CoreML/Program.cs:line 35 at Program.<Main>(String[] args)

In ./bin/Debug/net6.0/runtimes/osx-arm64 I have: libwhisper.coreml.dylib whisper.dylib

I see both ggml-base.bin and ggml-base-encoder.mlmodelc downloaded and unpacked successfully. I tried copying these files to the bin or the runtimes/osx-arm64 folders so they are in the same place as the DLLs, but that didn't change anything.

I tried updating the projects to .net7, updating my build tools, etc. and nothing helped.

Is there any additional information I can provide that would be helpful for debugging? Thanks!

[sandrohanea]

Hello @jettoblack , Can you, please, try with the latest whisper 1.5.0 to ensure that it wasn't some old issue that wasn't reproducing anymore?

I tested it on my mac (but it is intel-based). Also, ensured that build for arm64 is correctly executed.

Best way to test it with the setup that you described is to run:

go to whisper.net/examples/CoreML directory.

dotnet build ./CoreML.csproj --framework net8.0
cd bin/Debug/net8.0
./CoreML

Please, let me know if the issue persists.

[jettoblack]

@sandrohanea I tried both 1.4.7 and 1.5.0 on both arm Macs I have, and ran the CoreML example exactly as specified above, but I'm still getting the same error. I can run whisper.cpp in CoreML mode and it works on both of these Macs. I also got the Cublas example working on Windows.

The CPU-only version is pretty fast as it is and I just read that CoreML isn't really working great with medium/large models yet, so maybe this is a moot point and I should stick to the CPU runtime, but thanks for looking into it if you have a chance.

[sandrohanea]

Whisper.net.Runtime is now using Metal on ARM-based Macs, this is why it is fast already without CoreML :)

Can you, please, run otool -L ./libwhisper.dylib in the directory of that CoreML example /build/Debug/net8.0/runtimes/macos-arm64?

Also, can you, please, check if something relevant is displayed in the Console.app when you're trying to run the CoreML example? (search by CoreML in the upper right search box, for fewer results)

[philippjbauer]

I'm running into the problem that the native library can't be found when invoking transcription (.NET System.Exception thrown on the line). Perhaps a packaging issue again?

I ran the otool -L command above, here's the output:

➜ otool -L ./libwhisper.dylib 
./libwhisper.dylib:
        @rpath/libwhisper.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
        /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1953.255.0)
        /System/Library/Frameworks/Metal.framework/Versions/A/Metal (compatibility version 1.0.0, current version 306.3.4)
        /System/Library/Frameworks/MetalKit.framework/Versions/A/MetalKit (compatibility version 1.0.0, current version 154.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1953.255.0)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

[Sing303]

Similar problem, after executing

dotnet build ./CoreML.csproj --framework net8.0
cd bin/Debug/net8.0
./CoreML

Unhandled Exception. System Exception: Failed to load native whisper library. On my M1 Mac Mini

[sandrohanea]

I double-checked the package, and I don't see any issue with it :( Unfortunately, as I said before, I don't have an ARM-based mac to test it and attempt a fix.

Maybe someone can try to build a new runtime with the command:

make macos_arm64_coreml 

ensure that cmake + build tools (xcode) are installed.

then, copy the dylib manually from: ./Whisper.net.Runtime.CoreML/macos-arm64/libwhisper.dylib and ./Whisper.net.Runtime.CoreML/macos-arm64/libwhisper.coreml.dylib to (yourBuildPath)/runtimes/macos-arm64/ (make sure that you replace the existing ones and try to run the sample again.

This way we can check if there is some issue with the build script or with the build artifact.

Also, it would be great if someone can confirm if they manage to execute it.

[philippjbauer]

It errors out for me with:

Makefile:10: *** NDK_PATH not defined and NDK not found at default location on Mac.. Stop.

The Xamarin Android SDK is missing or something?

Edit: there's no macos_arm64_coreml option in the makefile. Did you mean macos_coreml? The Xamarin Android SDK isn't relevant for that build, correct?

[sandrohanea]

In the latest whisper.net main, that should be just a warning: https://github.com/sandrohanea/whisper.net/blob/4308888d4a202265dcc5d0f40bc4cce7dfdae255/Makefile#L10C9-L10C109

Please, make sure you pull the latest changes and also sync the whisper.cpp submodule:

git pull
git submodule update

[philippjbauer]

Removing the condition in the makefile and running macos_coreml gave me a working dylib file.

I got the following output regarding a missing ggml-metal.metal file. It still transcribed my audio file. I copied a metal file from a previous version next to the new libraries. The program crashed trying it this way though, I guess I need an updated one but don't know how to generate it / where to download it.

ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Pro
ggml_metal_init: picking default device: Apple M1 Pro
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSUnderlyingError=0x2a6edff60 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Pro
ggml_metal_init: picking default device: Apple M1 Pro
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSUnderlyingError=0x2a6935300 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}

libwhisper.coreml.macos_coreml.arm64.zip

[philippjbauer]

Nevermind, found the metal file in the whisper.cpp directory. Added it to the zip file:

libwhisper.coreml.macos_coreml.arm64_with_metal.zip

This works as expected now

[sandrohanea]

Nice, just to confirm, you synched latest changed from whisper.net and whisper.cpp? Because that NDK error was removed and also contained a different text in latest version: NDK_PATH not defined and NDK not found at default location on Mac. Android cannot be built

[philippjbauer]

Good call. It was still behind, I thought I had it but didn't! Here's a working version from the latest main branch.

libwhisper.coreml.dylib.metal.arm64.zip

[sandrohanea]

So latest main still worked without any additional change?

That's at least strange, as same build was done from github actions (but using an intel-based mac agent). ARM-based macos is not yet available free of charge for open source: https://github.com/orgs/community/discussions/69211

[philippjbauer]

That's correct, no additional changes needed. We have the same issue with our project (https://github.com/tryphotino/photino.Native), I have to compile our native libraries for macOS ARM64 manually and upload it for the build process to copy it into the resulting NuGet package.

I hope we'll get build machines for that soon 👍

[philippjbauer]

@sandrohanea I did some testing and ran into the following issue where it seems to load the model twice (?)

To test I recompiled everything again. And there was an update to whisper.cpp (see attachment). It works with those updates, but it still has that same error:

ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Pro
ggml_metal_init: picking default device: Apple M1 Pro
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/path-to-project/bin/Debug/net8.0/runtimes/macos-arm64/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M1 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 22906.50 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Pro
ggml_metal_init: picking default device: Apple M1 Pro
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/path-to-project/bin/Debug/net8.0/runtimes/macos-arm64/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M1 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 22906.50 MB
ggml_metal_init: maxTransferRate               = built-in GPU

I also experienced that the new large-v3 model produces garbage when tried with these binaries. The CPU/Metal version in Whisper.net.Runtime (1.5.0) run the model just fine. Not sure what's going on here.

whisper_osx_arm64_coreml-metal.zip