search

argmaxinc / WhisperKit

On-device Inference of Whisper Speech Recognition Models for Apple Silicon
https://takeargmax.com/blog/whisperkit
MIT License
2.92k stars 246 forks source link

make test seems to generate (a) lots of output (b) some test failures #182

Open jrp2014 opened 4 days ago

jrp2014 commented 4 days ago

Here's what I get

Test Case '-[WhisperKitTests.FunctionalTests testRealTimeFactorLarge]' started.
/Users/jrp/Documents/AI/whisperkit/Sources/WhisperKit/Core/Models.swift:34: error: -[WhisperKitTests.FunctionalTests testRealTimeFactorLarge] : failed: caught error: "Error Domain=com.apple.CoreML Code=0 "Unable to load model: file:///Users/jrp/Documents/AI/whisperkit/.build/arm64-apple-macosx/debug/whisperkit_WhisperKitTests.bundle/whisperkit-coreml/openai_whisper-large-v3/MelSpectrogram.mlmodelc/. Compile the model with Xcode or `MLModel.compileModel(at:)`. " UserInfo={NSLocalizedDescription=Unable to load model: file:///Users/jrp/Documents/AI/whisperkit/.build/arm64-apple-macosx/debug/whisperkit_WhisperKitTests.bundle/whisperkit-coreml/openai_whisper-large-v3/MelSpectrogram.mlmodelc/. Compile the model with Xcode or `MLModel.compileModel(at:)`. , NSUnderlyingError=0x60000163f180 {Error Domain=com.apple.CoreML Code=3 "/Users/jrp/Documents/AI/whisperkit/.build/arm64-apple-macosx/debug/whisperkit_WhisperKitTests.bundle/whisperkit-coreml/openai_whisper-large-v3/MelSpectrogram.mlmodelc/coremldata.bin is not a valid .mlmodelc file because the first word (1936876918) is not recognizable. : unspecified iostream_category error" UserInfo={NSLocalizedDescription=/Users/jrp/Documents/AI/whisperkit/.build/arm64-apple-macosx/debug/whisperkit_WhisperKitTests.bundle/whisperkit-coreml/openai_whisper-large-v3/MelSpectrogram.mlmodelc/coremldata.bin is not a valid .mlmodelc file because the first word (1936876918) is not recognizable. : unspecified iostream_category error}}}"
Test Case '-[WhisperKitTests.FunctionalTests testRealTimeFactorLarge]' failed (0.004 seconds).
Test Case '-[WhisperKitTests.FunctionalTests testRealTimeFactorTiny]' started.
/Users/jrp/Documents/AI/whisperkit/Sources/WhisperKit/Core/Models.swift:34: error: -[WhisperKitTests.FunctionalTests testRealTimeFactorTiny] : failed: caught error: "Error Domain=com.apple.CoreML Code=0 "Unable to load model: file:///Users/jrp/Documents/AI/whisperkit/.build/arm64-apple-macosx/debug/whisperkit_WhisperKitTests.bundle/whisperkit-coreml/openai_whisper-tiny/MelSpectrogram.mlmodelc/. Compile the model with Xcode or `MLModel.compileModel(at:)`. " UserInfo={NSLocalizedDescription=Unable to load model: file:///Users/jrp/Documents/AI/whisperkit/.build/arm64-apple-macosx/debug/whisperkit_WhisperKitTests.bundle/whisperkit-coreml/openai_whisper-tiny/MelSpectrogram.mlmodelc/. Compile the model with Xcode or `MLModel.compileModel(at:)`. , NSUnderlyingError=0x6000009d4870 {Error Domain=com.apple.CoreML Code=3 "/Users/jrp/Documents/AI/whisperkit/.build/arm64-apple-macosx/debug/whisperkit_WhisperKitTests.bundle/whisperkit-coreml/openai_whisper-tiny/MelSpectrogram.mlmodelc/coremldata.bin is not a valid .mlmodelc file because the first word (1936876918) is not recognizable. : unspecified iostream_category error" UserInfo={NSLocalizedDescription=/Users/jrp/Documents/AI/whisperkit/.build/arm64-apple-macosx/debug/whisperkit_WhisperKitTests.bundle/whisperkit-coreml/openai_whisper-tiny/MelSpectrogram.mlmodelc/coremldata.bin is not a valid .mlmodelc file because the first word (1936876918) is not recognizable. : unspecified iostream_category error}}}"
Test Case '-[WhisperKitTests.FunctionalTests testRealTimeFactorTiny]' failed (0.000 seconds).
Test Suite 'FunctionalTests' failed at 2024-07-05 23:41:03.648.
     Executed 10 tests, with 6 failures (5 unexpected) in 166.482 (166.482) seconds
Test Suite 'RegressionTests' started at 2024-07-05 23:41:03.648.

There are quite a few instances of 

/Users/jrp/Documents/AI/whisperkit/Tests/WhisperKitTests/RegressionTests.swift:115: error: -[WhisperKitTests.RegressionTests testRegressionAndLatencyForAllModels] : failed - Failed with error: invalidValue(inf, Swift.EncodingError.Context(codingPath: [CodingKeys(stringValue: "latencyStats", intValue: nil), CodingKeys(stringValue: "measurements", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "min", intValue: nil)], debugDescription: "Unable to encode Float.inf directly in JSON.", underlyingError: nil))
ZachNagengast commented 4 days ago

I see, make test may not be in a good state right now due to the recent changes. One thing to try is to make sure make download-models completes successfully, and also that xcode runs them properly too. I will investigate this shortly.