Not working on visionOS 2

[florentmorin]

Hello,

I’ve encountered an issue where WhisperKit, which works perfectly on visionOS 1, no longer functions correctly on visionOS 2 beta 3. (native code)

Here’s a snippet of the code I’m using:

let whisperKit = try await WhisperKit(load: true)
// ...
let results = try await whisperKit.transcribe(
    audioPath: audioPath, 
    decodeOptions: .init(language: language), 
    callback: callback
)

I have attempted the following troubleshooting steps:

• Uninstalled and reinstalled the app.
• Built the project using Xcode 15.4 and the latest Xcode 16 beta.

Despite these efforts, the issue persists.

🙏 Thanks for your help.

[atiorh]

Hello! What is the particular issue?

[florentmorin]

@atiorh

Result with ted_60.m4a in debug console mode returns "I am a very kind of a a a a a ..."

Console output

``` ▿ 1 element ▿ 0 : TranscriptionResult - text : "I am a very kind of a a a a a a a a a a a b a a a b e a e b e a b e a a b a a a f e d b d a a d f a b a a d b d f a a b a b a a d b a a a a a b b f a a a b b b a b b a a b a d b b b b b b b b b b f b b b b b b b b b b c b b b b b b b b b b b b b b b b b b b b b b c f b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b g e g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>" ▿ segments : 2 elements ▿ 0 : TranscriptionSegment - id : 0 - seek : 0 - start : 0.0 - end : 30.0 - text : "<|startoftranscript|><|en|><|transcribe|><|0.00|> I am a very kind of a a a a a a a a a a a b a a a b e a e b e a b e a a b a a a f e d b d a a d f a b a a d b d f a a b a b a a d b a a a a a b b f a a a b b b a b b a a b a d b b b b b b b b b b f b b b b b b b b b b c b b b b b b b b b b b b b b b b b b b b b b c f b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b g e g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g<|endoftext|>" ▿ tokens : 224 elements - 0 : 50258 - 1 : 50259 - 2 : 50359 - 3 : 50364 - 4 : 286 - 5 : 669 - 6 : 257 - 7 : 588 [...] - 221 : 290 - 222 : 290 - 223 : 50257 ▿ tokenLogProbs : 224 elements ▿ 0 : 1 element ▿ 0 : 2 elements - key : 50258 - value : 0.0 ▿ 1 : 1 element ▿ 0 : 2 elements - key : 50259 - value : 0.0 ▿ 2 : 1 element ▿ 0 : 2 elements - key : 50359 - value : 0.0 ▿ 3 : 1 element ▿ 0 : 2 elements - key : 50364 - value : 0.0 ▿ 4 : 1 element ▿ 0 : 2 elements - key : 286 - value : -3.8560698 ▿ 5 : 1 element ▿ 0 : 2 elements - key : 669 - value : -1.9917988 ▿ 6 : 1 element ▿ 0 : 2 elements - key : 257 - value : -2.342525 ▿ 7 : 1 element ▿ 0 : 2 elements - key : 588 - value : -3.8727763 ▿ 8 : 1 element ▿ 0 : 2 elements - key : 733 - value : -4.0190206 ▿ 9 : 1 element ▿ 0 : 2 elements - key : 295 - value : -0.53651595 [...] ▿ 222 : 1 element ▿ 0 : 2 elements - key : 290 - value : -0.005539653 ▿ 223 : 1 element ▿ 0 : 2 elements - key : 50257 - value : 0.0 - temperature : 1.0 - avgLogprob : -0.7730474 - compressionRatio : 8.504854 - noSpeechProb : 0.0 - words : nil ▿ 1 : TranscriptionSegment - id : 1 - seek : 480000 - start : 30.0 - end : 60.0 - text : "<|startoftranscript|><|en|><|transcribe|><|0.00|> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >><|endoftext|>" ▿ tokens : 224 elements - 0 : 50258 - 1 : 50259 - 2 : 50359 - 3 : 50364 - 4 : 902 - 5 : 902 - 6 : 902 - 7 : 902 [...] - 222 : 902 - 223 : 50257 ▿ tokenLogProbs : 224 elements ▿ 0 : 1 element ▿ 0 : 2 elements - key : 50258 - value : 0.0 ▿ 1 : 1 element ▿ 0 : 2 elements - key : 50259 - value : 0.0 ▿ 2 : 1 element ▿ 0 : 2 elements - key : 50359 - value : 0.0 ▿ 3 : 1 element ▿ 0 : 2 elements - key : 50364 - value : 0.0 ▿ 4 : 1 element ▿ 0 : 2 elements - key : 902 - value : -3.266967 ▿ 5 : 1 element ▿ 0 : 2 elements - key : 902 - value : -1.445334 ▿ 6 : 1 element ▿ 0 : 2 elements - key : 902 - value : -0.5025386 [...] ▿ 221 : 1 element ▿ 0 : 2 elements - key : 902 - value : -0.010405066 ▿ 222 : 1 element ▿ 0 : 2 elements - key : 902 - value : -0.010017514 ▿ 223 : 1 element ▿ 0 : 2 elements - key : 50257 - value : 0.0 - temperature : 1.0 - avgLogprob : -0.037366472 - compressionRatio : 62.57143 - noSpeechProb : 0.0 - words : nil - language : "en" ▿ timings : TranscriptionTimings - pipelineStart : 742328429.696212 - firstTokenTime : 742328429.96273 - inputAudioSeconds : 60.0 - modelLoading : 1.0579739809036255 - audioLoading : 0.028718948364257812 - audioProcessing : 0.0010149478912353516 - logmels : 0.07123291492462158 - encoding : 0.35921692848205566 - prefill : 2.5987625122070312e-05 - decodingInit : 0.005879998207092285 - decodingLoop : 32.84668803215027 - decodingPredictions : 24.473331809043884 - decodingFiltering : 0.022301673889160156 - decodingSampling : 1.4053410291671753 - decodingFallback : 32.41148316860199 - decodingWindowing : 0.0008039474487304688 - decodingKvCaching : 0.990674614906311 - decodingWordTimestamps : 0.0 - decodingNonPrediction : 7.911233186721802 - totalAudioProcessingRuns : 2.0 - totalLogmelRuns : 2.0 - totalEncodingRuns : 2.0 - totalDecodingLoops : 2423.0 - totalKVUpdateRuns : 2411.0 - totalTimestampAlignmentRuns : 0.0 - totalDecodingFallbacks : 5.0 - totalDecodingWindows : 2.0 - fullPipeline : 32.85519599914551 - seekTime : nil ```

[ZachNagengast]

@florentmorin This looks similar to CPU only output. I believe that since visonOS doesn't let us use ANE, the model is using only CPU in this case. Could you try specifying GPU like this:

let computeOptions = ModelComputeOptions(audioEncoderCompute: .cpuAndGPU, textDecoderCompute: .cpuAndGPU)
let whisperKit = try await WhisperKit(computeOptions: computeOptions, load: true)