AccessViolationException when whisper_full_with_state is called

[AurelienLc]

Hello,

When we start our pipeline, the Whisper model is downloaded and initialized.

However, when a speech event is detected and the "Process" method is called, in particular the "NativeMethods.whisper_full_with_state" method, we have an AccessViolationException with the comments : "Attempt to read or write protected memory." (see screen). We tried to solve the problem using "whisper_init_from_file" insteed of "whisper_init_from_file_no_state" in the LoadNativeContext() method of WhisperProcessorModelFileLoader.cs, or also to change "whisper_full_with_state" method with "whisper_full", without success.

Any suggestions?

Thanks !

[sandrohanea]

It sounds like some memory accessing issue in the underlying library. Is it one of the examples in this library with some custom audio?

Is it reproducing all the time? Can you share that audio with us?

[AurelienLc]

Yes, the problem occurs regardless of the audio played. We can't send you the audio for research privacy reasons.

To give you more details, we're trying to use Whisper in a real-time context with an IProducer<AudioBuffer, bool> that we pip to Whisper. The voice activity detector used is this from Microsoft Psi. After verification, we seem to have the exception presented above as soon as we receive the first audio event detected by the voice activity detector (bool true), and therefore a verbalization.

[sandrohanea]

Is the generated audio at 16khz? How are you getting the samples from that AudioBuffer to pass it to WhisperProcessor?

Are those samples mono or stereo and if stereo, how are they represented?

Also, can you, please, check if the issue is reproducible with the latest (1.5.0) version?

[AurelienLc]

We no longer have any errors, and are able to use Whisper. So it seems that the problem has been solved with the latest (1.5.0) version of Whisper.net.

Thanks !

[janjanusek]

I have update for this cause for me it did not solve just update, I figured out that disposing factory is making entire model crashing on this very place. SO in order to have everything fucntional with Whisper model you CANNOT dispose factory.

[sandrohanea]

Hello @janjanusek , Indeed, the factory (and processor) needs to stay "alive" as long as the processing is still happening as they held callback delegates which cannot be collected by GC.

As processing is happening in unmanaged code, these callbacks might be called at any time, so you need to manage their lifetime bound to the parent objects (processor / factory).

If you know that no processing is in progress, and that you won't start another one, you should be able to dispose factory / processor.