Canceling ProcessAsync returns before inference terminates.

[Alumniminium]

try
{
    await foreach (var segment in processor.ProcessAsync(decodedFileStream, ctx))
        yield return segment;
}
finally
{
    processor.Dispose();
}

// CPU Usage is still 100% here

I can see 100% CPU usage after ProcessAsync throws.

if I start to process another file before the CPU usage drops to zero, it pretty much crawls to a halt for minutes until the original instance terminates.

[Alumniminium]

Current Workaround:

try
{
    await foreach (var segment in processor.ProcessAsync(decodedFileStream, ctx))
        yield return segment;
}
finally
{
    processor.Dispose();
    float cpuUsage;
    while ((cpuUsage = await GetCpuUsage()) > 10)
        Console.WriteLine($"CPU usage too high ({cpuUsage:0.00})%, waiting...");
}
...

private static async Task<float> GetCpuUsage()
{
      var startTimestamp = Stopwatch.GetTimestamp();
      var startCPU = System.Diagnostics.Process.GetCurrentProcess().TotalProcessorTime;
      await Task.Delay(1000);
      var endCPU = System.Diagnostics.Process.GetCurrentProcess().TotalProcessorTime;
      var msPassed = Stopwatch.GetElapsedTime(startTimestamp).TotalMilliseconds;
      var msCPU = (endCPU - startCPU).TotalMilliseconds;
      var usage = msCPU / (Environment.ProcessorCount * msPassed);
      return (float)usage * 100f;
}

[sandrohanea]

Indeed, the processing is not finished once Process sync is cancelled, similar to #23. And it can even crash (as the callbacks will be unregistered). Will address it asap.

[sandrohanea]

I added a commit to fix it and will release it in 1.3.0 version. However, after cancelling, you'll have to call DisposeAsync() instead of Dispose now as otherwise Dispose method will throw (as the processing is still happening).

DisposeAsync will wait until all resources can be safely released.

Also, a newer encoder will be cancelled now, so the rest of processing will finish faster, but the decoders cannot be cancelled (as whisper.cpp is not supporting that option yet).

[Alumniminium]

Thank you. Please make sure that the1.3.0 version pulls the latest master from whisper.cpp, as we just resolved the quantization issue there.

jfk sample transcript time on ggml-large-v2.bin @ Ryzen 4600U
base   = 58sec
5_1b   = 45sec
4_0b   = 41sec

[Alumniminium]

I see you quickly released another tag 1.4.0 adding the latest master among other things. Thank you!