WebWorker work is unclear, console messages stops, debug is unreachable

[Nelfstor]

Describe the bug Hello! In Blazor WebAssembly I create a webWorker to processAudioData:

                var webWorker = await _workerService.GetWebWorker();
                _proccessingService = webWorker.GetService<IProcessingService>();

then i call it: var result = await _proccessingService.ProcessData(samples); For test - i hardcoded result but i always get the result of the parameterless constructor of the object response type. And console stops write any debug messages.

Here is Program.cs file:

builder.Services.AddBlazorJSRuntime();
builder.Services.AddWebWorkerService();
builder.Services.AddSingleton<IProcessingService, AudioProcessingService>();
await builder.Build().BlazorJSRunAsync();

Desktop

• OS: Windows
• Browser: Chrome 118.0.5993.118 (Official Build) (64-bit)

[LostBeard]

Hello @Nelfstor, thanks for posting. I am going to need more information to assist with this issue.

What version of .Net are you using? I do most of my tasting these days using .Net 8. Sometimes the other .Net versions act a little different.

What is the type you are trying to send to your WebWorker, the type of your 'samples' variable? Please show the entire type declaration (entire class) if it is not a built in type (string, int, etc.) Not all types are serializable and passable to WebWorkers.

What is the type of 'result'? Please show the entire type declaration (entire class) if it is not a built in type.

What does your 'ProcessData' method look like? If possible, please show your entire AudioProcessingService class and IProcessingService interface.

Please include your console output for your test.

I'm sure we can solve this with a little more information. Thanks!

[Nelfstor]

Hello @LostBeard !

I use .Net 7.0. I think i can switch to .Net 8.0.

Samples are array of float.

    public interface IProcessingService
    {
        Task<PitchDetectionResult> ProcessData(float[] audioBuffer);
    }

    public class AudioProcessingService : IProcessingService
    {
        public async Task<PitchDetectionResult> ProcessData(float[] audioBuffer)
        {
            var result = new PitchDetectionResult();

            result.setPitch(30);
            return result;    
        }
    }

    public class PitchDetectionResult
    {
        private float pitch;
        private float probability;
        private bool pitched;
}

Here is console:

(after playing note - it actually get the pitch (i've checked in debug) and then if the pitchResult.Pitch is negative - it should write point in the console. ( It should write something anyway).

            if (PitchDsp.PitchToMidiNote(pitch, out int midiPitchNote, out int midiCents))
            {
                PitchReadyEvent.Invoke(new PitchEventArgs(samples, midiPitchNote, midiCents));
                Console.WriteLine($" #MyLog || Note: {PitchDsp.GetNoteName(midiPitchNote, false, false)} || Cents: {midiCents}");
            }
            else
            {
                Console.Write($".");
            }

[LostBeard]

You have not included your entire 'PitchDetectionResult' class.

As is, it is understandable if it does not work as the only members of your 'PitchDetectionResult' class are private fields which are ignored when serializing with System.Text,Json.

System.Text,Json (Microsoft's) only serializes public properties.

From How to write .NET objects as JSON (serialize)

Fields are not supported in System.Text.Json in .NET Core 3.1. Custom converters can provide this functionality.

Can you include the entire 'PitchDetectionResult' class?

[LostBeard]

If you want the fields of your result class to be serializable, make them public properties instead like this:

    public class PitchDetectionResult
    {
        public float pitch { get; set; }
        public float probability { get; set; }
        public bool pitched { get; set; }
        // ... rest of your class
    }

[LostBeard]

As your PitchDetectionResult class is made of basic .Net types, you can check that it is serializing correctly with the code below.

var result = new PitchDetectionResult();
// modify your result so we can test the properties are being serialized and deserialized correctly.
// serialize
var resultSerialized = JsonSerializer.Serialize(result);
// deserialize
var resultDeserialized = JsonSerializer.Deserialize<PitchDetectionResult>(resultSerialized);
// compare result and resultDeserialized properties to see if they match

[Nelfstor]

@LostBeard thanks a lot! I forgot about the serializing side! Sorry! I turned fields into public properties and everything work!

Now there is another problem - that it works much slower relatively to the case when i did calculations directly (without webWorker).

Is there any way to give webWorker more resources? Or it is better to create pull of the webWorkers?

[LostBeard]

The WebWorkers work best for long running tasks. WebWorkers, depending on the system, can take anywhere from less than half a second to 10 seconds to start up. Make sure you are reusing them and not calling GetWebWorker() (which creates a new worker) for each request.

If your long running tasks overlap, you can take a look at WebWorkerPool, which I use in the face API demo (code). WebWorkerPool can start a set number of workers, then you can request a non-busy worker as needed for your WebWorker calls. In WebWorkerPool, a worker is considered busy if it is still processing a request. WebWorkerPool is a part of SpawnDev.BlazorJS.WebWorkers.

[Nelfstor]

Thank you!

Would you recommend webworkers to process realtime audio data?

[LostBeard]

Always happy to help.

Would you recommend webworkers to process realtime audio data?

I really can't say. I have no experience with processing audio in the browser other than using ffmpeg.wasm to convert videos with audio. That isn't realtime though.

Out of curiosity, what is the source of the audio? Audio files, streaming audio, MIDI, or other?

[Nelfstor]

Out of curiosity, what is the source of the audio? Audio files, streaming audio, MIDI, or other?

it is microphone input comes in pretty frequent events via audioContext. Array of 2048 float samples, several times a second. Even if there is no processing of this array - it slows down all. It looks like i need to write some algorithm to make a rest inside this data stream manually.

[Nelfstor]

And one more question, please.

It looks like the problem is huge amount of raw samples transferring from Js AudioContext to blazor code. And probably - the best way is to compress data or process it on the "JsSide".

Can i write function using Blazor.JS and implement it in "Js side"?

[LostBeard]

I do not understand the question.

[LostBeard]

If the data you are trying to process, I believe to be an array of floats (float[] audioBuffer), starts out in Javascript, you may get better performance by not pulling the array of floats into Blazor before sending it to the WebWorker. You could, instead, use an SpawnDev.BlazorJS.JSObjects.Array which is a reference to a Javascript array. Then when you pass the Array to the WebWorker you won't be doing an unnecessary serialization > deserialization into Blazor and then serialization > deserialization back to Javascript just to send it to the worker.

If you could share some more code or a mini version of code that acquires the data to be processed I can see if I can help or at least modify it to demonstrate what I am talking about.

[Nelfstor]

If the data you are trying to process, I believe to be an array of floats (float[] audioBuffer), starts out in Javascript, you may get better performance by not pulling the array of floats into Blazor before sending it to the WebWorker. You could, instead, use an SpawnDev.BlazorJS.JSObjects.Array which is a reference to a Javascript array. Then when you pass the Array to the WebWorker you won't be doing an unnecessary serialization > deserialization into Blazor and then serialization > deserialization back to Javascript just to send it to the worker.

Oh, wow! That's interesting approach. For now i already moved audio data processing to js to check if it is caused the problem. And after that performance increased dramatically.

Thank you a lot!

There is one problem i couldn't solve whatever side i approached. I need to do simultaneously 2 things:

• play midiFile ( or move visualization of the midi file).
• and receive audio from mic and process it.

Separately they work just fine, no delays, everything is fine. But just when i turn on the timer - everything slows down. I've tried c# standart timers, my own async timer, Js based timer - result is pretty much the same.

I know, it is not the topic of the library, so sorry for the offtop. I can share my repo with you, so you can click on "dynamic mode" and see what happens. Your advice will be valuable!

[LostBeard]

I think your particular project may benefit from Blazor WASM multithreading or even just using SharedArrayBuffers with WebWorkers to cut down on the data transfer to and from WebWorkers as it allows you to work on the data in place.

I do not currently have any examples to show how to use SharedArrayBuffer with WebWorkers but Microsoft has examples on how to use Blazor multithreading.

WebAssembly multithreading (experimental)

I can share my repo with you, so you can click on "dynamic mode" and see what happens.

I am willing to take a look to see what might help improve the processing speed.

[Nelfstor]

Thanks a lot!

For me the biggest problem is to bind js to blazor.

Here is how i get the data from processing node:

processorNode = audioContext.createScriptProcessor(bufferSize, 1, 1);
sourceNode = audioContext.createMediaStreamSource(audioStream);

      processorNode.onaudioprocess = function(event) {
            let inputData = event.inputBuffer.getChannelData(0);

            // Send the audio data to the Web Worker for processing
            audioWorker.postMessage({
                action: 'process',
                inputData: inputData
            });

Then i send it to Js Webworker, as soon i don't know how share buffer to process it in c#. When i tried to send entire buffer in events it caused performance to slow down significantly.

[LostBeard]

I'm going to play around with the example on MDN here. I'll post back here if I come up with something. I closed the issue, but feel free to write more here.

[Nelfstor]

I'm going to play around with the example on MDN here. I'll post back here if I come up with something. I closed the issue, but feel free to write more here.

ok! Thanks a lot! Any ideas are welcome!