Raw examples of running DeepFilterNet3 on the Web (in JS)

[grazder]

Hello! Recently we've merged PR https://github.com/Rikorose/DeepFilterNet/pull/452 with WASM conversion support. I've now uploaded a couple of raw examples using DeepFilterNet3 on the web. One of them using wasm and the other one using onnxruntime-web.

Here is branch - https://github.com/grazder/samejs/tree/first_demo/deepfilternet3

So, if anyone needed it, I uploaded it. These examples are very raw. I'm going to document it, explain how it works, and write better code. And then, perhaps, soon I will come with some kind of PR.

[hlevring]

It's probably be obvious to anyone trying it out, but just wanted to mention a small mistake in the directory copy targets below from the readme.md

cp -r libdf/pkg ../samejs/deepfilternet3/wasm_worklet_worker_sharedarraybuffer/ cp models/DeepFilterNet3_onnx.tar.gz ../samejs/deepfilternet3/wasm_worklet_worker_sharedarraybuffer/ cd ../samejs/deepfilternet3/wasm_worklet_worker_sharedarraybuffer/

->

cp -r libdf/pkg ../samejs/deepfilternet3/wasm_worker_sharedarraybuffer/pkg/ cp models/DeepFilterNet3_onnx.tar.gz ../samejs/deepfilternet3/wasm_worker_sharedarraybuffer/ cd ../samejs/deepfilternet3/wasm_worker_sharedarraybuffer/

I just started running the demo a min ago and noticed that audio breaks up a bit while getting these messages in console. Anyway, will have a look at the actual code later. Thanks a lot for this work - super cool.

[grazder]

These errors can be caused if model inference is too slow on your computer. Also there is very basic example without handling all the possible problems. There are some hacks and things that you can do.

Can you please check in profiler how fast is model (good if one inference is <8-10 ms) or you can do it with performance.now() around model call

[hlevring]

I am totally not a JS person, and just looked briefly at the code, but I suppose we want to measure execution time of df_process_frame

Tried to profile average frames and frametime during 200 ms

https://cloud.videocom.com/media/fi_01HJ0TSPPDCK6W8Y69RAH67P38

Looks like average time stays under 10ms now, but when I tested before I realize that my power cable was not plugged. If I unplug power I get back the glitches, and it seems the glitches also come back when mouse is busy or other windows get focus.

PS: Just for refence then I am running Win 11/Chrome using a decent high end business laptop. Processor 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz 2.80 GHz Installed RAM 16.0 GB (15.7 GB usable) System type 64-bit operating system, x64-based processor Pen and touch No pen or touch input is available for this display

[hlevring]

Just did another measurement with power cable plugged. This time I measured average process time during 500ms and number of frames> 10ms https://cloud.videocom.com/media/fi_01HJ0WE5Z08ZZMHZ9FWCSX3VF2

Looks like its very sensitive to alt-tab or the like, but as you said yourself then its a raw implementation so I am just impressed that something actually works ;)

[markovivl]

These errors can be caused if model inference is too slow on your computer. Also there is very basic example without handling all the possible problems. There are some hacks and things that you can do.

Thank you very much for the demo. Could you outline some of the hacks that one could implement to improve efficiency? Would this involve modification of the wasm.rs file or rather the javascript part?

[grazder]

Right now I believe that best way to improve efficiency of the model is to work on model speed. Model works slow for older processors, so some hacks with some operation fusing, quantization, different runtimes, different operation implementation, distillation, lowering number of parameters can help.

As for JS part - I can't say right now, I think JS / WASM part is minor here. May be we can work on better using of WASM SIMD, idk (in demo as I remember it turned on)

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.