Thalhammer
commented
3 years ago Webassembly is definitly something I would have looked into as well in the future. It honestly does not surprise me a lot that it did not work out of the box, because when reversing you have to deal with the result the compiler optimized on certain assumptions about the target, many of which probably don't hold in the browser.
There are a couple of things I found:
- Your fork is 4 commits behind. I did a lot of refactoring/bugfixes in those, so it would be good if you could retry it with the most uptodate code.
- Instead of using the microphone, record a short wave file and feed that in. In addition to making it easier to test, it also allows for repeatable testing (if it works once with a given wave it should always produce the same result). Ideally check if the wave gets correctly detected on the local code.
- I only skimmed the audio code, but I noticed that you set AudioGain to 5. Thats almost certainly wrong. An audiogain of 5 means the audiosignal is multiplied by 5 before processing, so if it works with such a high gain that probably means there is an issue somewhere else that just gets mitigated such a high gain.
In order to make it build I had to comment out several self_assert statements that assume a 64-bit architecture
Thats a result of the way how I reverse the library. I had to make sure the layout of my code matches the layout used during compilation of the orriginal code, while I mixed new and original code. However this is no longer the case and if you update to the latest commit you will notice they are gone.
and I found an issue stating that only 32bit ARM is supported.
I am not sure why they decided to not support 32bit x86, but I assume it was mostly a decision to cut down on support work, given 32bit x86 is effectively dead. I dont think there is anything in the code thats inherently 64 bit only. That being said: The current state of the code sometimes cast a pointer to a number and adds to it, which depends on the size of the pointed to type. I dont think I came across any of those being 64bit, but that might be a culprint. I have never compiled the code for 32bit yet, but I certainly intend on supporting it in the future.
I assume thus that the WebAssembly build might not work due to some math assuming 64 bits?
Most of the math is floating point maths anyway (mostly 32bit float, but in some parts double).
The steps to go further in this direction would probably be
- compiling the code and unit tests in 32bit mode and making sure they produce the same results.
- Actually implementing a test using live microphone input (or pieces of a wave) cause it might well be that detection works fine if the recording is supplied in one junk but not if its streamed (e.g. because I screwed up buffering somewhere).
Steps for debugging are probably dropping debug outputs in the pipeline and comparing the output between browser and "normal" build. I also need to build more unit tests cause I am like 99% sure there lurk many missed bugs in the code. TBH I never worked with wasm (apart from reading&likeing it and coding the obligatory hello world) so I dont really know what would the typical way of debugging it.
I assume to get it running in the browser you also need to crosscompile lapack & blas libraries. Snowboy only uses cblas in some special cases, all of which are contained in the Matrix&Vector classes. I intend to add discreet implementations of them as a build option, making the dependency to blas optional. The default would still use them, but disabling it would probably save lots of executable size which is good for both embedded and browser.
I also hate that snowboy/snowman does IO directly to disk in many places (Input&Output classes) deep inside the library, which makes it effectively impossible to use without filesystem support and prevents embedding the ressources&model inside the executable. I dont have a good plan how to change that yet without completely breaking API compatibility.
My guess would be that something inside the vad is broken. If it assumes the input audio is all silence it skips everything afterwards to save performance. But then again, there was a pretty major bug in Nnet, so maybe upgrading to the latest version already fixes it.
Its nice to see people actually likeing it and intending to use it. Lets hope it gets in a usable state soon.
Sincerely, Thalhammer
ccoreilly
Arnaudv6
sveinbjornt
Good morning! First of all thank you very much for the efforts you are putting in reverse engineering and open sourcing the original snowboy library, those are impressive skills!
I have wanted a snowboy WASM build running in the browser for a while and now with the source code it seems it might be possible. I have made an attempt you can find in my fork of the project but although it builds and runs detection on the audio data sent to it, it always returns silence (-2).
I put up a demo you can try out here (it unfortunately only works with Chrome as Firefox does not support retrieving user audio at 16000 Hz and resampling in the browser would add another step where things might go wrong, I would like to focus on getting it to work now)
In order to make it build I had to comment out several
self_assertstatements that assume a 64-bit architecture and I found an issue stating that only 32bit ARM is supported.I assume thus that the WebAssembly build might not work due to some math assuming 64 bits? I am a bit "überfragt" :) so I was hoping you could shine some light on the issue or changes needed to make it work.
Thanks!