snakers4
commented
1 year ago Hi!
This is definetely an interesting area to cover for v5, we did not really think about it before explicitly!
You see, we viewed VAD as speech / noised speech separation from everything else (silence, mild noise, music).
This poses a quesion of separating speech from extremely noisy backgrounds, if I understand correctly. Or when there always is noise and only sometimes speech.
However, for noise-only signals, I've been getting a consistent 2-3x worse result from v4 w.r.t. v3 Could this be due to v4's encoder shrinking w.r.t. v3's (see number of params below from JIT)? Or should this be more of a training-data issue?
We simply did not optimize for this metric, so it is more or less random. But our data construction prefers mild noise and more or less clean speech. In a nutshell, we simply did not optimize for this scenario.
Did you also observe this behaviour on non-speech-only data?
We observed that for very loud noise our VAD behaves not very well.
Does these numbers make sense at all? Am I doing something wrong? If so, I'd appreciate some directions. The numbers being measured are the sigmoid'ed output of both models' forward method (early returned from get_speech_timestamps() utility), with threshold of 0.5 and window size of 1536 samples.
Yes, this makes sense. There are a lot of gimmicks in the get speech timestamps method to make speech detection more robust. We will try to (i) replicate your metrics (ii) see if applying more of the above method will improve the results (iii) adopt the task long-term.
The good news also is that we got a bit of support for our project, so it will enjoy some attention in the near future with regard to customization, generalization and flexibility.
cassiotbatista
IntendedConsequence
dgoryeo
Jellun
rishikksh20
Hello!
First of all thanks for the VAD model, it is great and really helpful!
I've been doing some experiments with the 16 kHz ONNX models in order to establish a baseline on noisy-speech as well as on non-speech-at-all data. Results on the former for both Ava Speech and LibriParty datasets seem to be in accordance with the quality metrics section of Silero's wiki: v4 is indeed better than v3.
However, for noise-only signals, I've been getting a consistent 2-3x worse result from v4 w.r.t. v3 on ESC-50, UrbanSound8k and FSD50K. This is concerning especially in a always-on scenario (let's say a "wild" one) where the VAD is used a pre-processing front-end to avoid calling a more power-hungry system (which is often the case.)
The following table shows the values for the error rate metric, namely
1-acc, whereaccis sklearn's acuracy_score, so lower means better, and the best are highlighted in bold. The numbers being measured are the sigmoid'ed output of both models' forward method (early returned fromget_speech_timestamps()utility), with threshold of 0.5 and window size of 1536 samples.I'm sharing the same uttids of the files I've been using in my experiments. It is not exactly ready to go because I resegmented and dumped some resampled versions of the datasets to disk, but I believe it should be useful and even reproducible if necessary. The format is
uttid,bos,eos,label, where BOS and EOS are start and end of speech segments. The value -1 in those fields means there's no speech segment at all 😄test_files.tar.gz
My environment:
Finally, some questions:
Thanks!