Closed ethanabrooks closed 3 years ago
Try this model: https://talonvoice.com/research/talon-streaming-convnets-1.tar.gz
Also make sure your audio is in the right format. 16-bit 16khz mono
Hmm... even on the original dataset I'm getting a segfault:
❯ sudo docker run --rm -v $HOME:/root/host/ -it --ipc=host --name w2l -a stdin -a stdout -a stderr \
wav2letter/wav2letter:inference-latest sh -c \
"/root/wav2letter/build/inference/inference/examples/multithreaded_streaming_asr_example \
--input_files_base_path /root/host/talon-streaming-convnets-1 \
--input_audio_files /root/host/audio/LibriSpeech/dev-clean/777/126732/777-126732-0070.flac.wav \
--output_files_base_path /root/host/audio/LibriSpeech-dev-clean-transcribed"
Will process 1 files.
Started features model file loading ...
Completed features model file loading elapsed time=3072 microseconds
Started acoustic model file loading ...
Completed acoustic model file loading elapsed time=1372 milliseconds
Started tokens file loading ...
Completed tokens file loading elapsed time=1142 microseconds
Tokens loaded - 9998 tokens
Started decoder options file loading ...
terminate called after throwing an instance of 'std::runtime_error'
what(): failed to open decoder options file=/root/host/talon-streaming-convnets-1/decoder_options.json for reading
Aborted (core dumped)
I notice that the decoder_options.json
file and several others are missing from the talon-streaming-convnets-1/
directory.
❯ tar -xvf talon-streaming-convnets-1.tar.gz
talon-streaming-convnets-1/
talon-streaming-convnets-1/tokens.txt
talon-streaming-convnets-1/feature_extractor.bin
talon-streaming-convnets-1/acoustic_model.bin
Should I just copy them over from the one I was using previously?
Actually mine probably won’t work with that decoder as the tokens won’t match. I dunno then. Try the Test binary.
Are you able to share your test audio file? I can run it locally against my models and let you know if it's the audio or some other problem.
You can hear the audio with this link: https://drive.google.com/file/d/1zQ1lo6vrgkT-M3lLHQkj4NaGZAaunoJi/view?usp=sharing
Thanks!
That link doesn’t appear to be public
Apologies. Let's try this link: https://drive.google.com/file/d/14sbwdVObkSMFSmF9N9FypyG9eVkHPYTl/view?usp=sharing
That file is a bit noisy / quiet and dual speaker, I'm not entirely surprised. But some model should recognize the last half. I'll take a look shortly.
Excellent. Thank you!
On Thu, Mar 5, 2020 at 11:08 AM Ryan Hileman notifications@github.com wrote:
That file is a bit noisy / quiet and dual speaker, I'm not entirely surprised. But some model should recognize the last half. I'll take a look shortly.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/facebookresearch/wav2letter/issues/564?email_source=notifications&email_token=ACO5SJW6FJUF7LUPCYBBKBTRF7FATA5CNFSM4K7GEZC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN53L7I#issuecomment-595310077, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACO5SJWIURPVGDSH5R7VUZ3RF7FATANCNFSM4K7GEZCQ .
This is your problem, as I guessed:
Channels: 2 @ 25-bit
Samplerate: 48000Hz
facebook convnet:
|P|: h h
my convnet:
|P|:
# converted the file as follows:
$ sox out0000000000.wav -r 16000 -c 1 -b 16 out0000000000.flac
facebook convnet:
|P|: thought talking not just a normal voice as when good
but this has an interview with thomas reed brooks conducted on monday august
my convnet:
|P|: talking about just normal of voice imagine you ago
but this is an interview with thomas reed brooks andd conducted on monday august
(This is just Test output, no language model or Decoder was used)
Wow thank you so much. So should I just convert the files and use the instructions I referenced? Or would you recommend a different model? It seems like a language model would be beneficial, no?
Hmmm @lunixbochs, am I doing this correctly? It looks like I'm getting the same blank output as before, even after converting the file.
❯ sudo docker run --rm -v $HOME:/root/host/ -it --ipc=host --name w2l -a stdin -a stdout -a stderr \
wav2letter/wav2letter:inference-latest sh -c \
"/root/wav2letter/build/inference/inference/examples/multithreaded_streaming_asr_example \
--input_files_base_path /root/host/model \
--input_audio_files /root/host/audio/ThomasReedBrooks/segments/out0000000000.flac \
--output_files_base_path /root/host/audio/ThomasReedBrooks-dev-clean-transcribed"
❯ cat ThomasReedBrooks-dev-clean-transcribed/out0000000000.flac.txt
#start (msec), end(msec), transcription
0,1000,
1000,2000,h h h h h h h
2000,3000,h h h h
3000,4000,
4000,5000,
5000,5687,
5687,5687,
❯ sox --i ThomasReedBrooks/segments/out0000000000.flac
Input File : 'ThomasReedBrooks/segments/out0000000000.flac'
Channels : 1
Sample Rate : 16000
Precision : 16-bit
Duration : 00:00:10.01 = 160085 samples ~ 750.398 CDDA sectors
File Size : 182k
Bit Rate : 146k
Sample Encoding: 16-bit FLAC
Comment : 'Comment=Processed by SoX'
Facebook's released models are more research-y, mine are slightly more real world. You can't use the streaming frontend without a language model right now, and I don't have a language model trained on my streaming convnet model.
However if you look at their results, a language model didn't have a very big impact on the WER for streaming convnets: https://github.com/facebookresearch/wav2letter/tree/master/recipes/models/streaming_convnets/librispeech#results
I don't know anything about the streaming frontend. It should work if you use the Test frontend on the whole file at once (which requires the model from the recipe and not the model from the inference wiki)
Hmmm I see. That's interesting. So would I need to use your model in order to avoid the problem I'm having with no output? Was it any of the ones linked here? https://github.com/lunixbochs/wav2letter/tree/master/models
I ran Test with this model (the results I posted labeled facebook convnet): https://github.com/facebookresearch/wav2letter/tree/master/recipes/models/streaming_convnets/librispeech#pre-trained-acoustic-models
I think I might be getting tripped up on the file structure, but whenever I try to run the docker command I get a segfault. I've scanned through the docs and I didn't see instructions about what file structure the program expects for the model. Right now I am doing this:
curl https://dl.fbaipublicfiles.com/wav2letter/streaming_convnets/librispeech/models/am/am_500ms_future_context_dev_other.bin > ~/model/facebook-convnet/acoustic_model.bin
curl https://dl.fbaipublicfiles.com/wav2letter/streaming_convnets/librispeech/models/lm/3-gram.pruned.3e-7.bin.qt > ~/model/facebook-convnet/language_model.bin
curl https://dl.fbaipublicfiles.com/wav2letter/streaming_convnets/librispeech/librispeech-train-all-unigram-10000.tokens > ~/model/facebook-convnet/ltokens.txt
curl https://dl.fbaipublicfiles.com/wav2letter/tds/librispeech/librispeech-train%2Bdev-unigram-10000-nbest10.lexicon > ~/model/facebook-convnet/lexicon.txt
curl https://dl.fbaipublicfiles.com/wav2letter/streaming_convnets/librispeech/am_500ms_future_context.arch > ~/model/facebook-convnet/tds_streaming.arch
then
❯ sudo docker run --rm -v $HOME:/root/host/ -it --ipc=host --name w2l -a stdin -a stdout -a stderr wav2letter/wav2letter:inference-latest sh -c " /root/wav2letter/build/inference/inference/examples/simple_streaming_asr_example --input_files_base_path /root/host/model/facebook-convnet /root/host/audio/ThomasReedBrooks/segments/out0000000000.flac"
Started features model file loading ...
terminate called after throwing an instance of 'std::runtime_error'
what(): failed to open feature file=/root/host/model/facebook-convnet/feature_extractor.bin for reading
Aborted (core dumped)
So I ran
wget http://dl.fbaipublicfiles.com/wav2letter/inference/examples/model/feature_extractor.bin
In the hopes that it would also work for this model. Next I ran
❯ sudo docker run --rm -v $HOME:/root/host/ -it --ipc=host --name w2l -a stdin -a stdout -a stderr wav2letter/wav2letter:inference-latest sh -c " /root/wav2letter/build/inference/inference/examples/simple_streaming_asr_example --input_files_base_path /root/host/model/facebook-convnet /root/host/audio/ThomasReedBrooks/segments/out0000000000.flac"
Started features model file loading ...
Completed features model file loading elapsed time=3135 microseconds
Started acoustic model file loading ...
terminate called after throwing an instance of 'cereal::Exception'
what(): Failed to read 4 bytes from input stream! Read 0
Aborted (core dumped)
Again, am_500ms_future_context_dev_other.bin does not work with the streaming frontend, it is for the Test or Decode binary.
I'm sorry maybe I am getting confused by terminology. I'm not really clear what the "streaming frontend" is. Is there documentation for running the model that you are describing @lunixbochs?
@ethanabrooks: Looks like your input audio file is not in a correct format.
wav2letter
expects the input audio file to be mono-channel, sampled at 16 kHz.
You can convert an audio.mp3
to audio.wav
as follows:
ffmpeg -i audio.mp3 -ar 16000 -ac 1 -ab 256k -f wav audio.wav
, where 256k is the bitrate.
Here's how you would do streaming inference on microphone audio:
The pre-trained models should work on the audio.wav
file.
ffmpeg -hide_banner -loglevel error -f alsa -i default -ar 16000 -ac 1 -ab 256k -f wav - | /root/wav2letter/build/inference/inference/examples/simple_streaming_asr_example --input_files_base_path=/root/host/model/
Hi, I am trying to use these instructions (https://github.com/facebookresearch/wav2letter/wiki/Inference-Run-Examples) to perform ASR on a
.wav
file of my own.I can perform successful transcription on one of the LibriSpeech files with the following command:
Then
However, when I try to run the same command on my audio file (
ThomasReedBrooks/segments/out0000000000.wav
):I get
I am wondering if the audio quality is not good enough on my file, or if the fact that there are multiple voice/accents is confusing the model. You can hear the audio with this link: https://drive.google.com/file/d/1zQ1lo6vrgkT-M3lLHQkj4NaGZAaunoJi/view?usp=sharing