Kitt-AI / snowboy

Future versions with model training module will be maintained through a forked version here: https://github.com/seasalt-ai/snowboy
Other
3.11k stars 1k forks source link

How to specify recording device with python? #118

Open duoduo999 opened 7 years ago

duoduo999 commented 7 years ago

Hi,

I'm playing with Matrix Creator and snowboy. MC has a 8 mic array which maps to 9 input devices (the 9th is a synthesized one):

arecord -L ... mic_channel0 mic_channel1 mic_channel2 mic_channel3 mic_channel4 mic_channel5 mic_channel6 mic_channel7 mic_channel8

I can record with arecord with the "--device mic_channelN" argument, however I don't know how to pass in this information when creating snowboy detector (in python3). If I run the python demo directly it detects nothing.

Any help is greatly appreciated. Thanks!

duoduo999 commented 7 years ago

Btw, I'm running it on raspberry Pi 3 with latest raspbian.

chenguoguo commented 7 years ago

We haven't played with Matrix Creator, but let's try to sort it out. We will appreciate your feedback as well.

I assume the synthesized is the one that after beamforming (or anything related to microphone array processing), if that is the case, then Snowboy should only use that synthesized channel. What you have to do is to specify the device when start PyAudio.

You have to modify here: https://github.com/Kitt-AI/snowboy/blob/master/examples/Python/snowboydecoder.py#L47

Here is how you can get the device ID of your microphones: http://stackoverflow.com/questions/36894315/how-to-select-a-specific-input-device-with-pyaudio

Then you can set input_device_index when you open the PyAudio stream.

Note that Snowboy requires a sampling rate of 16000, not sure if that is supported by Matrix Creator. If not, you may have to capture the audio at whatever sampling rate that Matrix Creator supports, and then convert it to 16000 sampling rate, and finally feed the audio to Snowboy.

Let us know if this works.

duoduo999 commented 7 years ago

Thanks for the prompt response! Yes, you're right the synthesized channel is for beamforming.

Actually after messing around with alsa configuration I seem to have made the MC input device as default device, however now stuck with a problem with PyAudio. Either with snowboydemo or with standalone simple PyAudio program, I got crash like:

ALSA lib pcm_file.c:358:(snd_pcm_file_write_bytes) write failed: Bad file descriptor ALSA lib pcm_file.c:358:(snd_pcm_file_write_bytes) write failed: Bad file descriptor ALSA lib pcm_file.c:358:(snd_pcm_file_write_bytes) write failed: Bad file descriptor python3: pcm_file.c:397: snd_pcm_file_add_frames: Assertion `file->wbuf_used_bytes < file->wbuf_size_bytes' failed.

So this issue doesn't seem to have anything to do with snowboy. Since arecord works fine, I suspect it to be a compatibility issue with pyaudio/portaudio for MC. I've reported it to MC forum but haven't got any help there yet.

The next thing I plan to try is to spawn arecord process and pipe its stdout to snowboy's engine, so as to remove pyaudio dependency. Do you see that to be a feasible solution?

chenguoguo commented 7 years ago

Yes as long as you can pipe the desired audio (16000 sampling rate, signal channel, 16 bit signed integer) to Snowboy it should work.

duoduo999 commented 7 years ago

Got it working with arecord piping :). Thanks a lot for helping!

chenguoguo commented 7 years ago

Awesome! BTW, if you think it's worth doing, you are could add an example to examples/Python/ and show people how to use arecord pipe instead of PyAudio. This should help a lot for those who are struggling with PortAudio on Linux systems.

duoduo999 commented 7 years ago

It'll be my pleasure. I'll send a PR today or tomorrow.

shinselrobots commented 7 years ago

I am looking at hooking Snowboy to Houndify. I'd like to use arecord pipe for this, so duoduo999, please do post your code!

chenguoguo commented 7 years ago

@shinselrobots it's here: https://github.com/Kitt-AI/snowboy/pull/120

@xuchen is reviewing it and we will merge it soon.

evancohen commented 7 years ago

@shinselrobots I'm looking to add Houndify to sonus if you want to lend a hand, otherwise I'm going to be taking a stab at it next week.

samwel89 commented 7 years ago

Hi @duoduo999

I have the Matrix Creator and I want to use it with snowboy for a hotword detection. I had the same error as yours python: pcm_file.c:397: snd_pcm_file_add_frames: Assertion file->wbuf_used_bytes < file->wbuf_size_bytes' failed. I've modified my snowboydecoder.py by using arecord piping as you suggested above but I've always the same error. Am I missing something?

I need your help or suggestions please.

samwel89 commented 7 years ago

Hi @chenguoguo

If I understand well, in order to use the Matrix Creator with snowboy I have to use arecord piping (snowboydecoder_arecord.py) in a place of snowboydecoder.py? Is it that only to change or I'm missing something?

Thank you.

chenguoguo commented 7 years ago

If I understand correctly yes, see the PR here: https://github.com/Kitt-AI/snowboy/pull/120

On Sun, Apr 2, 2017 at 11:33 PM, samwel89 notifications@github.com wrote:

Hi @chenguoguo https://github.com/chenguoguo

If I understand well, in order to use the Matrix Creator with snowboy I have to use arecord piping (snowboydecoder_arecord.py) in a place of snowboydecoder.py? Is it that only to change or I'm missing something?

Thank you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Kitt-AI/snowboy/issues/118#issuecomment-291059433, or mute the thread https://github.com/notifications/unsubscribe-auth/ALPfk4oTZqKLJUoFulrVbzuNu7FigpZuks5rsJLXgaJpZM4L45TT .

samwel89 commented 7 years ago

Ok, Yes I have seen that PR and I've modified my snowboydecoder.py accordingly but I still I have the same error "Snowboy recognition failed" with this error code python: pcm_file.c:397: snd_pcm_file_add_frames: Assertion file->wbuf_used_bytes < file->wbuf_size_bytes' failed.

In my opinion, snowboy doesn't see my mic array as an input device (mic_channel8) when it starts the pyaudio/portaudio (or arecord pipe)

samwel89 commented 7 years ago

Because In my case I couldn't record with mic array until I modified my asoundrc file, capture.pcm{ type plug slave.pcm "hw:1,0" } I changed "hw:1,0" and fix it to "mic_channel8". Is there any effect to snowboy? (mic_channel8 is the beamforming of the 8 mic arrays)

duoduo999 commented 7 years ago

Hi,

Have you tried recording with arecord directly? The changes I made in snowboydetector_arecord simply uses arecord command to do the recording and consumes frames through pipe, so as long as arecord works, snowboy should too.

samwel89 commented 7 years ago

Hello @duoduo999,

Thank you for responding.

Yes, I can record directly with arecord it works fine, the problem comes when I want to use snowboy to detect the hotword (I'm using jarvis voice bot).

I cannot record when there is "hw:1,0" in my asoundrcfile, I had to put the mic directly mic_channel8. Is there a line in snowboydecoder.py to detect the input device (in my case mic_channel8) ?

And Here

The changes I made in snowboydetector_arecord

I hope you meant snowboydecoder_arecord. And did you change anything else in other snowboy files example snowboydetect....? Thank you.

duoduo999 commented 7 years ago

Hi @samwel89,

Yes, I meant snowboydecoder_arecord.py. Sorry that I don't have the MatrixCreator environment anymore so can't experiment anything :( now. It's weird that you still get the same assertion failure, because my understanding is that it's from PortAudio code which PyAudio is based on; however using arecord should have completely removed dependency to PortAudio (my understanding is arecord is directly based on Alsa, but I'm not Linux audio expert). I would suggest that maybe you reinspect your code and maybe add tracing to make sure it's actually using arecord PIPE instead of PyAudio ...

Btw, if .asoundrc is causing problem, arecord allows you to specify device from commandline with -D switch (please double check the manual). Maybe you can see if that works if you modify the command line for launching arecord in snowboydecoder_arecord.py?

Thanks, Yiming

samwel89 commented 7 years ago

Hi @duoduo999

Thank you for your help and suggestions. I can confirm that arecord pipe worked to remove that error of assertion, because I tested MC and snowboy directly by using demo.py and snowboydecoder_arecord.pyIt worked, Snowboy could detect my hotword and play the ding sound.

However, I've noticed that I still have that error when I use it in my project because there is a file in my project which still use pyaudio. I've modified it with arecord pipe but it still gives me some errors.

Thanks.

samwel89 commented 7 years ago

Hello @duoduo999

I can confirm again that your PR worked for me... :) I could solve the code errors I had and now it works. No more that Assertion failure.

Thank you for your help.

duoduo999 commented 7 years ago

Wonderful. Glad that I could help.

EranIDar commented 6 years ago

Hello @chenguoguo and @duoduo999,

First thank you for your snowboydecoder_arecord script, I'am using a matrix creator and I'm having a lot of issues with Pyaudio and Alsa.

But I have an other issue now, after adding snowboydecoder_arecord.py to my snowboy folder, I modified demo.py :

`import snowboydecoder_arecord import sys import signal

interrupted = False

def signal_handler(signal, frame): global interrupted interrupted = True

def interrupt_callback(): global interrupted return interrupted

if len(sys.argv) == 1: print("Error: need to specify model name") print("Usage: python demo.py your.model") sys.exit(-1)

model = sys.argv[1]

signal.signal(signal.SIGINT, signal_handler)

detector = snowboydecoder_arecord.HotwordDetector(model, sensitivity=0.5) print('Listening... Press Ctrl+C to exit')

detector.start(detected_callback=snowboydecoder_arecord.play_audio_file, interrupt_check=interrupt_callback, sleep_time=0.03)

detector.terminate()`

After launching python demo.py resources/alexa.umdl i get this error : Listening... Press Ctrl+C to exit Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/pi/Documents/rpi-arm-raspbian-8.0-1.1.1/snowboydecoder.py", line 97, in record_proc wav = wave.open(process.stdout, 'rb') File "/usr/lib/python2.7/wave.py", line 511, in open return Wave_read(f) File "/usr/lib/python2.7/wave.py", line 164, in __init__ self.initfp(f) File "/usr/lib/python2.7/wave.py", line 129, in initfp self._file = Chunk(file, bigendian = 0) File "/usr/lib/python2.7/chunk.py", line 63, in __init__ raise EOFError EOFError

Do you have any idea where the issue come from ?

Thank you for your response.

chenguoguo commented 6 years ago

@EranIDar Did you see the same error when you run: https://github.com/Kitt-AI/snowboy/blob/master/examples/Python/demo_arecord.py ?