Jarvis on Respeaker

[Oliv4945]

Hi,

I finally received my Respeaker: it is a MIPS processor running OpenWrt, so Jarvis will not work out of the box. Others might try the same thing so this thread is to share our experiences.

I created a new branch on my Jarvis fork. What is done for know:

• Detection of OpenWrt distribution
• Use opkg instead of apt-get if OpenWrt is detected.
• Required dependencies added.

Currently jq, mpg123 and whiptail are missing, but I am in the process of cross compiling them. Also Snowboy will probably not be usable, see this issue

Oliv'

[jongerenchaos]

You can use the microphone array with a raspberry or something. The raspberry detect this array as a microphone input.

[alexylem]

@Oliv4945 Ok waiting for your PR for support of OpenWrt. Basically you need to look into jv_install function in utils/dialog_XXX.sh.

[Oliv4945]

@jongerenchaos: I do know but really would like to use it on the Respeaker itself. So I can use the Respeaker+ rPi&mic array+Android app I am developing.

@alexylem: Yes, thank you. I already modified it in this commit. I have cross compiled jq and mpg123, they work ! But still fighting with whiptail package, it seems it has been removed from openWrt in 2015, it should be ok soon.

[Oliv4945]

It works with Bing STT and Google TTS :-) Even if there is lot of work remaining.

@alexylem two questions:

1. Bing api key is set, bing/test.sh works, verbose debug show it right but I have this error (missing key). Do you have any idea of where it can be unset ? I had to force it in main.sh file to make it work.
2. I had to manually cross-compile some packages that are not in OpenWrt repos for the MT7688, then install them locally thanks to opkg install /path/package.ipk. So, for you, what is the best way to handle them ?
   • Add them to the Git repo ?
   • Setup this kind of repository and add it to opkg sources ?
   • Add a zip file containing all packages to a server ?

Thank you

[alexylem]

@Oliv4945

1. absolutely no idea. Maybe search across your repo for bing_speech_api_key. Make sure your keep source-ing these files so that they inherit global variables.

2. I would like to avoid making the repo too big for the majority of people who aren't in OpenWrt. I prefer you keep them on a seperate repo/server. Your 2nd option seems better but you choose.

[Oliv4945]

absolutely no idea.

I did not modified any source-ing... I will be fun to debug :p

Your 2nd option seems better but you choose.

Ok, a zip at first, and maybe a repo after.

[alexylem]

I will be fun to debug

If you don't succeed, commit to your repo I'll have a look

[Oliv4945]

Oops, I forgot to git pull :/ Thank you for your proposition, I will ask if needed

[Oliv4945]

It is working with offline recognition thanks to Respeaker python library (based on PocketSphink) :-) But when hotword is detected, Jarvis enters in command mode, bing speech recognition works, but Jarvis is stucked in command mode. @alexylem , any advice ? Thank you

@ReSpeaker:~/jarvis# ./jarvis.sh -nv

------------ Config ------------
jv_branch            master 
jv_version           17.04.01 
jv_arch              mips 
jv_os_name            
jv_os_version         
language             fr_FR 
play_hw              false 
rec_hw               false 
speaker              Default 
microphone           Default 
recorder             sox 
trigger_stt          respeaker 
command_stt          bing 
tts_engine           google 
--------------------------------

grep: /dev/fd/63: No such file or directory
DEBUG: program_startup hook
Jarvis: Hello
DEBUG: start_speaking hook
DEBUG: stop_speaking hook
User defined commands:
*AIDE*          *BONJOUR*|*SALUT*   *COMMENT*APPELLE*
*MERCI*         *AU REVOIR*|*BYE*   ANNULE*|TERMINE*
ENCORE*         *TEST*          *VERSION*
*REPETE (*) ET (*)  *CA VA*         >*OUI*
>*NON*|*PAS*
Jarvis: Waiting to hear 'Jarvis'
Oliv: (listening...)
DEBUG: Starting Respeaker STT
DEBUG: DIR=/root/jarvis/stt_engines/respeaker
DEBUG: forder=/dev/shm/jarvis-order
Jarvis
DEBUG: entering_cmd hook
Oliv: (listening...)
DEBUG: start_listening hook
utils/timeout.sh 10 rec -V1 -q -r 16000 -c 1 -b 16 -e signed-integer --endian little /dev/shm/jarvis-record.wav gain 10 silence -l 1 0.1 1% 1 0.5 1% pad 0.5 0.5 trim 0 5
DEBUG: speech duration was 27 (10 = 1 sec)
DEBUG: stop_listening hook
DEBUG: token missing or expired
DEBUG: json=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzY29wZSI6Imh0dHBzOi8vc3BlZWNoLnBsYXRmb3JtLmJpbmcuY29tIiwic3Vic2NyaXB0aW9uLWlkIjoiNjYzZmQ4ZGVkZWY3NDhiYWEyYTlmYjk1MjIwNjhmMTciLCJwcm9kdWN0LWlkIjoiQmluZy5TcGVlY2guUHJldmlldyIsImNvZ25pdGl2ZS1zZXJ2aWNlcy1lbmRwb2ludCI6Imh0dHBzOi8vYXBpLmNvZ25pdGl2ZS5taWNyb3NvZnQuY29tL2ludGVybmFsL3YxLjAvIiwiYXp1cmUtcmVzb3VyY2UtaWQiOiIiLCJpc3MiOiJ1cm46bXMuY29nbml0aXZlc2VydmljZXMiLCJhdWQiOiJ1cm46bXMuc3BlZWNoIiwiZXhwIjoxNDkzNTUyODkwfQ.TH5K82NvsfPHfDu_X5itIDhs4cfHT-Zpms6XmGmRlCI
DEBUG: token will expire in 590 seconds
DEBUG: curl https://speech.platform.bing.com/recognize/query?version=3.0&requestid=d694ac41-85f5-419f-93f4-d3a2182dcabd&appid=D4D52672-91D7-4C74-8AD8-42B1D98141A5&format=json&locale=fr-FR&device.os=linux&scenarios=ulm&instanceid=E043E4FE-51EF-4B74-8133-B728C4FEA8AA&result.profanitymarkup=0
DEBUG: json={"version":"3.0","header":{"status":"success","scenario":"ulm","name":"test","lexical":"test","properties":{"requestid":"cb1f89a6-3562-49ac-9c1d-76a36d215d2a","HIGHCONF":"1"}},"results":[{"scenario":"ulm","name":"test","lexical":"test","confidence":"0.8927431","properties":{"HIGHCONF":"1"}}]}
test
$> say "Ca fonctionne!"
Jarvis: Ca fonctionne!
DEBUG: start_speaking hook
DEBUG: stop_speaking hook
Oliv: (listening...)
DEBUG: start_listening hook
utils/timeout.sh 10 rec -V1 -q -r 16000 -c 1 -b 16 -e signed-integer --endian little /dev/shm/jarvis-record.wav gain 10 silence -l 1 0.1 1% 1 0.5 1% pad 0.5 0.5 trim 0 5
DEBUG: speech duration was 13 (10 = 1 sec)
DEBUG: stop_listening hook
DEBUG: curl https://speech.platform.bing.com/recognize/query?version=3.0&requestid=2a6b3795-1e6e-46f0-821f-ca0cbe9358b7&appid=D4D52672-91D7-4C74-8AD8-42B1D98141A5&format=json&locale=fr-FR&device.os=linux&scenarios=ulm&instanceid=E043E4FE-51EF-4B74-8133-B728C4FEA8AA&result.profanitymarkup=0
DEBUG: json={"version":"3.0","header":{"status":"success","scenario":"ulm","name":"test","lexical":"test","properties":{"requestid":"ad85c215-dffd-4195-a374-7db35c314c2e","HIGHCONF":"1"}},"results":[{"scenario":"ulm","name":"test","lexical":"test","confidence":"0.9021909","properties":{"HIGHCONF":"1"}}]}
test
$> say "Ca fonctionne!"
Jarvis: Ca fonctionne!
DEBUG: start_speaking hook
^CDEBUG: stop_speaking hook
Oliv: (listening...)
DEBUG: start_listening hook
utils/timeout.sh 10 rec -V1 -q -r 16000 -c 1 -b 16 -e signed-integer --endian little /dev/shm/jarvis-record.wav gain 10 silence -l 1 0.1 1% 1 0.5 1% pad 0.5 0.5 trim 0 5
d^CDEBUG: stop_listening hook
DEBUG: timeout, end of conversation
DEBUG: listening_timeout hook
^C
DEBUG: program_exit hook

[alexylem]

Aaaah I just realized you added a new STT engine for trigger recognition! brilliant I'm looking forward your PR! Well your pb may not be a real one. By default Jarvis remains in command mode to wait for new commands until "ANNULER" or timeout. You can change this behavior by disabling conversation mode in Settings > General > Conversation mode.

[Oliv4945]

By default Jarvis remains in command mode to wait for new commands until "ANNULER" or timeout.

Oh, shame on me :p

brilliant I'm looking forward your PR!

I am far away from a bug free solution, but I am progressing...

By the way, Respeaker team made their library from this one which you might now, but it can unify most of your STT and add others

[Oliv4945]

Hi @alexylem,

I have created a new branch in order to have a clean integration to Jarvis. Could you please have a look at current diff to see if it does fit your coding style ? If so I will continue in this way, if not please tell me :-)

Thank you

[alexylem]

I think you are on the good path. Please reassure me you are now going to leave a separate version of Jarvis for OpenWRT, I see you have updated the README.md but that's temporary right? Your fork seems to be a bit old though, but no worries I'll help on the integration.

[Oliv4945]

Please reassure me you are now going to leave a separate version of Jarvis for OpenWRT,

I think you wanted to write "not" ? And yes, I do not want to create a separate branch, I am working to end with a PR.

Your fork seems to be a bit old though

Yes, I just merged your beta branch

[alexylem]

I think you wanted to write "not" ? And yes, I do not want to create a separate branch, I am working to end with a PR.

Yes, typo... thanks! PS: you are now officially Collaborator. With great power comes great responsibility 😉

[Oliv4945]

Thank you, it is an honor :) Do you want me to push Respeaker branch into this a branch in this project ?

[alexylem]

Feel free if you can create a branch. Else let me know I do it for you.

[Oliv4945]

Feel free if you can create a branch. Else let me know I do it for you

It should be ok, thank you.

I made huge progresses this evening : no more delays ! The recognition library given by Respeaker team takes 7 seconds to start: it was too long between two triggers. I wrapped it inside an "always on" webserver so Jarvis has just http calls to do. See this commit It is not the cleanest solution, by far, but it works. The next step will be to avoid using Respeaker python library but I think that it will be really harder...

[D-Kalck]

@Oliv4945 It's a bit off topic, but how far the ReSpeaker can recognize your voice ?

[Oliv4945]

@D-Kalck : I did not had time to try it yet, sorry...

In fact the Respeaker core is pretty useless as it is not stable, so I can not rely on it. I stopped my work for know, as my issue opened in May have no answer. I will try the mic array on another computer, but I do not want to make any advertisement for Seed for now...