Bot react on voice commands ?

[reger24]

Hi @thias15 ,

would you accept a pull request for implementing of a voice recognition feature into the android app, so bot can react on voice commands or is that out of scope ?

Describe The idea is, to use the ML example ("TensorFlow Lite Speech Command Recognition Android Demo") which can recognize (out-of-the-box) "yes, no, up, down, left, right, on, off, stop, go" as a starting point. I'd implement it in a way that it can be activated (or is active) in the auto-modes so the user can interfere with the controls by a voice command e.g. "stop" to overwrite/correct bot steering/speeds to perform a emergency stop or to have the bot rotate to find the object for tracking etc...... My preference would be to implement it as service so it can be used (activated) from/for any activity.

[thias15]

This would be a great feature. We actually had something similar planned (text2speech), then match text. But so far didn't get around to it. So your contribution would be very welcome!

[reger24]

OK, then I'll start coding. cu

[izivkov]

@reger24 How do you plan to do it? Would it be part of the main app or would it reside on another device, acting as a controller? I think both are valid options, but if it is separate phone, you don't have to shout at the robot. ;-)

[reger24]

@izivkov My plan is to make it part of the main app (so you would need to shout at the robot) ! Was also thinking about it as feature or replacement for the phonecontroller, but discarded the idea, because my interest is to have a way to influence the robot mainly during auto-modes, not to replace the joystick.

[izivkov]

👍

On Wed, Dec 21, 2022 at 9:48 AM Burkhard @.***> wrote:

@izivkov https://github.com/izivkov My plan is to make it part of the main app (so you would need to shout at the robot) ! Was also thinking about it as feature or replacement for the phonecontroller, but discarded the idea, because my interest is to have a way to influence the robot mainly during auto-modes, not to replace the joystick.

— Reply to this email directly, view it on GitHub https://github.com/isl-org/OpenBot/issues/320#issuecomment-1361422272, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37SOHIM5WY6ZXS5HC43WOMKCPANCNFSM6AAAAAATFTYI74 . You are receiving this because you were mentioned.Message ID: @.***>

[reger24]

fyi: progress can be viewed in clone branch https://github.com/reger24/OpenBot/tree/VoiceRecognition

(as it may take some days and commits before ready for a pull request).

[reger24]

fyi: first testing went not very well, due to the (higher as expected) negative effect of noise level from bot motors. After a start command barely any following voice command is recognized. 've to see if I find a tweak to overcome this hurdle.

[izivkov]

I'm also not sure about the latency. I have no experience with how fast voice commands are interpreted, but I imagine there would be some delay in processing them. Having worked on the various remote controllers for the bot, I know repeated steering commands need to be sent in fast succession for good performance. Maybe one solution is to intercept a voice command and to convert it to multiple finer control inputs to the robot.

But don't let me discourage your development. I think it would be a great feature if we can implement it.

On Tue, Dec 27, 2022 at 11:33 AM Burkhard @.***> wrote:

fyi: first testing went not very well, due to the (higher as expected) negative effect of noise level from bot motors. After a start command barely any following voice command is recognized. 've to see if I find a tweak to overcome this hurdle.

— Reply to this email directly, view it on GitHub https://github.com/isl-org/OpenBot/issues/320#issuecomment-1366031133, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37X3ST54EIW25GREEG3WPMK4HANCNFSM6AAAAAATFTYI74 . You are receiving this because you were mentioned.Message ID: @.***>

[reger24]

@izivkov Thanks for your hint and encouragement, but the issue during first testing was/is the background noise the motors make while driving and the voice recognition apparently too sensitive to background noise/sounds so the recognition itself doesn't work well as it does in quiet environment (with the given neural network). Currently I see still some options to try, 've to see if it improves it enough for an acceptable workaround.

P.S. right now, the recognition delay is from command to sending the control is approx. 1 or maybe 2 sec.

[reger24]

fyi - giving up on unsolved motor noise problem - for a direct robot app voice recognition

The noise from robot's motors is a bigger problem as anticipated, because the microphone is very close to the motors, so that this noise dominates the recording. The motor noise is as loud as the voice if one shouts at the robot from close distance.

I experimented with different noise filters to eliminate/reduce the motor noise without success, as the frequencies overlap with the voice command frequencies, with the result that recording quality is not high enough for recognition.

Tried filters are

• lowpass, highpass, bandpass/-reject filter from marytts.signalproc.filter
• and rnnoise4j library (w/o specific training data) as said without success.

I end up with last option, using voice recognition only if robot doesn't move or via a distant microphone. Both not intended (by me) at this time for the small command set that comes with the used voice recognition sample , so I'll stop working on this feature for now.

[thias15]

Hi @reger24 that's too bad. How about connecting a BT headset to the phone? Then you can just talk into the microphone and will avoid all of these noise issues.

[reger24]

@thias15 Now, as I found the manual of my BT headset how to bring it in pairing mode to connect to robot's phone, I'll give it a try with your suggestion (and see if I can get far enough away from the crying motors ;-)

[reger24]

Hi @thias15, short feedback on your input: first rudimentary testing shows the suggested BT headset workaround could work. at least voice commands where recognized (not all the time but much better as before), so worsewhile to spend a bit more time for further testing and tuning. cu

[thias15]

If you are standing a few meters away and microphone is close to your face it should be fine. Another option is to use a different motor, for example the 520. It makes less noise.