Need USB audio adapter to speak / listen

[JamesNewton]

Since there is no actual connection to audio on the MicroZed board, an external USB audio adapter is probably the best way to help Dexter find it's voice... and possibly it's ears.

This device was tested: https://www.amazon.com/dp/B01J7P0OGI as it's known to work with Linux without extra drivers, and supports the RasPi, which is our closest popular neighbor. It's also nice that it has an external volume control and mutes for speaker and mic (incase anyone worries about robots listening...)

EDIT: This all-in-one USB sound adapter, amplifier and speakers also works, the only disadvantage being that there is no physical volume control (see notes on amixer below): https://www.amazon.com/HONKYOB-Speaker-Computer-Multimedia-Notebook/dp/B075M7FHM1/ref=asc_df_B075M7FHM1/? $11.99 https://www.adafruit.com/product/3369 $12

After connecting, the light on the USB adapter comes on and a quick look at ls /dev/*audio* shows /dev/audio1 so we have a device installed!

Of course, nothing is that easy (except USB 2 to HDMI adapters, and keyboards) so when we try speaker-test we get: ALSA lib confmisc.c:768:(parse_card) cannot find card '0' which makes sense as the device was audio1 (not zero) and aplay -l had returned:

**** List of PLAYBACK Hardware Devices ****
card 1: Device [USB PnP Audio Device], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

Some googling found: http://forums.debian.net/viewtopic.php?f=7&t=53516 and a quick edit of /etc/modprobe.d/alsa-base.conf to change the last line from: options snd-usb-audio index=-2 to options snd-usb-audio index=0 then a restart, seems to have fixed the issue. speaker-test is able to produce pink noise and sine waves.

DDE core (Job Engine) doesn't seem to know what speak is: But this is probably just an issue with the imports as with other libraries.

Sadly, the full DDE, (electron app) running on Dexter (via XWindows) doesn't seem to have any voices available. window.speechSynthesis.getVoices() returns an empty array.

beep() does work from electron, but again, "beep is not defined" in the Job Engine.

So the hardware is working, but effort is needed in the software area.

[JamesNewton]

It appears Electron has some issues with speech on Linux. https://github.com/noffle/electron-speech/issues/9 Our friend Bret may know something about it. It looks like they are using Google for speech output after all. I had hoped it was being done locally.

[JamesNewton]

There are onboard TTS packages for Linux, including the old standard Festival and Pico TTS.

Pico is quite impressive: https://soundcloud.com/nerduno/sample-pico-tts-recording and has been ported for the RasPi, which is close to our platform. https://github.com/mfurquim/picopi Sadly it must be built from source, as the apt-get failed. Not sure it's worth the time. Also, it really wants to write to a .wav file instead of just streaming sound, but there may be away around that: https://unix.stackexchange.com/questions/325019/pipe-output-from-program-which-only-outputs-to-a-file/325020#325020

I was able to install espeak, apt-get install espeak and it only took up 204K on the SD card. espeak "Hello" produces a robotic, but recognizable output along with a ton of warning / error messages which seem to be about accessing other audio systems. Despite the warnings / errors it works: The sound comes out.

"say" is also available, apt-get install gnustep-gui-runtime but takes 11MB on the drive, so I held off.

And there are node packages for Festival at least: https://github.com/Marak/say.js

[JamesNewton]

To make aplay work by defaut, e.g. without specifying the card, https://stackoverflow.com/questions/39552522/raspberry-pi-aplay-default-sound-card nano ~/.asoundrc to:

pcm.!default {
#        type hw
#        card 0
# on Dexter, we want the default device to be a plug into the hardware
        type plug
        slave {
                pcm "hw:0,0"
        }
}

ctl.!default {
        type hw
        card 0
}

This allows espeak "hello" --stdout | aplay which avoids all the warning messages that espeak "hello" wants to return (still not sure why all those are listed).

Moving speak into dde_init doesn't work because exec needs const {exec] = require('child_process') and that generates "require is not defined". I /think/ that happens because dde_init is not in the node module folder? So next step is to try moving it into the code folder as speak.js or something?

[JamesNewton]

To get the job engine working, we needed to add a special version of "speak" to the core/out.js file just before the requires at the end:

function speak({speak_data = "hello", volume = 1.0, rate = 1.0, pitch = 1.0, lang = "en_US", callback = null}) {
    //var text = stringify_for_speak(speak_data)
    //msg.text   = text
    //msg.volume = volume; // 0 to 1
    //msg.rate   = rate;   // 0.1 to 10
    //msg.pitch  = pitch;  // 0 to 2
    //msg.lang   = lang;
    //msg.onend  = callback
    exec("espeak \"" + speak_data + "\" -a "+ (volume*200) + " -p " + (pitch * 50) + " -s " + (rate * 37 + 130) );
    return speak_data
    }

module.exports.speak = speak

And then in index.js, near the end, we had to import speak from out.js (along with out)

var {out, speak} = require("./out.js")

and make it a global.

global.speak = speak

Now we can do function(){speak({speak_data:"Now moving really really fast", rate:5, volume:0.5, pitch:2})}, in a .dde job.

[cfry]

A more advanced version of speak that is in post dde 3.4.6 releases and also works on DDE is:

function speak({speak_data = "hello", volume = 1.0, rate = 1.0, pitch = 1.0, lang = "en_US", voice = 0, callback = null} = {}){
    if (arguments.length > 0){
        var speak_data = arguments[0] //, volume = 1.0, rate = 1.0, pitch = 1.0, lang = "en_US", voice = 0, callback = null
    }
    var text = stringify_for_speak(speak_data)
    if(window.platform == "node"){
        exec("espeak \"" + speak_data + "\" -a "+ (volume*200) + " -p " + (pitch * 50) + " -s " + (rate * 37 + 130),
             callback );//this callback takes 2 args, an err object and a string of the shell output
                        //of calling the command.
    }
    else {
        var msg = new SpeechSynthesisUtterance();
        //var voices = window.speechSynthesis.getVoices();
        //msg.voice = voices[10]; // Note: some voices don't support altering params
        //msg.voiceURI = 'native';
        msg.text   = text
        msg.volume = volume; // 0 to 1
        msg.rate   = rate;   // 0.1 to 10
        msg.pitch  = pitch;  // 0 to 2
        msg.lang   = lang;
        var voices = window.speechSynthesis.getVoices();
        msg.voice  = voices[voice]; // voice is just an index into the voices array, 0 thru 3
        msg.onend  = callback //this callback takes 1 arg, an event.
        speechSynthesis.speak(msg);
        }
    return speak_data
}

This version also requires the definition of

function stringify_for_speak(value, recursing=false){
    var result
    if ((typeof(value) == "object") && (value !== null) && value.hasOwnProperty("speak_data")){
        if (recursing) {
            dde_error('speak passed an invalid argument that is a literal object<br/>' +
                'that has a property of "speak_data" (normally valid)<br/>' +
                'but whose value itself is a literal object with a "speak_data" property<br/>' +
                'which can cause infinite recursion.')
        }
        else { return stringify_for_speak(value.speak_data, true) }
    }
    else if (typeof(value) == "string") { result = value }
    else if (value === undefined)       { result = "undefined" }
    else if (value instanceof Date){
        var mon   = value.getMonth()
        var day   = value.getDate()
        var year  = value.getFullYear()
        var hours = value.getHours()
        var mins  = value.getMinutes()
        if (mins == 0) { mins = "oclock, exactly" }
        else if(mins < 10) { mins = "oh " + mins }
        result    = month_names[mon] + ", " + day + ", " + year + ", " + hours + ", " + mins
        //don't say seconds because this is speech after all.
    }
    else if (Array.isArray(value)){
        result = ""
        for (var elt of value){
            result += stringify_for_speak(elt) + ", "
        }
    }
    else {
        result = JSON.stringify(value, null, 2)
        if (result == undefined){ //as happens at least for functions
            result = value.toString()
        }
    }
    return result
}

and

window.month_names = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September','October', 'November', 'December']

for speaking dates.

All of the above (except for the month_names) is in the post DDE 3.4.6 version of core/out.js

[JamesNewton]

We can control the volume of the audio output in software with the amixer command:

root@localhost:~# amixer
Simple mixer control 'PCM',0
  Capabilities: pvolume pvolume-joined pswitch pswitch-joined
  Playback channels: Mono
  Limits: Playback 0 - 255
  Mono: Playback 255 [100%] [-127.00dB] [on]
root@localhost:~# amixer scontrols
Simple mixer control 'PCM',0
root@localhost:~# amixer set 'PCM' 80%
Simple mixer control 'PCM',0
  Capabilities: pvolume pvolume-joined pswitch pswitch-joined
  Playback channels: Mono
  Limits: Playback 0 - 255
  Mono: Playback 204 [80%] [-127.20dB] [on]
root@localhost:~#

[JamesNewton]

One issue with this code is that you can end up with multiple child processes running all saying things at the same time, instead of one after the next. To avoid that, a que or stack can be added:

var to_speak = []
function speak({speak_data = "hello", volume = 1.0, rate = 1.0, pitch = 1.0, lang = "en_US", callback = null}) {
    //var text = stringify_for_speak(speak_data)
    //msg.text   = text
    //msg.volume = volume; // 0 to 1
    //msg.rate   = rate;   // 0.1 to 10
    //msg.pitch  = pitch;  // 0 to 2
    //msg.lang   = lang;
    //msg.onend  = callback
    if (speak_data != null) { //new speech text
      let speak_cmd = "espeak \"" + speak_data + "\" -a "+ (volume*200) + " -p " + (pitch * 50) + " -s " + (rate * 37 + 130)
      to_speak.push(speak_cmd)
      }
    else { //finished speaking prior text
      if (to_speak.length > 0) {
        exec(to_speak[0], function(){ to_speak.shift(); speak({speak_data:null}); })
        }
      }
    if (to_speak.length == 1) {
      exec(to_speak[0], function(){ to_speak.shift(); speak({speak_data:null}); })
      }
    return speak_data
    }

Note: This code is missing the improvements made by Fry to handle different data types, and will only correctly speak strings, and will not work on PC DDE as it doesn't check what platform it's running on.

[JamesNewton]

To make it easy to get Dexter to talk, and avoid any possible problems if there is no audio adapter available, use a script that checks that everything is ok. It's easy to a script create after SSHing into Dexter using the nano editor:

cd /srv/samba/share
nano say

and then copy in the code below. In most terminal programs, paste works by right clicking. save the file by pressing Ctrl+w and answering "Y"

#!/bin/sh
if [ -c /dev/audio ]; then
        espeak "$@" --stdout | aplay
else
        echo no audio adapter found
fi

To make the script executable, chmod it:

chmod +x say-ip.sh

and then you can run it by typing:

./say-ip.sh

To have Dexter speak it's IP address, a script can be created as follows:

#!/bin/sh
./say 'Dexter is at I P. address'
hostname -I | sed 's/ \([^\n]\)/, or, \1/g' | ./say

You might call that file 'say-ip' and don't forget to chmod +x say-ip then call it with ./say-ip

This could be added to the end of the RunDexRun startup script.

WARNING: Be very careful about adding any attempt to speak or play any sound to a script that needs to run on startup. If you don't have a sound card plugged in, that will generate an error, and the error may make the script file. E.g. adding those commands to speak the IP address to RunDexRun then unplugging the adapter will cause the robot to fail to start the DexRun firmware, etc... This issue can be avoided by using a bash script called that checks to verify the audio device is connected.

[cfry]

Having this at the end of RunDexRun is a great idea. Requires no learning or other UI. Let's you know your audio is working AND gives you useful information. If there was an easy way for a user to get it to repeat that info, telling them how in the speech would be great.

On Wed, Mar 11, 2020 at 1:43 PM JamesNewton notifications@github.com wrote:

To have Dexter speak it's IP address, a script can be created as follows:

espeak "Dexter is at I P. address " --stdout | aplay hostname -I | sed 's/ ([^\n])/, or, \1/g' | espeak --stdout | aplay

that can be called e.g. say-ip.sh and is easy to create after SSHing into Dexter via the nano editor:

cd /srv/samba/share nano say-ip.sh

and then copy in the above. In most terminal programs, paste works by right clicking. save the file by pressing Ctrl+w and answering "Y"

To make the script executable, chmod it:

chmod +x say-ip.sh

and then you can run it by typing:

./say-ip.sh

and it can be added to the end of the RunDexRun startup script.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/HaddingtonDynamics/Dexter/issues/72#issuecomment-597773296, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJBG7JW7PHRLYDJR2QHJLLRG7ETNANCNFSM4IUBJW5Q .

[JamesNewton]

If (and only if) the audio output device becomes a standard feature on Dexter, THEN... PHUI and other scripts could be updated to support triggering that output. e.g. one of the slots in PHUI could be the "read out your IP address" slot. Probably best if it were the "exit PHUI" slot so that DDE could actually connect after reading it out.

[JamesNewton]

P.S. Bitshift Variations in C-minor works from the Dexter command line if you have this setup: https://github.com/JamesNewton/BitShift-Variations-unrolled