TJbot is thinking - but not talking back to me

[EsthervanLuit]

Dear all,

My TJbot hears what I say, but does not respond to me, either in the terminal or on audio. No errors are given either. The output looks as follows:

pi@TJbot:~/tjbot/recipes/conversation $ sudo node conversation.js
info: Hello from TJBot! My name is TJ.
You can ask me to introduce myself or tell you a joke.
Try saying, "TJ, please introduce yourself" or "TJ, who are you?"
You can also say, "TJ, tell me a joke!"
info: TJBot heard: could 
info: TJBot heard: I 
info: TJBot heard: TJ tell me a joke 
info: TJBot heard: but 
{ filename: '/tmp/tjbot117825-9216-l4te4b.skrq',
  gain: 100,
  debug: true,
  player: 'aplay',
  device: 'plughw:0,0' }

At first I thought it was an audio problem, as I had found out through aplay that plughw needed to be 2,0 rather than 0,0. I have bought a standard ADAfruit TJbot package that contains a USB speaker and USB microphone. Testing both devices separately, they work fine. You can find the output from aplay and arecord below:

Aplay

pi@TJbot:~/tjbot/recipes/conversation $ aplay -l
** List of PLAYBACK Hardware Devices **
card 0: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7
card 0: ALSA [bcm2835 ALSA], device 1: bcm2835 ALSA [bcm2835 IEC958/HDMI]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 2: Device_1 [USB2.0 Device], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

Arecord

pi@TJbot:~/tjbot/recipes/conversation $ arecord -l
** List of CAPTURE Hardware Devices **
card 1: Device [USB PnP Sound Device], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

I tried changing the alsa.conf file, and when that didn't work I made sure the raspberry pi itself recognized my audio devices and set the USB speaker to default, I created an asound.conf file and I even adapted the conversation.js and sst.js files. See changes below:

Alsa.conf

defaults.ctl.card 2
defaults.pcm.card 2
defaults.pcm.device 0
defaults.pcm.subdevice -1
defaults.pcm.nonblock 1
defaults.pcm.compat 0

Asound.conf

GNU nano 2.2.6                       File: asound.conf                                                     

pcm.!default {
        type hw
        card 2
}

ctl.!default {
        type hw
        card 2
}

Conversation.js & sst.js

// set up TJBot's configuration
var tjConfig = {
    verboseLogging: true,
    speakerDeviceId: 'plughw:2,0',
    microphoneDeviceId: 'plughw:1,0',
        robot: {
                name: 'TJ',
                homophones: ['DJ','Deejay','Teejay'],
                gender: 'female'
    }
};

I followed the tutorial of instructables when setting up TJ and also looked at the troubleshooting section:

If you see TJBot's response on the terminal but don't hear TJBot talking, there is a good chance that one of these two things has happened: (1) The audio output is directed to a wrong channel (you can fix it from raspi-config), (2) your sound modules are blocked. In that case, go to /etc/modprobe.d/ and remove blacklist-rgb-led.conf

Then run the following command: sudo update-initramfs -u Reboot and confirm the "snd" modules are running by executing the command "lsmod". This should solve the problem. lsmod

The first option does not seem to apply to me since I have a USB speaker, no audio jack or HDMI. For the second option, I don't even have the file blacklist-rgb-led.conf. Since I tried most audio-related troubleshooting, I'm wondering whether it is something with TJ just not responding, but as far as I'm aware I have installed the Conversation, Speech-to-Text and Text-to-Speech correctly and my credentials have been correctly inserted. Can you please help me?

[victordibia]

Hi @EsthervanLuit ,

I do agree with you that its an audio problem (the app uses deviceid 0,0, but should use 2,0). With the tjbot library, you can specify the speakerDeviceId .. and in your case, this should be set to 2,0. I would say we update tjConfig to

var tjConfig = {
    verboseLogging: true,
    speakerDeviceId: 'plughw:2,0',
    microphoneDeviceId: 'plughw:1,0',
        robot: {
                name: 'TJ',
                homophones: ['DJ','Deejay','Teejay'],
                gender: 'female'
    },
speak: {
    language: 'en-US', // see TJBot.prototype.languages.speak
    voice: undefined, // use a specific voice; if undefined, a voice is chosen based on robot.gender and speak.language
    speakerDeviceId: "plughw:2,0" // plugged-in USB card 1, device 0; see aplay -l for a list of playback devices
  }
};

For the full list of parameters you can configure with the TJBot library, please see here. (If the above does not work, one final thing you might have to do is roll back changes to you might have made to alsa.conf and asound.conf)

-V.

[EsthervanLuit]

Hi @victordibia,

I just made the changes using your suggestions and rolling back asound.conf and alsa.conf and they were successful. TJbot would finally talk to me!

However, only one run (so it was gone after reboot) did I get a reply at normal speed. All the other times the latency is really long (30secs+) or TJ doesn't bother to answer at all. I have tried several networks (ethernet cable with shared wifi on laptop, wifi directly, tethering from my mobile phone), but it does not seem to make any change. Is there maybe another setting that's off? I'm in the UK and my Bluemix account is also registered to the UK area, so that can't be it. My wifi download speed is 37Mb/s and my upload speed 44Mb/s, so that can't be it either. It's a shame - I was so elated I finally got it to work.

[victordibia]

HI @EsthervanLuit ,

Glad you were able to get it to work. Based on my experience, latency with a good internet connection usually is in the milliseconds range (300 - 800ms) for transcripts (at least here in the US). Rebooting the bot should have absolutely no effect on speed. Just to verify, you can try out the stt and tts demos to verify if response is speedy or lagged.

[mimoraes]

The problem here is that the TJ isn't listening correctly. However, works perfect on translator.google.com ..

[mimoraes]

@victordibia TJBot isn't speaking. But the speaker is working fine, playing audios and youtube.

arecord -l

List of CAPTURE Hardware Devices card 1: Device [USB PnP Sound Device], device 0: USB Audio [USB Audio] Subdevices: 1/1 Subdevice #0: subdevice #0

aplay -L -l

/Desktop/tjbot/recipes/conversation $ aplay -l List of PLAYBACK Hardware Devices /Desktop/tjbot/recipes/conversation $ aplay -L null Discard all samples (playback) or generate zero samples (capture) default

conversation.js

var tjConfig = { debugLogging: true, speakerDevideId: 'plughw:1,0', microphoneDeviceId: 'plughw:1,0', robot: { name: 'Rose', gender: 'female' }, speak: { language: 'en-US', voice: undefined, speakerDeviceId: 'plughw:1,0' } };

stt.js:

var tjConfig = { debugLogging: true, speakerDevideId: 'plughw:1,0', microphoneDeviceId: 'plughw:1,0', robot: { name: 'Rose', gender: 'female' }, speak: { language: 'en-US', voice: undefined, speakerDeviceId: 'plughw:1,0' } };

command

pi@Rose:/Desktop/tjbot/recipes/conversation $ node conversation.js info: Hello from TJBot! My name is Rose. You can ask me to introduce myself or tell you a joke. Try saying, "Rose, please introduce yourself" or "Rose, who are you?" You can also say, "Rose, tell me a joke!" info: TJBot heard: Joe info: TJBot heard: tell me I don't info: TJBot heard: tell me a
error: the speech_to_text service returned an error. Error: Session timed out. at emitError (/home/pi/Desktop/tjbot/recipes/conversation/node_modules/watson-developer-cloud/speech-to-text/recognize_stream.js:145:13) at W3CWebSocket.socket.onmessage (/home/pi/Desktop/tjbot/recipes/conversation/node_modules/watson-developer-cloud/speech-to-text/recognize_stream.js:165:7) at W3CWebSocket._dispatchEvent [as dispatchEvent] (/home/pi/Desktop/tjbot/recipes/conversation/node_modules/yaeti/lib/EventTarget.js:107:17) at W3CWebSocket.onMessage (/home/pi/Desktop/tjbot/recipes/conversation/node_modules/websocket/lib/W3CWebSocket.js:234:14) at WebSocketConnection. (/home/pi/Desktop/tjbot/recipes/conversation/node_modules/websocket/lib/W3CWebSocket.js:205:19) at emitOne (events.js:116:13) at WebSocketConnection.emit (events.js:211:7) at WebSocketConnection.processFrame (/home/pi/Desktop/tjbot/recipes/conversation/node_modules/websocket/lib/WebSocketConnection.js:552:26) at /home/pi/Desktop/tjbot/recipes/conversation/node_modules/websocket/lib/WebSocketConnection.js:321:40 at _combinedTickCallback (internal/process/next_tick.js:131:7) at process._tickCallback (internal/process/next_tick.js:180:9)

[asyka]

@mimoraes - were you able to solve the problem that you see in the command window above? I get the same problem either of time out or payload limit, and so far was not able to find a solution.

error: the speech_to_text service returned an error. Error: Session timed out. at emitError (/home/pi/Desktop/tjbot/recipes/conversation/node_modules/watson-developer-cloud/speech-to-text/recognize_stream.js ....

[mimoraes]

worked here https://github.com/ibmtjbot/tjbot/issues/69