Audio back not working

[JorgMuskens]

Hi Dan, The latest updates in the ARI bridge work just fine. The next thing i am running into is the audio stream back from google. It looks like the follwing code is not fired

// if we got the output audio then send it back to asterisk (streamingDetectIntent) if (data.outputAudio && data.outputAudio.length !== 0) { this._sendAudioToAsterisk(data);

      // we got the audio, so now we need to restart the dialogflow stream
      this._createNewDialogFlowStream();
    }

Either my intent in dialogflow is false or something is perhaps missing Hope you can help

[danjenkins]

Ah thats an odd one because I know that code is definitely right as it works in my demo. your best bet is to console.log(data) at the very top of that function where we get the data back from dialogflow. in particular setups of dialogflow the field in the data is different but that shouldn't be the case here :)

[JorgMuskens]

Yeah i know and i am not sure if it is dialogflow. The transcript is workign just fine! But the data received from Dialogflow does not hold outputAudio. I can not seem to find the issue

[JorgMuskens]

Hi Dan This is the data what i get back.

{"level":30,"time":1583143802608,"pid":3187,"hostname":"tron-asterisk","name":"Dialogflow-AudioServer","id":"1583143799.88","msg":"got data from dialogflow {\"alternativeQueryResults\":[],\"responseId\":\"\",\"recognitionResult\":{\"speechWordInfo\":[],\"messageType\":\"TRANSCRIPT\",\"transcript\":\"hallo\",\"isFinal\":false,\"confidence\":0,\"stability\":0.009999999776482582,\"speechEndOffset\":{\"seconds\":\"2\",\"nanos\":40000000}},\"queryResult\":null,\"webhookStatus\":null,\"outputAudio\":{\"type\":\"Buffer\",\"data\":[]},\"outputAudioConfig\":null}","v":1}

If i test it directly in the api in dialog flow the raw json looks different

{ "responseId": "60b9036b-9813-46f7-93a0-8c967eff9e8d-19db3199", "queryResult": { "queryText": "hallo", "action": "input.welcome", "parameters": {}, "allRequiredParamsPresent": true, "fulfillmentText": "Hallo!", "fulfillmentMessages": [ { "text": { "text": [ "Hallo!" ] } } ], "intent": { "name": "projects/.../agent/intents/2d561d5a-9f3a-4347-9cc2-6b64ccd487e0", "displayName": "Default Welcome Intent" }, "intentDetectionConfidence": 1, "languageCode": "nl" }, "outputAudio": "UklGRiB5AABXQVZFZm10IBAAAAABAAEAwF0AAIC7AAACABAAZGF0...", "outputAudioConfig": { "audioEncoding": "OUTPUT_AUDIO_ENCODING_LINEAR_16", "synthesizeSpeechConfig": { "speakingRate": 1, "voice": {} } } }

[JorgMuskens]

I got the audio back now but only static is played. Issue was indeed in Dialoglfow itself. Now i only need to figure out why i hear static and not the spoken text

[danjenkins]

static to asterisk?

I've had an issue where the file from dialogflow contains 1 channel of audio and 1 channel of static, but asterisk played it back fine.

What version of asterisk are you running?

[JorgMuskens]

Asterisk 17.2.0 I am unsure if it is related to my config in asterisk i am going to reinstall the complete server.

[JorgMuskens]

Could it be an encoding issue? As the outputAudio object is full of decimal chars instead of normal ones like above. Also the fs saves static noise while testing

[danjenkins]

If you can, try running asterisk 16.6.2 - that's what I've been running. And even though the audio contains that static noise , it plays back fine for me. I'm going to raise an issue with dialogflow about the static as it shouldn't be being returned by them

[JorgMuskens]

Hi Dan, I think Asterisk 17.2.0 is working just fine. The reason i am stating this is that i copied the json object as soon as it arrives to a wav file. When i play this file also static is heared so i think your code works just fine as i hear the static also on the phone. So i also think it must be google which is producing the static. note that i am running the dutch language. I will try to use english and see if that makes any difference. WIll come back on this

[danjenkins]

I get static back too (from dialogflow) but in Asterisk 16.6.2 I don't hear it in Asterisk. I'll test 17 next week!

[danjenkins]

@JorgMuskens - I've just made a copy of my asterisk server to upgrade it to 17 later tonight and I'll run a test and get some pcaps/file outputs of what I get back from dialogflow and report back!

[JorgMuskens]

@JorgMuskens - I've just made a copy of my asterisk server to upgrade it to 17 later tonight and I'll run a test and get some pcaps/file outputs of what I get back from dialogflow and report back!

Superb!

[danjenkins]

Hey @JorgMuskens I just tested with 17.2 and i get no static..... what are you sending back to Asterisk? slin16 8k ?

[danjenkins]

Enquiry to Dialogflow about the static issue - https://github.com/googleapis/nodejs-dialogflow/issues/559

But I'm not getting static on my install of asterisk. What environments are you running Asterisk and the RTP Server on @JorgMuskens ?

[JorgMuskens]

Hi Dan, This is what i am using right now. Everything is running locally on the below server

ict@tron-asterisk:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

ict@tron-asterisk:~$ sudo asterisk -rvvvvv
Asterisk 17.2.0, Copyright (C) 1999 - 2018, Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Running as user 'asterisk'
Running under group 'asterisk'
Connected to Asterisk 17.2.0 currently running on tron-asterisk (pid = 1122)

[JorgMuskens]

@danjenkins A little step but i now the issue with the noise. The audio back from dialogflow is encoded base64. This is what i tried and got results with

_sendAudioToAsterisk(dialogFlowData) {
    this.log.info('Got audio to play back from dialogflow');
    this.log.info('DATA:',dialogFlowData.outputAudio.toString('base64'));
    util.promisify(fs.writeFile)(`${this.id}.ogg`, Buffer.from(dialogFlowData.outputAudio.toString('base64').replace('data:audio/ogg; codecs=opus;base64,', ''), 'base64'));

When i copy the ogg flie to windows and play in media player it works. So we need to change the code to first decode the base64 audio data before handling it through the rest of the code. I will see if i can more results

[danjenkins]

Oh were you asking for opus/ogg data from dialogflow?

Can you output the log taht contains the audio data? It should hold the outputAudioConfig.

[JorgMuskens]

util.promisify(fs.writeFile)(`${this.id}.wav`, Buffer.from(dialogFlowData.outputAudio.toString('base64').replace('data:audio/wav; codecs=pcm;base64,', ''), 'base64'));

Just changed it to wav and also works. So we need to look at base64 I will work on this further

[danjenkins]

So this is what I dont understand. Everything works fine for me on 17.2 with slin16 8k back to asterisk.

[JorgMuskens]

Dialogflow issue then perhaps? as soon as i encode the audio with base64 run it to decoding process i get speech

[danjenkins]

Yeah Im wondering if youre being given back ogg/opus data etc

I'm pretty sure if you look at the message that contains the audio you'll be told what theyre giving you

[JorgMuskens]

No that was just a test in my code to save a audio with ogg encoding as i was not trusting PCM. I get wav back form dialogflow but it is encoded with base64. So i have to decode first to get the speech working before it is send to Asterisk.

[JorgMuskens]

Can you use this in your code and download the file and play it. This will save a wav file

 _sendAudioToAsterisk(dialogFlowData) {
    this.log.info('Got audio to play back from dialogflow');
    this.log.info('DATA:',dialogFlowData.outputAudio.toString('base64'));
    //util.promisify(fs.writeFile)(`${this.id}.ogg`, Buffer.from(dialogFlowData.outputAudio.toString('base64').replace('data:audio/ogg; codecs=opus;base64,', ''), 'base64'));
    util.promisify(fs.writeFile)(`${this.id}.wav`, Buffer.from(dialogFlowData.outputAudio.toString('base64').replace('data:audio/wav; codecs=pcm;base64,', ''), 'base64'));

[danjenkins]

@JorgMuskens I'll check this out later today at some point - thanks!

if you then send the output of that to asterisk does it play back ok?

[JorgMuskens]

@danjenkins you know what i was on the wrong branch for DialogflowConnector.js for some reason the code was of the branch jared. Not sure how this happend.

[danjenkins]

Oh i should just delete that branch - it was completely designed for debugging purposes

Does that mean you get back audio into Asterisk fine with the master branch?

[JorgMuskens]

Yes it is working now you can close

[danjenkins]

Great news!!!

[SubhashPeshwa]

Hi folks,

Apologies for commenting on a closed thread, but I'm facing the issue with the static sound being returned back from dialogflow too. I've been trying to debug this from the last few days but to no avail!

Some context:-

1. Asterisk version: 16.6.2 running on FreePBX on a Virtualbox instance (Bridged not NAT)
2. The audioserver running on host
3. Speech enabled - 16 bit linear, not MP3 or OGG Opus

On dialing the extension linked to the stasis, the stasis event is fired, the rtp-audio server is invoked, dialogflow sends a response as well! But there's no audio response back to the X-Lite SIP phone.

I thought I'd isolate the issue one at a time and started with the rt-audioserver. When the response file is stored as a wav, it's just static. I even tried uploading the sln file to aserisk and playing it as part of the dialplan, still static.

I have a sneaking suspicion that even though it's static sound when saved as a file, it's not really the root cause of the audio not being played back to SIP phone. I should at least get back some static noise then.

I'm really out of ideas and hence the post here. Apologies in advance if this turns out to be something silly.

Any pointers would be massively appreciated!

Here are some log files:-

ari-logfile.txt

df-reesponse.txt

Asterisk Config:-

exten => 1002,1,Verbose(1, "") same => n,Stasis(dialogflow) same => n,Hangup()

1002 is a pjsip extension

[danjenkins]

hey @SubhashPeshwa can you just make a new issue for me please? Otherwise I won't see this when I come in to look and try and help you :)

(when i get some time to indeed help)

[SubhashPeshwa]

@danjenkins Will do!

@JorgMuskens 's solution of decoding to base64 first before saving works for me! I'm able to get the audio response back from dialogflow and save it as a wav file which is no longer just static. Still not able to get back the audio response to the SIP phone though :(

Let me give it one more day of troubleshooting before I trouble you (Now that I know dialogflow is indeed sending the right response)