search

drachtio / drachtio-freeswitch-modules

A collection of open-sourced freeswitch modules that I use in various drachtio applications
MIT License
174 stars 119 forks source link

Dialogflow service cannot handle audio more than 60s. Google transcribe cannot handle audio more than 305s #7

Open prady77 opened 5 years ago

prady77 commented 5 years ago

Hello Dave

Amazing work !!!

I am encountering issue of 60s exceed for DF and 305s for speech. They ask to refresh the channel. Any ideas on how to refresh the stream every 60s or 305s?

"Reason-Code":11,"Reason-Msg":"Exceeded maximum allowed stream duration of 305 seconds." GRPC status details.

davehorton commented 5 years ago

could you turn freeswitch log levels to debug and recreate, then send me the (entire) log? Feel free to either send me a link to a private gist or email it to me directly

viktorsperl commented 5 years ago

This also happend to me. Only these three lines are relevant in the log:

2019-08-12 15:47:59.633164 [ERR] google_glue.cpp:165 grpc_read_thread: error Exceeded maximum allowed stream duration of 305 seconds. (11)
2019-08-12 15:47:59.633164 [DEBUG] google_glue.cpp:221 grpc_read_thread: got 0 responses
2019-08-12 15:47:59.633164 [DEBUG] google_glue.cpp:231 grpc_read_thread: finish() status Exceeded maximum allowed stream duration of 305 seconds. (11)
davehorton commented 5 years ago

OK, just wondering about the scenario where this happens. Was there literally 5 minutes with no spoken audio? Or was it five minutes with some sort of speech input, but it just wasn't recognized as an intent?

viktorsperl commented 5 years ago

There was continuous speech on the call, and detection returned several transcripts correctly. However only on one channel of the two, the other one was completely silent. Maybe I should set GOOGLE_SPEECH_SEPARATE_RECOGNITION_PER_CHANNEL on this call?

davehorton commented 5 years ago

OK, just to confirm: you are using mod_google_transcribe, correct?

davehorton commented 5 years ago

actually, how are the two channels arranged? Are they bridged together, and you want to transcribe each? are you calling uuid_google_transcribe on each separately?

Looking at the code, it doesn't look like I implemented support for sending two channels at once to google (and transcribing each separately) so perhaps I should implement that.

But I would like to understand how you are trying to transcribe the two channels in your app.

viktorsperl commented 5 years ago

Two channels are bridged, yes. One has no mic, so no sound. uuid_google_transcribe is started only on the channel uuid which has incoming speech.

davehorton commented 5 years ago

ok, then I am confused. The channel that has speech -- the one you are transcribing -- was returning transcripts correctly? Or it was for a bit, and then eventually it went 305 seconds without returning a transcript?

Assuming the latter, I think all I could (should) do is to return a timeout event to your application. Do you agree?

viktorsperl commented 5 years ago

The channel that has speech -- the one you are transcribing -- was returning transcripts correctly? Or it was for a bit, and then eventually it went 305 seconds without returning a transcript?

It did returned transcripts correctly, but after 305 seconds, it stopped.

A bridged call has 2 channels (uuid-s), each channel (uuid) has 2 audio streams (sent and received). As I mentioned earlier, the uuid_google_transcribe was only started on one channel, but that still means 2 audio streams. One of these contained speech, but the other one was mute. Could this be the issue?

davehorton commented 5 years ago

no, if you called uuid_google_transcribe on a single channel, then it would be sending only the received audio on that channel to google speech.

Are you saying it returned several transcripts correctly, THEN there was a pause of 305 seconds, followed by that error? Or are you saying that 305 seconds from calling uuid_google_transcribe that error was returned, even though several transcripts were returned during that 305 seconds?

viktorsperl commented 5 years ago

no, if you called uuid_google_transcribe on a single channel, then it would be sending only the received audio on that channel to google speech.

Ok ,thanks, good to know.

Are you saying it returned several transcripts correctly, THEN there was a pause of 305 seconds, followed by that error? Or are you saying that 305 seconds from calling uuid_google_transcribe that error was returned, even though several transcripts were returned during that 305 seconds?

There was no pause. I was talking, it returned the transcriptions almost continuously. It stopped 305 seconds from uuid_google_transcribe start.

davehorton commented 5 years ago

and you got a final transcription during that time ?

I'd really like to see the entire freeswitch log (debug level) from the time of the call to uuid_google_transcribe start to the error

viktorsperl commented 5 years ago

Here you go. Timestamp difference is exactly 305s. call.log

davehorton commented 5 years ago

ok, thanks that was helpful. It looks like google simply limits the length of a long-running recognize operation. Perhaps I just need to respond to this error under the covers by starting another one, or else send an event to the application so that it can restart it. Let me think about that.

By the way, note the GOOGLE_SPEECH_SINGLE_UTTERANCE channel variable option -- not sure of your use case, but if you are doing a command and response type of use case, you may want to use this. If you are just wanting to transcribe a complete phone call or something, then the current way you are doing it is better.

prady77 commented 5 years ago

Actually it is Google limitation (GSR), they recommend to refresh the grpc channel every 5 mins. i.e destroy and recreate every 5 mins.

davehorton commented 5 years ago

OK, I (finally) have a partial fix for this -- I've addressed the issue with using google speech to text where transcriptions end after 305 seconds.

There is some work on the application side to handle this -- see the updated google_transcribe example to see a working example.

FIrst, of all, I have added a new freeswitch event (google_transcribe::max_duration_exceeded) that the application will receive when google terminates a transcription due to the 305 second limit.

The application is responsible for handling that event and restarting the transcription, as shown here.

This requires a build of mod_google_transcribe with the commit referenced above.

The similar dialogflow issue is still outstanding (probably needs a similar fix), but @prady77 if possible please test and let me know if this resolves your issue.