gRPC Error (11, Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time.)

[dsourav]

Describe the bug I am trying to convert speech to text using mic. But if I press the stop listening button without saying anything then the app is crashing. I am trying to use the mic stream example from the GitHub repo.

`class AudioRecognize extends StatefulWidget {
@override
State<StatefulWidget> createState() => _AudioRecognizeState();
 }

class _AudioRecognizeState extends State<AudioRecognize> {
final RecorderStream _recorder = RecorderStream();

bool recognizing = false;
bool recognizeFinished = false;
String text = '';

@override
void initState() {
super.initState();

_recorder.initialize();
}

 void streamingRecognize() async {
await _recorder.start();

setState(() {
  recognizing = true;
});
final serviceAccount = ServiceAccount.fromString(
    '${(await rootBundle.loadString('assets/langoSpeech.json'))}');
final speechToText = SpeechToText.viaServiceAccount(serviceAccount);
final config = _getConfig();

final responseStream = speechToText.streamingRecognize(
    StreamingRecognitionConfig(config: config, interimResults: true),
    _recorder.audioStream);

responseStream.listen((data) {
  if (data.results.isNotEmpty) {
    setState(() {
      text = data.results
          .map((e) => e.alternatives.first.transcript)
          .join('\n');
      recognizeFinished = true;
    });
  } else {}
}, onDone: () {
  setState(() {
    recognizing = false;
  });
},
onError: (){
  print("error");
}
);
 }

void stopRecording() async {
print(_recorder.hashCode);

await _recorder.stop();

setState(() {
  recognizing = false;
});
}

RecognitionConfig _getConfig() => RecognitionConfig(
  encoding: AudioEncoding.LINEAR16,
  model: RecognitionModel.basic,
  enableAutomaticPunctuation: true,
  sampleRateHertz: 16000,
  languageCode: 'en-US');

@override
  Widget build(BuildContext context) {
return Scaffold(
  appBar: AppBar(
    title: Text('Audio File Example'),
  ),
  body: Center(
    child: Column(
      mainAxisAlignment: MainAxisAlignment.spaceAround,
      children: <Widget>[
        if (recognizeFinished)
          _RecognizeContent(
            text: text,
          ),
        RaisedButton(
          onPressed: recognizing ? stopRecording : streamingRecognize,
          child: recognizing
              ? Text('Stop recording')
              : Text('Start Streaming from mic'),
        ),
      ],
    ),
  ), // This trailing comma makes auto-formatting nicer for build methods.
);
 }
}

  class _RecognizeContent extends StatelessWidget {
  final String text;

 const _RecognizeContent({Key key, this.text}) : super(key: key);

 @override
 Widget build(BuildContext context) {
return Padding(
  padding: const EdgeInsets.all(16.0),
  child: Column(
    children: <Widget>[
      Text(
        'The text recognized by the Google Speech Api:',
      ),
      SizedBox(
        height: 16.0,
      ),
      Text(
        text,
        style: Theme.of(context).textTheme.bodyText1,
      ),
    ],
  ),
);
 }
  }`

[josiebhai]

How to stop the stream as soon as the recording is stopped ?

[felixjunghans]

The stream should stop automatically when the audio stream is closed. So the only thing you have to do is to close the passed audio stream when the recording stops.

[josiebhai]

The stream should stop automatically when the audio stream is closed. So the only thing you have to do is to close the passed audio stream when the recording stops.

After the recording stops, it takes approx 10 seconds for the stream to close. It closes with this error.

[VERBOSE-2:ui_dart_state.cc(177)] Unhandled Exception: gRPC Error (code: 11, codeName: OUT_OF_RANGE, message: Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time., details: [])

My problem here is, if i start the recording again, before the stream is closed, im getting the text from the old stream initally, and then it shows the text from the new recording.

Updated Video Link -> https://drive.google.com/file/d/1cMQ41mEypO8ZsMKBqd6d3DoFB2Z7-yvN/view?usp=sharing. See at seconds 17-19, as soon as i click on the record button, i get the old text initially, before i start to talk

[felixjunghans]

Can you give a small code example how you close the audioStream? The error indicates that although the recording was stopped, the audio stream is still open. For this reason empty audio data is sent to Google and the api throws this error after a while.

You have to make sure that the audio stream containing the audio data is really closed when the recording is finished. So that no more audio data is sent to Google.

GoogleSpeech listens to the audio stream and as soon as it is closed, it also closes the request to Google Api.

  _audioStreamSubscription.onDone(() {
      // Close the request stream, if the audio stream is finished.
      request.close();
    });

[josiebhai]

Can you give a small code example how you close the audioStream? The error indicates that although the recording was stopped, the audio stream is still open. For this reason empty audio data is sent to Google and the api throws this error after a while.

I have been using the example code only.

• streamingRecognize function is called when starting the recording.
• stopRecording function is called when the recording is stopped. In this we need to stop the stream.

void streamingRecognize() async {
    setState(() {
      recognizing = true;
      // text = '';
    });
    await _recorder.start();
    final serviceAccount = ServiceAccount.fromString(
        '${(await rootBundle.loadString('assets/service.json'))}');
    final speechToText = SpeechToText.viaServiceAccount(serviceAccount);
    final config = _getConfig();

    responseStream = speechToText.streamingRecognize(
        StreamingRecognitionConfig(config: config, interimResults: true),
        _recorder.audioStream);

    responseStream.listen((data) {
      setState(() {
        text =
            data.results.map((e) => e.alternatives.first.transcript).join(' ');
        recognizeFinished = true;
      });
      updateMessageList(data);
    }, onDone: () {
      print('Stream Closed ' + DateFormat("Hms").format(DateTime.now()));
      setState(() {
        recognizing = false;
      });
    });
  }

  void stopRecording() async {
    await _recorder.stop();
    setState(() {
      recognizing = false;
    });
    print('recorder Stopped ' + DateFormat("Hms").format(DateTime.now()));
  }

  updateMessageList(data) {
    if (recognizing) {
      String text123 =
          data.results.map((e) => e.alternatives.first.transcript).join('\n');

      messsages[0]['message'] = text123;
    }
  }

Here is my output.

Reloaded 7 of 1016 libraries in 473ms.
flutter: recorder Stopped 10:17:43
flutter: What was that?
[VERBOSE-2:ui_dart_state.cc(177)] Unhandled Exception: gRPC Error (code: 11, codeName: OUT_OF_RANGE, message: Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time., details: [])
flutter: Stream Closed 10:17:53

In these logs, Done is called after 10 seconds after recording stopped.

[felixjunghans]

Ok I could recreate the problem. It is not a problem of google_speech but of sound_stream. I'm trying to update the sample code and implement a way to work around the problem.

I will also try to provide an example with Flutter_Sound in the near future, which I think is a better alternative to sound_stream.

[josiebhai]

@felixjunghans , Thank you for clarifying. I was wondering, if there could be some stop or unsubscribe (as in TS) where we can stop listening to that stream.

Yeah Flutter Sound is a good alternative, I am about to try that next.

Thank you.

[felixjunghans]

I have updated the streaming example. It should now work as expected and not produce the error anymore.

@josiebhai Please check if it fits now. If not I can open the issue again.

[josiebhai]

@felixjunghans , I tried the new code, but Im still facing the same error. The previous output is still in the streamingRecognize response and is being fetched in the listen function.

Please see my output logs. I started and stopped recording five times. And the response stream got closed with error after a timeout of 10s after the 5th (final) recording is stopped.

See the Logs

Reloaded 7 of 1087 libraries in 1,287ms.
flutter: Recording Start 17:44:34
flutter: recorder Stopped 17:44:37
flutter: Recording Start 17:44:37
flutter: recorder Stopped 17:44:40
flutter: Recording Start 17:44:41
flutter: recorder Stopped 17:44:43
flutter: Recording Start 17:44:44
flutter: recorder Stopped 17:44:46
flutter: Recording Start 17:44:48
flutter: recorder Stopped 17:44:52
[VERBOSE-2:ui_dart_state.cc(177)] Unhandled Exception: gRPC Error (code: 11, codeName: OUT_OF_RANGE, message: Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time., details: [])
flutter: Stream Closed 17:45:02
[VERBOSE-2:ui_dart_state.cc(177)] Unhandled Exception: gRPC Error (code: 11, codeName: OUT_OF_RANGE, message: Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time., details: [])
flutter: Stream Closed 17:45:02
[VERBOSE-2:ui_dart_state.cc(177)] Unhandled Exception: gRPC Error (code: 11, codeName: OUT_OF_RANGE, message: Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time., details: [])
flutter: Stream Closed 17:45:02
[VERBOSE-2:ui_dart_state.cc(177)] Unhandled Exception: gRPC Error (code: 11, codeName: OUT_OF_RANGE, message: Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time., details: [])
flutter: Stream Closed 17:45:02
[VERBOSE-2:ui_dart_state.cc(177)] Unhandled Exception: gRPC Error (code: 11, codeName: OUT_OF_RANGE, message: Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time., details: [])
flutter: Stream Closed 17:45:02

Here is my source code

final RecorderStream _recorder = RecorderStream();
  StreamSubscription<List<int>> _audioStreamSubscription;
  BehaviorSubject<List<int>> _audioStream;
  bool recognizing = false;
  bool recognizeFinished = false;
  String text = '';

_micPressed() async {
    setState(() {
      isRecording = !isRecording;
    });

    if (isRecording) {
      messsages.insert(0, {"data": 1, "message": "Listening..."});
      //startrecording
      //call Function to convert speech to text and display.
      streamingRecognize();
    } else {
      //stop recording
      stopRecording();
      //call Function to AI Service
      // response(messsages[0]['message']);
    }
  }

  var responseStream;
  void streamingRecognize() async {
    _audioStream = BehaviorSubject<List<int>>();
    _audioStreamSubscription = _recorder.audioStream.listen((event) {
      _audioStream.add(event);
    });

    print('Recording Start ' + DateFormat("Hms").format(DateTime.now()));
    await _recorder.start();
    setState(() {
      recognizing = true;
      // text = '';
    });
    final serviceAccount = ServiceAccount.fromString(
        '${(await rootBundle.loadString('assets/service.json'))}');
    final speechToText = SpeechToText.viaServiceAccount(serviceAccount);
    final config = _getConfig();

    responseStream = speechToText.streamingRecognize(
        StreamingRecognitionConfig(config: config, interimResults: true),
        _recorder.audioStream);

    responseStream.listen((data) {
      setState(() {
        text =
            data.results.map((e) => e.alternatives.first.transcript).join(' ');
        recognizeFinished = true;
      });
      updateMessageList(data);
    }, onDone: () {
      print('Stream Closed ' + DateFormat("Hms").format(DateTime.now()));
      setState(() {
        recognizing = false;
      });
    });
  }

  void stopRecording() async {
    await _recorder.stop();
    await _audioStreamSubscription?.cancel();
    await _audioStream?.close();
    setState(() {
      recognizing = false;
    });
    print('recorder Stopped ' + DateFormat("Hms").format(DateTime.now()));
  }

  updateMessageList(data) {
    if (recognizing) {
      String text123 =
          data.results.map((e) => e.alternatives.first.transcript).join('\n');

      messsages[0]['message'] = text123;
    }
  }

[felixjunghans]

@josiebhai you forget to pass _audioStream to streamingRecognize.

change

responseStream = speechToText.streamingRecognize(
      StreamingRecognitionConfig(config: config, interimResults: true),
      _recorder.audioStream);

to

responseStream = speechToText.streamingRecognize(
      StreamingRecognitionConfig(config: config, interimResults: true),
     _audioStream);

should work.

[josiebhai]

should work. Sorry missed while redoing the code.

@felixjunghans , this works. Great.. Thanks a lot.