[feature request] can we support realtime recording & playback ?

[imiskolee]

can we support realtime recording & playback ？

i want to realtime get voice stream and apply some filters, then realtime play.

[hyochan]

@imiskolee Could you kindly describe the feature in detail? This would help us understanding and drive to our future roadmap.

[DreamsInCS]

@hyochan My request is rather similar to @imiskolee. Is there some way to grab bytes of audio data from a file that is in the middle of being recorded, in order to send these snippets to a server? At the same time, can these snippets then be played back live to another mobile user while the recording is still happening?

I would understand if there might be a slight amount of delay between the "send" and "receive" parts. Thank you for your response! :)

[Larpoux]

This feature will be very useful if we want to apply our own codec during recording. iOS and Android codecs are very limited by the native OS codecs.

[uniquejava]

I'd like to do some real-time speech to text stuff, also need this kind of streaming capability/API. Because these stt cloud service often expects some real-time blob/bytes data.

[Larpoux]

I plan to work soon on the flutter_sound recorder. I will look what we can do in this area. I agree that it will be very great if we can stream the recording data during the recording. Speech to text, or custom encoding, or ...

Probably we will have to use lower iOS and android tools than those actually used by flutter_sound But I would really like to do that. I will tell you what is possible when I will work again on flutter_sound ( in two weeks, or so).

[Larpoux]

Flutter Sound V5.0.0 will allow Recording Raw PCM to a Dart Stream. This version is supposed to be released sometimes next week.

Is this what you are expected from Flutter Sound ? Do you also need to playback from a Dart Stream ?

[imiskolee]

@hyochan sorry for too later, i leaved flutter stuff now, the user story is:

1. Choose a voice filter
2. click button Record
3. playing filtered voice from MIC.

[dhalloop7]

As V5.1.0 is available is it now recording with PCM bytes and returns the stream of bytes and play the same bytes as well?

[ZZYSonny]

I think having a frequency filter will be helpful. So we can record human voice while getting rid of (mostly) background noise.

[imiskolee]

@ZZYSonny yes, it's also based on voice steaming data.

[Larpoux]

I am currently working on the V6.0 . I hope to be able to do streaming on ios (actually i am able to do that on Android, but still not on ios). Please be patient, and stay tuned.

[Larpoux]

V 6.0 is released. With this version we can do recording PCM-LINEAR-16 to a Dart Stream and playback from a Dart Stream.

[jtkeyva]

@Larpoux great stuff! Any idea if a Speech To Text example will be available anytime soon?

[Larpoux]

Yes, such an example would be really useful. This is one of many tasks that I would like to achieve. But if someone else does it, I will be very grateful.

[jtkeyva]

i just smashed the 2 example projects from speech_to_text & flutter_sound_web together and it proves that recording works on iOS

` /*

• Copyright 2018, 2019, 2020 Dooboolab.
• This file is part of Flutter-Sound.
• Flutter-Sound is free software: you can redistribute it and/or modify
• it under the terms of the GNU Lesser General Public License version 3 (LGPL-V3), as published by
• the Free Software Foundation.
• Flutter-Sound is distributed in the hope that it will be useful,
• but WITHOUT ANY WARRANTY; without even the implied warranty of
• MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
• GNU General Public License for more details.
• You should have received a copy of the GNU Lesser General Public License
• along with Flutter-Sound. If not, see https://www.gnu.org/licenses/. */

import 'package:flutter/material.dart'; import 'package:flutter_sound/flutter_sound.dart'; import 'dart:async'; import 'dart:io'; import 'package:path_provider/path_provider.dart'; import 'dart:typed_data' show Uint8List;

// stt import 'dart:async'; import 'dart:math'; import 'package:speech_to_text/speech_recognition_error.dart'; import 'package:speech_to_text/speech_recognition_result.dart'; import 'package:speech_to_text/speech_to_text.dart';

/*

• This is an example showing how to record to a Dart Stream.
• It writes all the recorded data from a Stream to a File, which is completely stupid:
• if an App wants to record something to a File, it must not use Streams.
• The real interest of recording to a Stream is for example to feed a
• Speech-to-Text engine, or for processing the Live data in Dart in real time.

• */

const int SAMPLE_RATE = 8000; typedef fn();

/// Example app. class RecordToStreamExample extends StatefulWidget { @override _RecordToStreamExampleState createState() => _RecordToStreamExampleState(); }

class _RecordToStreamExampleState extends State {

FlutterSoundPlayer _mPlayer = FlutterSoundPlayer(); FlutterSoundRecorder _mRecorder = FlutterSoundRecorder(); bool _mPlayerIsInited = false; bool _mRecorderIsInited = false; bool _mplaybackReady = false; String _mPath; StreamSubscription _mRecordingDataSubscription;

// STT bool _hasSpeech = false; double level = 0.0; double minSoundLevel = 50000; double maxSoundLevel = -50000; String lastWords = ""; String lastError = ""; String lastStatus = ""; String _currentLocaleId = ""; List _localeNames = []; final SpeechToText speech = SpeechToText();

@override void initState() { super.initState(); // Be careful : openAudioSession return a Future. // Do not access your FlutterSoundPlayer or FlutterSoundRecorder before the completion of the Future _mPlayer.openAudioSession().then((value){ setState( (){_mPlayerIsInited = true;} );} ); _mRecorder.openAudioSession().then((value){ setState( (){_mRecorderIsInited = true;} );} ); }

@override void dispose() { stopPlayer(); _mPlayer.closeAudioSession(); _mPlayer = null;

  stopRecorder();
  _mRecorder.closeAudioSession();
  _mRecorder = null;
super.dispose();

}

// stt Future initSpeechState() async { bool hasSpeech = await speech.initialize( onError: errorListener, onStatus: statusListener); if (hasSpeech) { _localeNames = await speech.locales();

  var systemLocale = await speech.systemLocale();
  _currentLocaleId = systemLocale.localeId;
}

if (!mounted) return;

setState(() {
  _hasSpeech = hasSpeech;
});

}

Future createFile() async { Directory tempDir = await getTemporaryDirectory(); _mPath = '${tempDir.path}/flutter_sound_example.pcm'; File outputFile = File(_mPath); if (outputFile.existsSync()) await outputFile.delete(); return outputFile.openWrite(); }

// ---------------------- Here is the code to record to a Stream ------------

Future record() async { assert (_mRecorderIsInited && _mPlayer.isStopped); IOSink sink = await createFile(); StreamController recordingDataController = StreamController(); _mRecordingDataSubscription = recordingDataController.stream.listen ((Food buffer) { if (buffer is FoodData) sink.add(buffer.data); } ); await _mRecorder.startRecorder( toStream: recordingDataController.sink, codec: Codec.pcm16, numChannels: 1, sampleRate: SAMPLE_RATE, ); setState(() {}); } // --------------------- (it was very simple, wasn't it ?) -------------------

Future stopRecorder() async { await _mRecorder.stopRecorder(); if (_mRecordingDataSubscription != null) { await _mRecordingDataSubscription.cancel(); _mRecordingDataSubscription = null; } _mplaybackReady = true; }

fn getRecorderFn() { if (!_mRecorderIsInited || !_mPlayer.isStopped) return null; return _mRecorder.isStopped ? record : (){stopRecorder().then((value) => setState((){}));};

}

void play() async { assert (_mPlayerIsInited && _mplaybackReady && _mRecorder.isStopped && _mPlayer.isStopped); await _mPlayer.startPlayer(fromURI: _mPath, sampleRate: SAMPLE_RATE, codec: Codec.pcm16, numChannels: 1,whenFinished: (){setState((){});}); // The readability of Dart is very special :-( setState(() {}); }

Future stopPlayer() async { await _mPlayer.stopPlayer(); }

fn getPlaybackFn() { if (!_mPlayerIsInited || !_mplaybackReady || !_mRecorder.isStopped) return null; return _mPlayer.isStopped ? play : (){stopPlayer().then((value) => setState((){}));}; }

// ----------------------------------------------------------------------------------------------------------------------

@override Widget build(BuildContext context) {

Widget makeBody()
{
  return Column( children:[
    Container
      (
      margin: const EdgeInsets.all( 3 ),
      padding: const EdgeInsets.all( 3 ),
      height: 80,
      width: double.infinity,
      alignment: Alignment.center,
      decoration: BoxDecoration
        (
        color:  Color( 0xFFFAF0E6 ),
        border: Border.all( color: Colors.indigo, width: 3, ),
      ),
      child: Row(
          children: [
            RaisedButton(onPressed:  
            getRecorderFn(),

            color: Colors.white, disabledColor: Colors.grey, child: Text(_mRecorder.isRecording ? 'Stop' : 'Record'), ),
            SizedBox(width: 20,),
            Text(_mRecorder.isRecording ? 'Recording in progress' : 'Recorder is stopped'),
          ]
      ),
    ),

    Container
      (
      margin: const EdgeInsets.all( 3 ),
      padding: const EdgeInsets.all( 3 ),
      height: 80,
      width: double.infinity,
      alignment: Alignment.center,
      decoration: BoxDecoration
      (
        color:  Color( 0xFFFAF0E6 ),
        border: Border.all( color: Colors.indigo, width: 3, ),
      ),
      child: Row(
          children: [
            RaisedButton(onPressed: getPlaybackFn(), color: Colors.white, disabledColor: Colors.grey, child: Text(_mPlayer.isPlaying ? 'Stop' : 'Play'), ),
            SizedBox(width: 20,),
            Text(_mPlayer.isPlaying ? 'Playback in progress' : 'Player is stopped'),
          ]
      ),
    ),

    Container(
        child: Column(
          children: <Widget>[
            Row(
              mainAxisAlignment: MainAxisAlignment.spaceAround,
              children: <Widget>[
                FlatButton(
                  child: Text('Initialize'),
                  onPressed: _hasSpeech ? null : initSpeechState,
                ),
              ],
            ),
            Row(
              mainAxisAlignment: MainAxisAlignment.spaceAround,
              children: <Widget>[
                FlatButton(
                  child: Text('Start'),
                  onPressed: !_hasSpeech || speech.isListening
                      ? null
                      : startListening,
                ),
                FlatButton(
                  child: Text('Stop'),
                  onPressed: speech.isListening ? stopListening : null,
                ),
                FlatButton(
                  child: Text('Cancel'),
                  onPressed: speech.isListening ? cancelListening : null,
                ),
              ],
            ),
            Row(
              mainAxisAlignment: MainAxisAlignment.spaceAround,
              children: <Widget>[
                DropdownButton(
                  onChanged: (selectedVal) => _switchLang(selectedVal),
                  value: _currentLocaleId,
                  items: _localeNames
                      .map(
                        (localeName) => DropdownMenuItem(
                          value: localeName.localeId,
                          child: Text(localeName.name),
                        ),
                      )
                      .toList(),
                ),
              ],
            )
          ],
        ),
      ),
      Expanded(
        flex: 4,
        child: Column(
          children: <Widget>[
            Center(
              child: Text(
                'Recognized Words',
                style: TextStyle(fontSize: 22.0),
              ),
            ),
            Expanded(
              child: Stack(
                children: <Widget>[
                  Container(
                    color: Theme.of(context).selectedRowColor,
                    child: Center(
                      child: Text(
                        lastWords,
                        textAlign: TextAlign.center,
                      ),
                    ),
                  ),
                  Positioned.fill(
                    bottom: 10,
                    child: Align(
                      alignment: Alignment.bottomCenter,
                      child: Container(
                        width: 40,
                        height: 40,
                        alignment: Alignment.center,
                        decoration: BoxDecoration(
                          boxShadow: [
                            BoxShadow(
                                blurRadius: .26,
                                spreadRadius: level * 1.5,
                                color: Colors.black.withOpacity(.05))
                          ],
                          color: Colors.white,
                          borderRadius:
                              BorderRadius.all(Radius.circular(50)),
                        ),
                        child: IconButton(icon: Icon(Icons.mic)),
                      ),
                    ),
                  ),
                ],
              ),
            ),
          ],
        ),
      ),
      Expanded(
        flex: 1,
        child: Column(
          children: <Widget>[
            Center(
              child: Text(
                'Error Status',
                style: TextStyle(fontSize: 22.0),
              ),
            ),
            Center(
              child: Text(lastError),
            ),
          ],
        ),
      ),
      Container(
        padding: EdgeInsets.symmetric(vertical: 20),
        color: Theme.of(context).backgroundColor,
        child: Center(
          child: speech.isListening
              ? Text(
                  "I'm listening...",
                  style: TextStyle(fontWeight: FontWeight.bold),
                )
              : Text(
                  'Not listening',
                  style: TextStyle(fontWeight: FontWeight.bold),
                ),
        ),
      ),

  ],
  );
}

return Scaffold(backgroundColor: Colors.blue,
  appBar: AppBar(
    title: const Text('Record to Stream ex.'),
  ),
  body: makeBody(),
);

}

// tts

void startListening() { lastWords = ""; lastError = ""; speech.listen( onResult: resultListener, listenFor: Duration(seconds: 60), localeId: _currentLocaleId, onSoundLevelChange: soundLevelListener, cancelOnError: true, listenMode: ListenMode.confirmation); setState(() {}); }

void stopListening() { speech.stop(); setState(() { level = 0.0; }); }

void cancelListening() { speech.cancel(); setState(() { level = 0.0; }); }

void resultListener(SpeechRecognitionResult result) { setState(() { lastWords = "${result.recognizedWords} - ${result.finalResult}"; }); }

void soundLevelListener(double level) { minSoundLevel = min(minSoundLevel, level); maxSoundLevel = max(maxSoundLevel, level); // print("sound level $level: $minSoundLevel - $maxSoundLevel "); setState(() { this.level = level; }); }

void errorListener(SpeechRecognitionError error) { // print("Received error status: $error, listening: ${speech.isListening}"); setState(() { lastError = "${error.errorMsg} - ${error.permanent}"; }); }

void statusListener(String status) { // print( // "Received listener status: $status, listening: ${speech.isListening}"); setState(() { lastStatus = "$status"; }); }

_switchLang(selectedVal) { setState(() { _currentLocaleId = selectedVal; }); print(selectedVal); }

}

`

[jtkeyva]

@Larpoux

[Larpoux]

Thank you so much JT 🥇 I will insert that in the next Flutter Sound version. I really appreciate your contribution 💯

[jtkeyva]

Well, I just literally copied the code from one example to the other so I didn't really do anything. It's really crude but proves it works.

[Larpoux]

Copy and Paste instead of reinventing the wheel is the B-A-BA for our job. Thank you again

[jtkeyva]

Sure thing, thank YOU for the great package!

[Larpoux]

Hi @jtkeyva .

Your example is now in Release 6.3. Thank you for your contribution.

[jtkeyva]

Great! Hope it helps people make cool stuff 👍

Oh, I messed up my comment on line 249 was supposed to say STT (not TTS) but pretty irrelevant ha