[HELP]: Handle audio bytes in memory in real time

[agbossi]

Context

I'm building an app that measures things in voice signal using flutter sound. I haven't had any issues for mobile, using pcm16 codec i can get microphone data to a stream from which i can work with bytes. However I can't do the same for web because pcm16 is not fully supported on browsers, and flutter sound says that to work with streams i specifically need pcm16.

---

Here is my question :

Is it possible to get the same result i achieved on mobile? Is there any workaround to the codec problem?

[Larpoux]

Hi @agbossi ,

I am very glad that you was able to handle a PCM16 streams on mobiles. Things was not finished, and since I do not work anymore on Flutter Sound, Flutter Sound is not in a great shape .

I suggest that you have a look on something that I think is terrific : Web Audio API.

This API is incredible, and I wanted to port this API on mobiles. Unfortunately as I told you I do not work anymore on Flutter Sound.

There is an official DART/Flutter implementation of this API. I think you will be really happy with this API. But of course you will complain that this API is not compatible on mobiles Flutter.

[Larpoux]

Hi @agbossi ,

Nobody has worked/improved Flutter Sound for almost one year. I am sad with that. Unfortunately. I do not work anymore with Dart/Flutter.

You can have a look to this RFC that explain what was my project. I am really convinced that porting Flutter Sound Streams on Web is really not the good way to do things. I really suggest the opposite : porting Web Audio API on iOS and Android.

Actually you do not have many solution : Flutter Sound Stream are actually not ported on web and Web Audi API is not implemented on Mobiles . If you need a good compatibility WEB/Mobiles you can have two approach :

• Try to emulate Flutter Sound Mobile on Web using Web Audi API.
• Try to implement Web Audi API on Flutter Mobile

The later is really the good way to look to this problem. But needs a major development.

Please tell us if you do not find the dependency to use Web Audi API on Flutter Web.

This module is official and you should not have any difficulty to find it

[Larpoux]

Here is the dart Web Audio API.

Implement this API on mobiles is certainly the king solution but is a major work.

a simpler approach: Because you already have something running on mobiles using Flutter Sound and because you seem happy with that, I suggest that you port your app on web, using Web Audio API. This API is so powerful that you will probably not have major difficulties.

You will have the mobile app using Flutter sound, and a web app using Web Audio API. Not perfect having 2 different versions, but probably the only practical solution

[agbossi]

Thank you so much for your reply, I really needed the guidance. I will go for the easy solution since i already have something working and this is just a part of my work. I did came across web audio api on another forum but I couldn't find how to use it or any examples/tutorials at all. I don't have much experience with flutter/dart so the api docs is not enough for me to start working. Hope it's not asking too much, but do you have any dart usage example of Web Audio Api or any tutorial resource? Again, thanks for your time and insight.

[Larpoux]

No, I haveno experience. And there is no example, no documentation for flutter/dart

The module is just a simple interface between Dart and JavaScript. If you study the module, you will see that it is a very simple wrapper for Dart. You will find many good documentation for JavaScrip and many examples. Just use the Dart API instead of JS API

[bljubisic]

@agbossi Can you share how did you handle the audio bytes in memory using flutter_sound? Currently I am trying to do the real time word transcription using Whisper AI model and flutter_sound, but have difficulty converting the PCM16 used by microphone output in flutter_sound to PCM32 that is needed by whisper, can you help?

[agbossi]

I did by hand, this is not exactly the code, but here are the most relevant parts

late StreamController<Food> _audioStream;
_audioRecorder = FlutterSoundRecorder();

await _audioRecorder!.startRecorder(
        toStream: _audioStream.sink,
        codec: Codec.pcm16,
        numChannels: 1,
        sampleRate: _sampleRate);

_audioStream.stream.listen((data) {
    FoodData dataStream = data as FoodData;
    ByteData bd = getBytes(dataFromDevice);

    if (bd.lengthInBytes == 0) return Int16List(0);
    int audioLen = bd.lengthInBytes ~/ 2;
    int audioIdx = 0;
    Int16List audio = Int16List(audioLen);

    for (int i = 0; i < bd.lengthInBytes; i += 2) {
      audio[audioIdx++] = bd.getInt16(i, Endian.little);
    }

    // do something with audio
})

[bljubisic]

Thanks :)

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Leave a comment or this will be closed in 7 days.