Buffering in real time while playing MIDI file?

[wsievern]

Hello!

With the help of this repository, @postacik's port of the MIDI file support, and raw_sound, I have MIDI files being played back in my flutter app. Thank you all (and @sinshu) for putting this together!

I have run into an issue in my use case however: I am trying to playback MIDI files with no perceptible latency, and give the user the ability to adjust the speed of the MIDI file in real time. Rendering a 4-5 minute MIDI track takes on the order of 20 seconds and adjusting the playback speed of the sequencer isn't possible after playback has started (with the example setup @sinshu provides).

I am currently trying to find a solution to these concerns, but this is my first audio application and I fear I might be barking up the wrong tree(s). I'm looking into implementing a circular buffer that the synthesizer continuously renders audio into while the sequencer plays the MIDI file. Does this seem feasible? Any recommendations or guidance would be appreciated!

All the Best and thanks again,

Will

[chipweinberger]

In my app, I render the notes "on demand".

here are the steps:

• decide how much to buffer (i.e 20 milliseconds)
• in a loop:
  • check how many samples are still in the buffer (💩)
  • render until buffer is full again

Unfortunately the step with the poop emoji is not possible with raw_sound.

In my app, I estimate it using timestamps, but that's not ideal.

IMO you should add a function to raw_sound to get the number of queued samples. It won't be a perfect number because by the time the number reaches Dart it will be out of date, but we can just increase the buffer size to compensate.

[wsievern]

I will look into this. Thanks!

[rodydavis]

Would love to see if it is possible to have an alternative to raw_sound that uses FFI rust audio library instead.

Currently for my app, i layer the buffers on top of each other and play them as keys are pressed.

import 'package:dart_melty_soundfont/dart_melty_soundfont.dart';
import 'package:flutter/material.dart';
import 'package:flutter/services.dart' show rootBundle;
import 'package:raw_sound/raw_sound_player.dart';

import '../widget/piano_deck.dart';

class Home extends StatefulWidget {
  const Home({super.key});

  @override
  State<Home> createState() => _HomeState();
}

class _HomeState extends State<Home> {
  Synthesizer? synth;
  static int bufferSize = 4096 << 4;
  static int nChannels = 1;
  static int sampleRate = 16000;

  final _players = <int, List<RawSoundPlayer>>{};
  double get keyWidth => 80 + (80 * _widthRatio);
  double _widthRatio = 0.0;
  bool _showLabels = true;

  @override
  void initState() {
    super.initState();
    rootBundle.load('assets/sounds/Piano.sf2').then((bytes) async {
      final settings = SynthesizerSettings(
        sampleRate: 44100,
        blockSize: 64,
        maximumPolyphony: 64,
        enableReverbAndChorus: false,
      );
      if (mounted) {
        setState(() {
          synth = Synthesizer.loadByteData(bytes, settings);
        });
      }
    });
  }

  Future<void> play(int midi) async {
    synth!.reset();
    synth!.noteOn(
      channel: 0,
      key: midi,
      velocity: 120,
    );
    final current = _players[midi] ??= [];
    RawSoundPlayer player = RawSoundPlayer();
    await player.initialize(
      bufferSize: bufferSize,
      nChannels: nChannels,
      sampleRate: sampleRate,
      pcmType: RawSoundPCMType.PCMI16,
    );
    current.add(player);
    await player.play();
    final buffer = synth!.pcm(3);
    await player.feed(buffer);
    if (current.length > 1) {
      current.remove(player);
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('The Pocket Piano'),
      ),
      body: PianoDeck(
        octaves: 8,
        onStart: (midi) {
          play(midi);
        },
        onStop: (_) {},
      ),
    );
  }
}

extension on Synthesizer {
  Uint8List pcm(int seconds) {
    final buf16 = ArrayInt16.zeros(numShorts: 44100 * seconds);
    renderMonoInt16(buf16);
    return buf16.bytes.buffer.asUint8List();
  }
}

[chipweinberger]

Hey rody!

Cool seeing you found my project!

I remember seeing your account awhile back when searching through your midi plugins! We've come full circle.

It would be great if people made PRs for example/raw_sound and example/rust_audio

[rodydavis]

Yep that's what I was planning on!

[chipweinberger]

I wrote flutter_pcm_sound to fix this issue.

And the example app now demonstrates realtime audio using it.