Playing back a midi file

[Dimev]

I'm porting over a simple midi player I made in python to rust with rodio and midly. I'm now stuck at how I can properly play back a midi file, by looping over all note-ons in order, and waiting the correct amount of time before playing the next note

[PieterPenninckx]

This is not super straightforward to implement. I've implemented part of it in rsynth (Note: it's not yet in the version of rsynth that you can find on crates.io). Using rsynth just for that is overkill, I think.

I'm not sure what would be the best place for this functionality.

[Dimev]

Ah thanks! I'll look at it!

[PieterPenninckx]

I've given this some more thoughts. The task at hand consists of two parts:

• part 1: create an iterator that iterates over all events in all tracks, in order, with an "absolute" timestamp in midi ticks (and the index of the track). The heavy lifting of this can be done by the kmerge_by method in the itertools crate. The "only" part that is missing is some "glue code" so that it is ready to use in combination with midly
• part 2: given that iterator, create another iterator that has the timing (absolute or relative) in microseconds.

Both parts are currently convoluted in rsynth, but rsynth isn't really the place where this should be and these parts should best be separated. The first part, I think, is best suited in midly (at least the glue code), maybe behind a feature flag (so as to not force a dependency on itertools for those who don't use it). @negamartin, do you agree? If it's ok for you, I can create a pull request.

For the second part, I'm not sure. I think I'll just leave this in rsynth until I have found a better place. rsynth currently also has an event-queue, maybe they can together move to a timed-event-tools crate or something? Suggestions are welcome.

There's also the inverse task: recreate an Smf from an iterator of events with timestamps and track indices, which is similar (only reversed). I think we'll best tackle these together. I think I can express this more clearly in code, so if I create the pull request mentioned above, I'll also include some code for the inverse task.

[Boscop]

@Dimev Btw, this is a midi player in Rust: https://github.com/PolyMeilex/Neothesia It uses midly: https://github.com/PolyMeilex/Neothesia/blob/master/lib_midi/Cargo.toml and has a discord: https://discord.com/invite/sgeZuVA. I haven't looked at its code but you can just create a midi player by converting event delta times into absolute times and having a method that gives you all events in a certain tick window, and then you call this method at a certain frame rate, calculating the length of the tick window based on nanoseconds since last iteration and BPM you want to play it at (it could be the same BPM that was stored in the file, if none is stored, the midi standard says it's 120). And remember to not throw away the fractional part of your tick window but add it to the window for the next frame, otherwise you'll play it slightly slower than the intended BPM. Then you can use any constant-fps iterator/loop such as https://crates.io/crates/spin_sleep or https://crates.io/crates/game-loop. Or, if you're playing it as audio, you have an audio callback that is already being called at regular intervals, so you can compute the tick window from the buffer length in each call. Make sure to call the method to give you non-overlapping event slices that are continuous (events which time is in start .. end (not including end). The next frame will have its start as the same value as current frame's end, to make the frames continuous, not missing any events). If you want to support not just forward playing but jumping, you can create a random access player that will integrate events from the start into a 16-channel midi state, and then diffs this state from its previous state (the diff results in midi events), unless the current call's start equals the previous call's end (continuous case, which doesn't require integrating events from the start). You could even optimize this further by saving pre-integrated midi states for every N number of msgs. But usually, for any regular midi song, integrating from the start is fast enough. It would even be fast enough for larger midi files (depending on what else you're doing every frame). But if you want a general random-access player solution that works for multi-GB black midi, then you could pre-integrate the midi event state every N msgs or ticks..

@PieterPenninckx I think a midi player is better implemented in another crate that depends on midly, since midly's scope is mostly the low-level midi format, and there are infinite things to build on top of it. E.g. you could name it midly-player, to make the relationship clear. (Also, then you wouldn't be blocked by waiting to get PRs merged into midly.) Also, the midi player crate would be opinionated, so that's another reason why it shouldn't be in midly, because there are different ways to implement a midi player. E.g. I wouldn't have much use for a player that uses ms-timed events, since my BPM is varying at runtime based on user input, and I also wouldn't want to use different ticks-per-beat values for the different midi files I'm loading, since everything in my engine uses the same "timing grid", so I "resample" every loaded midi file to 8192 ticks per beat and then sample them with the above random-access method. So I think it's better to keep midly's scope on the midi format, and have higher-level opinionated stuff in dependent crates :) For saving an absolute-timed midi arrangement as a Smf file using midly, you could also create another crate (that uses the same absolute-timed format, defined in a common crate), since midi players usually don't need to write SMF files, only read them.

[Dimev]

Ah thanks I think a separate crate like midly-player would be nice to have.

[kovaxis]

Just like boscop says, playing back a MIDI file is not a trivial task, and is out of scope for midly.

Playing back a MIDI file would go something like this:

1. Use Smf::parse from midly to read all tracks.
2. Convert all delta-times into absolute times.
3. Merge all tracks into a single large list of events and sort them by time.
4. Loop over all events in order, sleeping for the appropiate time.
5. If you simply want to play a sound on every note-on, just do that on every note-on. If you want more complex MIDI-compliant behaviour, use a crate like midir to redirect MIDI events to the system MIDI player.

Just like Boscop says, if you want to seek around the file things get more complex. If you don't want to use the system MIDI player, things get ridiculously complex.

Yes, a high-level MIDI crate with simple functions like load, play and pause might be nice. If you feel like it, you could create one yourself!

[Boscop]

If you don't want to use the system MIDI player, things get ridiculously complex.

You can send your midi to any port with midir, and on the other end you can use any GM-compatible player, like a soundfont player or synth with a GM bank. You can find a lot of GM soundfonts on the internet.

(It only gets ridiculously complex if you would want to write your own soundfont player :)

[kovaxis]

Indeed, MIDI is pretty flexible.

(It only gets ridiculously complex if you would want to write your own soundfont player :)

That's exactly what I mean 😆.

[PieterPenninckx]

Hi all. I think the question "how to organise the ecosystem around playing midi files" is an interesting topic. I've created a thread on the rust-audio discourse group to discuss this. This can give it a larger audience (and also takes it away from this issue). Feel free to join the discussion there. For those who do not want to create an account at that discourse instance, if @negamartin agrees, we can also continue (part of) the discussion here.

[chriscoomber]

(It only gets ridiculously complex if you would want to write your own soundfont player :)

I ported TinySoundFont to a rust crate, which should solve that step. Essentially you can just feed it noteOn/noteOff events, and ask for wave data with its Tsf::render_float function and it gives you the bytes. It's still a WIP but the core functionality is there.

The next step is you probably have some library that interfaces with your audio devices. I've been working on Android so for me it's oboe but it may depend on whatever your target OS is (another example is SDL) . That library will probably have some kind of stream which you setup with a callback, and in the callback the stream will for X samples of wave data, which you would produce by calling Tsf::render_float. With this architecture, you let the callback stream do all the driving - every time the callback function is called it just needs to progress your midi sequencer by however much time has passed, pass the noteOn/noteOff events to the synthesizer (in this case Tsf), and then request the right amount of wave data from the synthesizer to return.

I'm actually currently working on putting all of this together right now (because the default midiplayer in Android is crap). I will probably use midly (hence why I'm reading this).

This thread was really interesting to read, and relieving because Boscop's description of a midi sequencer is the same as what I was imagining. For random-access seeking, I was just replaying all events from the start to find state changes (e.g. tempo change and preset change events), which seemed good enough.

[PieterPenninckx]

Hi all,

Going back to @negamartin 's list:

1. Use Smf::parse from midly to read all tracks.
2. Convert all delta-times into absolute times.
3. Merge all tracks into a single large list of events and sort them by time.
4. Loop over all events in order, sleeping for the appropiate time.
5. If you simply want to play a sound on every note-on, just do that on every note-on. If you want more complex MIDI-compliant behaviour, use a crate like midir to redirect MIDI events to the system MIDI player.

I've published the crate midi-reader-writer, which covers steps 2. and 3.