search

kovaxis / midly

A feature-complete MIDI parser and writer focused on speed.
The Unlicense
138 stars 26 forks source link

Incremental writing? #11

Closed Ploppz closed 4 years ago

Ploppz commented 4 years ago

I don't know if this is something typical to do, but I think it could kind of make sense? It seems at least like the MIDI file format should lend itself perfectly to writing incrementally to a MIDI file, if I am not mistaken. I could help develop it, I just don't know the best API.

Boscop commented 4 years ago

What I do is I incrementally build the tracks and then I create the SMF and save it. Any reason why that wouldn't be sufficient? Are you concerned about RAM usage? (Are you processing multi-GB "black midi" files?) Btw, the SMF type 1 format wouldn't be suitable for incremental writing because tracks are stored after another.

My main issue is the ergonomics of constructing the track data, because I can't just append an owned sysex or MetaMessage, because they only borrow: https://github.com/negamartin/midly/issues/7#issuecomment-632147145

Ploppz commented 4 years ago

What I want to do is to forever listen to MIDI to save whatever is going on always. Could be into multiple midi files or one big file. Anyhow, I think it is best to write to the MIDI file periodically in order to not lose any data. Every minute or so. That's why I'm looking into incremental writing. But maybe it's unnecessary to worry about losing data in my case... and I suppose I could as well write the entire file ever so often to backup the "WIP" midi. I thought track chunks could be interleaved, looking briefly to the MIDI file specification.

Boscop commented 4 years ago

@Ploppz In SMF type 1, the tracks have to be written one after another. In SMF type 0, all events appear on a single track, but the size of that track data is still stored before it. In theory you COULD implement a streaming writer for type 0, which under the hood also seeks back to update the size field, like the streaming wav writer in the hound crate (for wav it makes sense since the data is large). BUT: Unless you're dealing with a "black midi" stream, it doesn't really require incremental writing: From experience[1], 5 minutes of normal full 16-channel midi song material is ~60 kb. That'd be 720 kb per hour. Surely you have enough RAM to spare? :) (Depending on your definition of "forever".) I'm interested in your use case, what are you doing with midi? :)

In my application I store the recorded midi in RAM and save it every once in a while (atomically, using atomicwrites).

Right now I think there are more important things to spend time on.. Like the above mentioned ergonomic improvements for writing midi files. And then https://github.com/negamartin/midly/issues/8

[1]: E.g. this midi file that I recorded is ~5:21 long and is a dense mix of 9 songs, using all 16 channels and it's 64 kb. So 5 min would be ~60 kb. (Btw, I did a scan of my midi song library, the avg song file is 43 kb, and some of them even contain embedded lyrics which skews the average higher.)

Ploppz commented 4 years ago

I see, thanks for correcting me!

Ploppz commented 4 years ago

And thanks for your experience with midi file size.

I'm writing a daemon that listens for midi inputs and saves everything. That way I can go improvise on piano without worrying about forgetting it or recording it if I do something good :D

Boscop commented 4 years ago

@Ploppz Ah, in that case, your kb/min will be even less (and zero when not playing), because you will only have 1 channel of data, and very few non-note msgs. In my mix above, at song transitions, all channels contain msgs to switch (i.e. serialize the diff of) their state (prog chg, CCs, pitch bend etc.) due to how my midi mixer works, but you won't have that data..