Backpressure exposure for asynchronous send()

[agoode]

We have so far been able to use synchronous send but this provides no mechanism for backpressure.

The current implementation in Chrome uses large internal buffers to hide this issue, but this is wasteful and results in the browser silently dropping messages when the buffer is overrun. This condition is not common with normal MIDI messages, but sysex messages can be of arbitrary length and can be slow to send, so Chrome also puts limits on those as well.

In order to allow implementations to avoid these buffers and arbitrary limits, we need a mechanism for non-blocking send.

[cwilso]

There are too many missing bits of information for me to fully grok your example (e.g. the undefined parameter, the tick, how a while loop in the main thread doesn't turn into a synchronous mess). However, I feel VERY strongly that we do not need a new MIDI timeout API that calls back into the main UI thread - that's setTimeout. Yes, it has an unfortunate throttling - however, 1) that problem should be solved universally for setTimeout/setInterval users that know what they're doing, via a flag or some such, and 2) there's a fairly trivial workaround, which is to use worker threads (seriously, read through the metronome code; I pretty much packaged the worker tick as a drop-in to any other sequencer). You should not have the problem you describe, UNLESS there is a bug in Chrome's timestamped-future-send of MIDI messages with the new scheduler, if you have properly used the worker workaround.

(Note that it's relatively easy to switch back and forth on window.blur and window.focus, too, so you don't even always need a worker thread running.

[notator]

Understood. I was just brainstorming, and in a bit of a hurry...

Is the setTimeout issue already being dealt with in the proper places?

Your metronome works fine for me in a background tab in Chrome, so I assume there is no bug in "Chrome's timestamped-future-send of MIDI messages".

As far as I'm concerned, the backpressure issue could be solved if send would simply throw an exception of some sort when the message is too big for the available buffer space. Is there a standard BufferOverflow exception ? ( I have been unable to discover where the Web MIDI API exception types are defined.)

[toyoshim]

As I repeatedly said, WICG is the right place to discuss such throttling, and there are threads for the setTimeout throttling in some cases, e.g., background tab, and cross origin iframe. Actually there is more aggressive proposal to suspend background tabs completely. IMO, current Web MIDI use case isn't fair enough to avoid such throttling.

Exception does not solve the problem because users can not be notified when buffer is available again. Retry with setTimeout won't be an acceptable choice.

[notator]

Exception does not solve the problem because users can not be notified when buffer is available again. Retry with setTimeout won't be an acceptable choice.

But I'd like to be told when the buffer overflows, even if I can't recover.

[cwilso]

To be clear - the setTimeout issue is not relevant here. There's a workaround - the worker setTimeout, as I use in the metronome (https://github.com/cwilso/metronome/commit/7b78d49d297107bec2c10803ee2702f5b4e9e3cc). No one seemed particularly interested in fixing the higher-level issue, but that would land in DOM-space somewhere.

The issue here is 1) how we expose backpressure for sending data to MIDIOutput, in order to enable "large" sends, and 2) whether we use Streams to do that (and a corollary, whether there's any significant utility in adding Streams to MIDIInput then.

I feel VERY strongly, btw, about not removing send() as-is. I also think SOME backpressure fix should go in v1.

[notator]

+1 for Chris' summary. Maybe a couple more scenarios would help (its not just a SysEx problem):

Scenario 1: a) An app allows its users to change the speed at which a MIDI file is played back. b) A user plays the file so fast that data starts getting lost. c) The user doesn't know this, and goes faster and faster. Info is being silently ignored. This means that patch-changes, noteOns, noteOffs etc. are getting lost randomly, and the file is going to sound wrong. There's no point in continuing in this situation, so the app ought to be able to stop it. The app ought to be able to tell the user that the file can't be played that fast. Ideally, the user should be told that before the app starts sending the too fast messages.

Scenario 2: A composer is creating a MIDI composition containing large amounts of data being sent very fast. The problem is that there is no way of knowing that the buffer is overflowing, except that each performance is going to sound randomly different from the one before. Having a professional attitude, the composer wants to use the browser's capabilities to the full, but still be sure of having predictable performances. There's no point in writing piccolo music for a tuba. This scenario also raises the question of garbage collection. Can compositions of arbitrary length be performed without pauses for GC?

[cwilso]

WG decision: move backpressure to v2 incubation; streams is the likely right API.

[starfishmod]

Hi I wanted to re-raise the awareness of this issue. I am currently developing prototypes for some of the new MIDI-CI specifications. A large part of what is required is to receive and send lots of SysEx packets at one time. As I am using USB MIDI my bandwidth can be much higher than 31.25 kbps.
Currently I can receive SysEx at high rates at the speed of whatever the USB MIDI sends through it's data. When it comes to sending data I have to hobble the speed to 31.25Kbps (using the send with timestamp) as I can't determine the bandwidth. And if I send the data too quickly then packets are silently dropped and the receiving device misses vital packets. I know this is slated for v2 but can this be reviewed sooner. I am happy to discuss any details I can.

• Andrew

[toyoshim]

I'm revisiting here during Web MIDI bug triage for Chrome.

So, I just want to say that chromium side bug for this topic is here, https://crbug.com/383578, and we will work on that once something is decided here, in the spec side thread.

Note, here is the right place to discuss the API surface. Please do not discuss it at the crbug, just in case.

[starfishmod]

@toyoshim what can we do to help? Is it as simple as using send and receiving a buffer is full error? and therefore go to a queue? Should SysEx use a different buffer? What are the (good) possible options that should be considered? thanks for looking into this :)

[toyoshim]

@cwilso 's last comment is a current plan. I agreed that using Streams would be a reasonable idea to solve this kind of issues.

When we discussed last time, Stream was not stable yet maybe? Also, any other UAs didn't have much progress on implementation. So, it was marked as V2.

But both situations have been changed. We may be able to revisit this for V1. I'm fine if we go.

@padenot or @qdot may have some thoughts from Mozilla

[solitud]

I am not competent to discuss but I am wondering if there is a solution or workaround to send larger SysEx dumps with Web Midi? I am maintaining a SysEx librarian under https://f0f7.net and I have problems to send bulk dumps (~11kb) to older synths like MicroWave1. It seems like Web Midi is sending data faster than some interfaces or synth can process which results in buffer overflows/data loss. I tried to send the data in timestamped chunks but than of course I get System exclusive message is not ended by end of system exclusive message I am guessing regarding to the age of this thread it will need some more time to have full working SysEx buffer size/speed support in Web Midi so I would really appreciate if someone can point out if there is a current workaround to slow down the transmission?

[toyoshim]

I don't think 11kB is not so large to be handled in Chrome Web MIDI implementation. On Windows, Chrome Web MIDI backend has 256kB limit per each SysEx message size, and also all platforms including Windows has 10MB limit for in-flight buffer size.

As far as I know, only sampling data transfer would hit this limit, and I heard it can be split into multiple SysEx messages.

At this moment, what users can do is to estimate the worst case throughput, then send split messages step by step instead of sending it at once. E.g., using setInterval(), and send each 128kB in the timer callback would work. Of course, each message should be a complete SysEx message as you can see the error.

[agoode]

Note that for Linux specifically, there is a long standing kernel bug that silently drops parts of most sysex messages unless the writer explicitly writes slowly. Unfortunately, Chrome uses the broken API and this will cause problems on Linux and Chrome OS devices. I suspect other Linux Web MIDI implementations will also use the ALSA seq API which has this problem.

I would like to fix this Linux bug but have not yet. Some more background can be found at http://crbug.com/917708.

[inno]

How can we get some traction on finalizing this?

Limiting buffer length, either via bug or definition, is not aligned with the MIDI specification. There is actually no formal expectation of use or length of sysex messages. Per MIDI 1.0 96.1v3:

System Exclusive messages ... are used to transfer any number of data bytes in a format specified by the referenced manufacturer.

If folks are looking for a real world example of an arbitrarily large sysex message, I have two words: firmware updates. Different manufacturers handle these different ways, but many simply have "large" encoded binaries and checksums in a single message. Manufacturers are allowed to do this, but unfortunately, not required to handle chunking this data in any way (by spec).

Happy to provide more info and/or obscure use cases if needed!

[toyoshim]

When we started this discussion, Streams API was not matured yet. But today, having WritableStream in MIDIOutput interface sounds the best choice? We already have many tips to write Promise-based asynchronous code in a simple way, that says adding another asynchronous trick won't be a good idea.

We prefer to keep the existing send() method together for compatibility and simple uses. That sounds reasonable to me.

@cwilso Any thoughts? You were sceptical about using Streams in the past, but WDYT today?

I'm happy to implement it in Chrome if the spec is updated.

[diegodorado]

What is the current status of this issue? Any custom code example to implement meanwhile?

[toyoshim]

Discussed at TPAC, and agreed that using Streams is the best here, but still we need to clarify detailed API surface.

[ghost]

I built a midiWebApi sequenser that work to playback,,playup, record "simultaneously" well more or less. But the playup lags because i use timeout, i would like to change the timeout solution to a buffered playup. Will it remove the playup lag? Can you direct me to an example on howto implement a buffered playup, i do not quite get what it is. Since i just use timeouts/waitstate beteween note playup.

Very few examples showing how buffered playup work.

https://jonasth.github.io/

[cwilso]

Hey @jonasth, I wrote an article a long time ago about Web Audio clocks, that also applies to Web MIDI - in particular, why using setTimeout is not a good idea. https://www.html5rocks.com/en/tutorials/audio/scheduling/.

The issue you're having is unrelated to this issue (this isssue is essentially about sending massive amounts of data on a slow MIDI link).

[ghost]

But i heard webmidi players that i at least "perceive" as in synch not slowing down passing notes, sure enough there is no bar to measure if they lag or not so i really don't know, but you claim it is webMidi itself create the bottleneck for passing notes?

Thanks for your kind reply, seldom i get any.

Jonas

---

Från: Chris Wilson notifications@github.com Skickat: den 26 februari 2021 20:41 Till: WebAudio/web-midi-api web-midi-api@noreply.github.com Kopia: jonasth jonas.thornvall@hotmail.com; Mention mention@noreply.github.com Ämne: Re: [WebAudio/web-midi-api] Backpressure exposure for asynchronous send() (#158)

Hey @jonasthhttps://github.com/jonasth, I wrote an article a long time ago about Web Audio clocks, that also applies to Web MIDI - in particular, why using setTimeout is not a good idea. https://www.html5rocks.com/en/tutorials/audio/scheduling/.

The issue you're having is unrelated to this issue (this isssue is essentially about sending massive amounts of data on a slow MIDI link).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/WebAudio/web-midi-api/issues/158#issuecomment-786882710, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEX6WHG4J7MJG3HCQ3VFP5TTBABN5ANCNFSM4B27BMUA.

[ghost]

function STARTPLAY(){ console.log("STARTPLAY()"); copyEv = playEv.slice(); last = copyEv[copyEv.length - 1] copyEv[copyEv.length] = last; var waittime = copyEv.shift(); console.log("waittime="+waittime); midEvent = 0; myTime=waittime; //console.log(SF2PLAY); if (SF2PLAY==true){stopJump=setTimeout(SF2Playup, waittime);} else {stopJump=setTimeout(doPlayup, waittime);} } //WHILE MESSAGES TO PLAY DO PLAYUP function doPlayup() { if (keepGoing) { if (copyEv.length) { if (track[playtrack].midiMess[midEvent].data0<192 || track[playtrack].midiMess[midEvent].data0>207) {noteMessage = [track[playtrack].midiMess[midEvent].data0, track[playtrack].midiMess[midEvent].data1, track[playtrack].midiMess[midEvent].data2]; } else { noteMessage = [track[playtrack].midiMess[midEvent].data0, track[playtrack].midiMess[midEvent].data1];} if (mode=="Play"){ pianoKeypressOut(); scrollPianoOut(); } outportarr[outportindex].send(noteMessage); waittime = copyEv.shift(); midEvent++; setTimeout(doPlayup, waittime); } else { stopPLAY();} } }

This is the playup function

---

Från: Chris Wilson notifications@github.com Skickat: den 26 februari 2021 20:41 Till: WebAudio/web-midi-api web-midi-api@noreply.github.com Kopia: jonasth jonas.thornvall@hotmail.com; Mention mention@noreply.github.com Ämne: Re: [WebAudio/web-midi-api] Backpressure exposure for asynchronous send() (#158)

Hey @jonasthhttps://github.com/jonasth, I wrote an article a long time ago about Web Audio clocks, that also applies to Web MIDI - in particular, why using setTimeout is not a good idea. https://www.html5rocks.com/en/tutorials/audio/scheduling/.

The issue you're having is unrelated to this issue (this isssue is essentially about sending massive amounts of data on a slow MIDI link).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/WebAudio/web-midi-api/issues/158#issuecomment-786882710, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEX6WHG4J7MJG3HCQ3VFP5TTBABN5ANCNFSM4B27BMUA.

[rianhunter]

What is the status of this issue? Without any way to detect backpressure I have to artificially slow down sending bursts of messages to account for older midi devices that don't use USB.

For what it's worth I agree with @bome:

I'm somewhat reluctant for adding streams, it's seems like mainly adding clutter to the WebMIDI API and implementation, while providing very little new functionality. Can't we just add a function sendAsync(data, timestamp) with a Promise as return, or define a MIDIOutput listener which fires whenever a MIDI message is delivered?

A method like his proposed sendAsync would be sufficient, extensible, and relatively easy to spec.

[gudsoft]

I second the other requests for providing a solution for this issue. It is important. For many of the reasons already discussed in this thread.

First, I would like to commend all the work done on Web MIDI API so far... it is truly exciting (to me) what browser-based MIDI means! At the same time, and at this stage of the development of the API, it is quite surprising this foreseeable and basic reality (the need for SysEx throttling) has not yet been accommodated by the API (and is the subject of so much debate and "needed explanation"). The need for throttling is simply a reality of the mature MIDI ecosystem and its legacy, much of which is just as relevant today as it ever was. Pointing fingers at old or new devices, drivers, etc. (or wherever the need for throttling may arise in a particular system) does very little to advance all the good that the Web MIDI API otherwise brings.

Every piece of MIDI software of consequence created in the last few decades that deals with SysEx has recognized that providing an ability to throttle SysEx transmission is a necessity, either to accommodate limitations of the receiving device or transmitting system (OS, drivers, interfaces, etc. -- and now potentially even arbitrary internal browser limitations!). Indeed, the computer, with its infinite flexibility for adaptation and reprogramming, is in the best possible position to accommodate "harder" limitations found elsewhere in a MIDI system (within "set-in-stone" old hardware, or "hard-to-update" firmware, etc.). As a component of that flexibility, the Web MIDI API very clearly has a role to play here, IMO.

The Web MIDI API should provide (or at least enable) a solution for throttling SysEx transmission, period. If it does not, whole classes of "imperfect", yet still valuable and viable, MIDI devices and software (synthesizers, interfaces, and even apps, etc.) will be needlessly cut out of the benefits of browser-based MIDI.

An arguably important "modern" case in point: Many of the cheap "commodity" USB MIDI cables available today (which usually have full-speed SysEx transmission bugs) could benefit (read, be made to "actually work") via a simple adaptation of the software running on the computer (to throttle SysEx). Wouldn't it actually further the goals of the Web MIDI API to enable as many MIDI devices (and users!) as possible, such as these commodity interfaces, despite their flaws?

Web MIDI API's current lack of any means for SysEx throttling, as well as its insistence that whole SysEx messages be provided for transmission are, IMO, arbitrary and harmful limitations of the API, and prohibit certain valid and creative uses of MIDI. For example, why can't my browser-based MIDI app "stream" SysEx (generating it over time), or "open" a bidirectional SysEx connection using a single 0xF0 on each side, and then communicate asynchronously entirely with an "exclusive" protocol of 7-bit messages? Both things are trivially possible with MIDI itself, but currently not the Web MIDI API. (It should be pointed out, some of the earliest -- and perfectly valid -- SysEx implementations in Casio and Roland devices are effectively asynchronous 7-bit protocols, reliant on intra-SysEx handshaking, something the Web MIDI API is currently incompatible with, for no good reason, IMO.)

MIDI has grown to its popularity and ubiquity today, yes, based on the efforts of many to adhere to a robust, simple, and dare we say "perfect" common specification, but also in no small part due to the creative (and arguably "necessarily obvious") accommodations that have been made, especially in software, aimed at making sure MIDI works as well as possible even for some of the most "imperfect" members of the ecosystem, and that unforeseen innovations are not precluded by assumed "valid uses" of features like System Exclusive.

It would be awesome if the Web MIDI API wholeheartedly recognized and continued in this tradition.

[gudsoft]

As a separate note, I would like the suggest that, the sendAsync method proposed earlier in this thread should not only allow an individual SysEx message to be sent asynchronously, with completion signaling via Promise or other mechanism, but it should allow a partial SysEx message to be sent (nominally to be followed by the sending of subsequent part(s) until completion of the whole message).

Adding these two basic abilities... asynchrony and divisibility... are enough to "re-enable" everything that is natively possible with SysEx over MIDI itself, and would provide the means for the Web MIDI API to fully accommodate past and future innovations, and other vital use-cases that have been with us since MIDI's birth (see below). A suitable sendAsync would accomplish this, as would some kind of streaming interface, but the important thing is that the API expose both abilities together.

It should also be pointed out that SysEx throttling isn't the only issue of importance here... the ability for apps to freely interject MIDI realtime messages (such as Timing Clock) during a long-running SysEx message (something the MIDI protocol has always been designed to handle) is an important use-case for playback and synchronization. Again, fine control over the divisibility of SysEx data from the perspective of the API is required.

I applaud any and all principled efforts to keep the Web MIDI API simple and easy-to-use (achieving this for any API is no small feat), but for all the reasons above, a synchronous, atomic send method is, in fact, too simple, and precludes much that the API could and should enable for browser-based MIDI apps.

[hoch]

2023 TPAC Audio WG Discussion: The WG is still undecided about whether this feature will be included in V1 or the next version of the specification. It is likely to be added later, after developer surveys, prototyping, and field experiments have been conducted.

[mjwilson-google]

I will schedule this for CR / V1 until we have a chance to discuss it in the Audio Working Group, but we will likely push it to future work.

[mjwilson-google]

Audio Working Group 2023-10-05 meeting conclusions:

• This looks like it will be a significant amount of spec and implementation work
• It also looks like it is important
• The Working Group doesn't have enough information to make a final decision on how to handle this, so we will conduct a developer survey

[rianhunter]

Thanks @mjwilson-google looking forward to seeing progress on this issue!