Modify MIDIOptions to be able to specify software synths to be included

[toyoshim]

Now, sysex is the only flag that MIDIOptions contain. But not only sysex, but native software virtual synths have security risks even on normal messages. So, we should introduce a permission to use native software virtual synths, and if it isn't requested, we should hide software synths against the returning MIDIAccess.

Here is why virtual synths have risks even on normal messages: Especially on Windows, user land drivers run in the same process with the browser for virtual synths, and it might crash the browser in various ways. Since sound synthesis from MIDI events are complicated, they may have crash bugs that could be security vulnerabilities.

[notator]

I dont think that crashing virtual synths are just a security problem. Even if I get permission to use a virtual synth, that is no guarantee against it crashing. And that's true on all operating systems, not just Windows. Is there some way my web app can detect, and recover from, a crashed software synth?

[toyoshim]

notator: Your proposal isn't a spec topic, but an implementation detail, I think. If underlying systems have a critical problem, applications constructed on the top of the system could not be safe potentially. In some lucky cases, the application could find a workaround, but in other cases, it couldn't. So this is a proposal to have safeguard against such risks coming from underlying systems that are out of browser's control.

[notator]

toyoshim: I'm really not making a proposal. I just wanted to know if, using the architecture you are envisaging, there is any way a browser could help me detect, and recover from a crashed software synth (e.g. by sending me an exception). Your answer seems to be no. My application will simply crash somehow (either its tab or the whole browser) if its slave synth crashes. So, if I want to ensure that my application won't crash, I have to test all the inputs it can send to its slave synth, and filter out anything harmful. In my particular case, that's no problem at all. :-) The general answer is that the authors of host applications and software synths will cooperate to improve the quality of their software. No problem. :-)

[toyoshim]

Can you file a separate bug? Recovering scheme and this proposal should not affect each other, and could be an independent topic.

The best design to save the browser from software synths' crashes is that browser launches an isolated service process to host software synths, and just communicates with it to control synths. Under this design, if the service process crashes, the browser could restart it.

But if the software synth can cause system crashes, ...no way to recover from it. I'm not sure if it is really possible. The problem is a software synth isn't a slave technically on Windows, but it's a part of an application, or I can even say it's a master because it's working as like a part of the operating system.

Since the Chrome applies multi-process architecture, this design could be a choice if it worth taking. But for now, I could not find enough reason to take the design. Also, in person, I'm negative to introduce operating system dependent features, and I'd pay much attention to web technology based virtual synth discussed at #45 .

[nwsw]

My distrust of general, open web site access to MIDI ports is not limited to just sysex and soft synths. I would prefer a page/site authorize mechanism for the whole MIDI subsystem.

When I do authorize a page, I would certainly want access to the soft synth.

[toyoshim]

Current spec already says: Requesting MIDI access should prompt the user for access to MIDI devices, particularly if system exclusive access is requested. In some scenarios, this permission may have already been implicitly or explicitly granted, in which case this prompt may not appear.

So, another browser may prompt always, and even Chrome may do in the future. Anyway, if MIDIOptions can contain more information, browsers and users can take a smart choice.

[toyoshim]

By the way, if we prompt for all cases, it means that Web MIDI is not available over HTTP any more since Chrome does not allow permission against non-secure sites for new APIs. Is it ok?

[cwilso]

It may be worthwhile to mention this in the spec, but I think it's okay.

On Tue, Jul 14, 2015 at 10:11 PM, Takashi Toyoshima < notifications@github.com> wrote:

By the way, if we prompt for all cases, it means that Web MIDI is not available over HTTP any more since Chrome does not allow permission against non-secure sites for new APIs. Is it ok?

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-midi-api/issues/150#issuecomment-121490498 .

[bome]

uhm, only enable WebMIDI for https? I don't think that's a good idea (not yet).

[cwilso]

Well, any permission-requiring API is going to need TLS. On Jul 15, 2015 3:59 PM, "Florian" notifications@github.com wrote:

uhm, only enable WebMIDI for https? I don't think that's a good idea (not yet).

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-midi-api/issues/150#issuecomment-121771955 .

[notator]

I've been thinking some more about this, and have come to the conclusion that browsers can (should, and probably will) ignore software synths altogether. Especially those built in to operating systems. Website authors only actually need to invoke a browser's implementation of the Web MIDI API when they want to access hardware devices.

This idea ought to help keep the both the Web MIDI API, and the code browsers use to implement it, simple.

Consider the following scenario: My website needs a MIDI output device so that people who visit my site can hear the sounds the device produces when I send it MIDI messages. I want to reach as many people as possible, and most computers are not attached to third-party MIDI hardware, so this needs to be a software device.

It would, of course, be best if the device implemented the WebMIDIAPI MIDIOutput interface, so that I can use soft- and hardware synths interchangeably.

If such a Javascript device existed, I would not have to go through the browser's implementation of the WebMIDIAPI at all, and my visitors wouldn't need to install any plugins.

There are examples of Web Audio synths out there already, but I need one with a simple MIDI API, that is able to load and play custom sounds. I don't want its interface to be a complicated GUI.

I have a more detailed proposal to make about setting up an open-source project which would support the delivery of concrete Web MIDI Synths, but that's off topic for this forum. I think, after getting some feedback here, that should be done at public-audio-dev@w3.org. The discussion there may well throw up a need for standardizing aspects of software synths beyond the current WebMIDIAPI MIDIOutput interface, but we can wait and see about that.

Would the security issue that lead to Chrome banning the Microsoft GS Wavetable Synth affect such Synths as well? I imagine not, since they would, after all, just be ordinary Web Audio applications.

All the best, James

This posting is also an answer to threads #45, #124 and #151.

[ryoyakawai]

I have developed sort of a wrapper which behave like MIDI devices. That is provided as Polymer Element. [Source] https://goo.gl/3YPqSK [Live Demo] https://goo.gl/vJ8aRo (the devicename is "GMPlayer(WebMIDILink)")

This element is a bridge between the web applications, which is using Web MIDI API, and Web Audio Synthesiser with Web MIDI Link[1]. To using this elemennt, you do NOT need physical device to play MIDI messages. Good things is that Web MIDI application does not need to integrate Web Audio Synthesiser to play. Good things is that everything are happening on the Web Platform. And technically, the element is using "iframe" to call the Web Audio Synthesiser, so it is easy to import Web Audio Synthesiser that has already release to the public.

By the way, the code to use this web application is bellow: (1) import the element (2) add to the element to the list of MIDI output device which is obtained by requestMIDIAccess(); Only to add one line of JavaScript code allow you to use the element as MIDI device.(Line 30 in [Live Demo]).

I hope this element is one of the solution for the virtual MIDI device issue, because it is easy to implement and easy to use any Web Audio Synthesiser with tiny integration.

At default, the element is using GMPlayer[2], which is developed by g200kg[3]. The GMPlayer is using service worker for caching all of the voices and application itself, so it is also able to run on offline.

[1] http://www.g200kg.com/en/docs/webmidilink/ [2] https://webmusicdevelopers.appspot.com/webtg/gmplayer/index.html [3] http://www.g200kg.com/

By the way, I have also developed virtual MIDI input device. https://goo.gl/D9g4hu This device is also displayed in the [Live Demo]'s input list as "PC Keyboard".

[toyoshim]

Let me revisit this topic. I would expand this thread to cover discussing options to specify requesting MIDI device backends, or physical layers.

OS providing software synth was one topic, but I'd count in BLE MIDI device here. In Android M, the native MIDI API request to obtain a coarse location permission to use BLE MIDI. That means if the Web MIDI want to enumerate all devices, it will ask coarse location permission always. I feel it's curious. If we will add more physical layers to support MIDI in the future, it will request more permissions to just enumerate devices since each physical backend may have each new risk to ask.

So, my opinion is to change the MIDIOptions to something like this; dictionary MIDIOptions { boolean sysex; boolean software; boolean ble; };

Can we have a time to discuss this in f2f meeting at the coming TPAC?

[cwilso]

SGTM (to discuss at TPAC).

On Tue, Oct 20, 2015 at 11:30 PM, Takashi Toyoshima < notifications@github.com> wrote:

Let me revisit this topic. I would expand this thread to cover discussing options to specify requesting MIDI device backends, or physical layers.

OS providing software synth was one topic, but I'd count in BLE MIDI device here. In Android M, the native MIDI API request to obtain a coarse location permission to use BLE MIDI. That means if the Web MIDI want to enumerate all devices, it will ask coarse location permission always. I feel it's curious. If we will add more physical layers to support MIDI in the future, it will request more permissions to just enumerate devices since each physical backend may have each new risk to ask.

So, my opinion is to change the MIDIOptions to something like this; dictionary MIDIOptions { boolean sysex; boolean software; boolean ble; };

Can we have a time to discuss this in f2f meeting at the coming TPAC?

— Reply to this email directly or view it on GitHub https://github.com/WebAudio/web-midi-api/issues/150#issuecomment-149795299 .

[cwilso]

Incidentally: I support a "software" type, I would rather not separate BLE devices (although I realize this will cause prompting for all midi services then.)

[cwilso]

WG resolution: support software option, don't support BLE. Can revisit later if necessary.