search

leafo / sightreading.training

🎹 Sight reading training tool
http://sightreading.training
445 stars 77 forks source link

Web MIDI Browser compatibility (iOS) #8

Open pianogospel opened 7 years ago

pianogospel commented 7 years ago

Add IPAD compatibility is a good idea.

leafo commented 7 years ago

I don't have an ipad (or any tablet) for that matter, to test. If someone else would like to list the issues that would be great.

pianogospel commented 7 years ago

The issue is that when I open your website using Chrome at the iPad, the api midi doesn't work (it doesn't open so Apple must be blocking it or something like that) and if I try to use Safari, it doesn't recognize what I play in the keyboard (the midi does not work).

leafo commented 7 years ago

@pianogospel are you trying to use a midi device connected to your ipad or just the onscreen keyboard?

pianogospel commented 7 years ago

I'm using a digital piano (casio) connected with a midi cable on Ipad and the Ipad doesn't recognize the digital piano on sightreading.training, in chrome or safari.

Marcos

leafo commented 7 years ago

@pianogospel thanks for the information. Do you know of any other websites that use MIDI and works with that combination. I'm not aware if the Web MIDI API is available for iPad.

I found this when googling: https://itunes.apple.com/us/app/web-midi-browser/id953846217?mt=8

pianogospel commented 7 years ago

Hi, thanks for the tip. I tried the app. I tried these things: I logged in your page with the app directly by itself, like a web browser, with safari and keeping web midi browser opened on background and with chrome keeping web midi browser opened on background, but in none of these tests the midi connection was recognized, unfortunately. I never found any web site that use midi and works with Ipad, only softwares. Thanks for all. Marcos

aik099 commented 6 years ago

The Pianu website (https://pianu.com) is connecting to the MIDI keyboard on Computer. It doesn't connect on iPad (any browser). Use https://pianu.com/detect-my-keyboard page to see if MIDI keyboard is detected.

I guess the same problem with them.

aik099 commented 6 years ago

As far as I understand on iPad the apps (e.g. Piano Partner 2 or Garageband) can connect to a MIDI keyboard via Bluetooth or cable, but Web MIDI interface of the Web Browser used on Computer isn't implemented.

So if there would be a browser wrapper app, that can connect to MIDI keyboard and open website that uses Web MIDI API it all can work.

aik099 commented 6 years ago

Ah, that what Web MIDI Browser app is. It did work with Pianu website, but not with https://sightreading.training/ unfortunately.

leafo commented 6 years ago

I'm going to close this since there are no plans to make a native app version to support iPad. Maybe one day chrome for iOS will support midi.

Alternatively, it works great on android devices with an USB OTG device.

aik099 commented 6 years ago

@leafo , please reopen. No need for a native app. At least please make it work via Web MIDI Browser app.

From the code it looks like it should be working, but maybe you're not handling case, when Browser provides several .inputs, but one of them doesn't work?

I'll try to setup website copy locally to see what's failing.

aik099 commented 6 years ago

I'm totally stuck trying to get this project running locally, because:

@leafo , maybe you can open https://sightreading.training/ from within Web MIDI Browser and see what happens. You can use Firebug Lite to make console.log and such messages to be shown.

leafo commented 6 years ago

@aik099 I don't own any apple devices so I have no way to test it. The code here works fine with many midi inputs and outputs. It's likely the API of the webmidibrowser isn't consistent with Chrome. There doesn't appear to be any documentation on the webmidibrowser page

If it's possible for you to run the site with a browser console open so you can collect error messages paste them here that would be really helpful.

aik099 commented 6 years ago

It doesn't show any errors. So either navigator.requestMIDIAccess call was successful, but haven't returned any devices or it wasn't successful, but error message wasn't shown.

Maybe you can add some console.log calls along the way so that it's clear if we'll getting to the point of requesting MIDI access at all?

aik099 commented 6 years ago

Since Web MIDI Browser doesn't have a Console I had to:

  1. proxy your website through my development website
  2. inject FireBug Lite in a result

That had undesired effect of Service Worker failed to register. Not sure if that is breaking MIDI detection.

leafo commented 6 years ago

The service worker is optional, it's okay if it doesn't work. If you think it's blocking the page from working at all then I can disable it temporarily.

aik099 commented 6 years ago

Yes, please disable it. Also I'm having other JS errors on the page (while I'm logged-in) and they might blocking MIDI detection (or any other JS code on the page):

Probably service worker it attempting to cache everything and stumbles upon unsupported data formats.

leafo commented 6 years ago

@aik099 alright, I've temporarily removed it from the site. You may have to clear some caches to make sure it's not being used

aik099 commented 6 years ago

No sw.js errors now, but still no MIDI devices detected.

Could you please add some console.log and similar debug output along the way of MIDI-related activities so that I could compare what happens during wired and bluetooth MIDI connection?

aik099 commented 6 years ago

@leafo , any updates on this?

leafo commented 6 years ago

@aik099 I added some console.log (and a console.warn) to list the midi devices or report that there are no midi devices.

aik099 commented 6 years ago

I'm now seeing this:

Each time I attempt to press "Select device" button in top right corner I'm getting 2 more of "Script error. (,0)".

Then the MIDI devices are detected, but attempt to auto-select 1st one fails and that prevents attempts to communicate with 2nd device.

The Web MIDI Browser app creates 2 devices:

Regardless the fact, that Session 1 device is always returned as available device it fails to respond to .open() method call (the returned promise is never resolved).

I guess you need to add more output, than "Script error. (,0)" to determine what actual error is happening. Likely you're calling some methods on 1st available MIDI input and one of calls fails with an exception that isn't handled.

leafo commented 6 years ago

Script error. (,0) isn't something that comes from our code, it's probably coming from the browser. Do you have access to the console inside of this browser? It might be easier for you to manually type in some of the lines of code to see which line is causing the error.

Here's how MIDI gets accessed from selecting an input:

  1. requestMIDIAccess is used, the result is stored in a variable called midi. This is the object from the browser. https://github.com/leafo/sightreading.training/blob/master/static/js/components/app.es6#L42
  2. When selecting a MIDI input, a helper method uses [...midi.inputs.values()] to put all browser MIDIInput objects into a plain javascript array (spread operator is transpiled ahead of time): https://github.com/leafo/sightreading.training/blob/master/static/js/components/app.es6#L207
  3. The UI for the picking the midi input iterates over the array returned from the helper method above (stored in the prop midiOptions) https://github.com/leafo/sightreading.training/blob/master/static/js/components/midi_selector.es6#L19
    • The only field from the browser's MIDIInput object used to render the UI is .name, for example: midiOptions.map(option => option.name)
  4. After selecting an input, a callback function is stored in input.onmidimessage to listen to MIDI events (where input = midiOptions[idx_of_selected_input]) https://github.com/leafo/sightreading.training/blob/master/static/js/components/app.es6#L221

If all of that succeeds, then all the pages can receive midi input

aik099 commented 6 years ago

I've put window.onerror = ... callback in there and I only see that some error (no message) happened on line 0 and col 0. Very strange really.

Since from all console.log statements you've added I'm only seeing MIDI inputs: one, then I guess the next thing, that happens fails.

I know for sure, that .values() call works fine, because I'm using it on another project. Then something after that fails.

Since I can't (no really manual for it) get a project building locally my only hope for you to add more debugging output.

aik099 commented 6 years ago

According to https://blog.sentry.io/2016/05/17/what-is-script-error article the Script error is actually error message text, that is sent, when JavaScript error originates from different origin. Ha, that must be because I'm proxying your website though my website to get FireBug Lite injected and that's why I'm not getting actual error.

Could you please connect FireBug Lite from https://getfirebug.com/releases/lite/1.2/ page based on URL parameter, e.g. https://sightreading.training/?debug=1 would do this. This way I'll be able to both see your console. messages and actual error text.

aik099 commented 6 years ago

I've updated my proxying code and now I'm seeing actual JS error, that happened:

"Error: " "TypeError: midi.inputs.values()[Symbol.iterator] is not a function. (In 'midi.inputs.values()[Symbol.iterator]()', 'midi.inputs.values()[Symbol.iterator]' is undefined)" "/static/main.min.js?1529953093" 1 26644 Object line=1 column=26301 sourceURL="..."

Something must be wrong with midi.inputs if the .values method isn't defined. It does work in WebMidi project (see https://github.com/djipco/webmidi/blob/master/src/webmidi.js#L1029).

aik099 commented 6 years ago

Here is how WebMidi project is doing iteration over .values() iterator: 

var inputs = midiAccess.inputs.values();
for (var input = inputs.next(); input && !input.done; input = inputs.next()) {
    promises.push(input.value.open());
}

I haven't found matching code in your project. Likely you're using some of ES6 iterator stuff, that either isn't converted to ES5 analog during JS compilation OR you don't have polyfill for it to support other browsers.

aik099 commented 6 years ago

@leafo , do my findings help in detecting where bug comes from?

aik099 commented 6 years ago

Any updates?

aik099 commented 6 years ago

Any updates?

aik099 commented 6 years ago

@leafo , where you able to make https://github.com/leafo/sightreading.training/issues/8#issuecomment-400975385 proposed change?

skalter commented 4 years ago

This would be very nice to integrate and get into production if this will get things working on ipad. I assume this would still require the Web MIDI app?

aik099 commented 4 years ago

@skalter , I have't checked on iOS 13 (maybe Safari has Web MIDI support now), but likely the Web MIDI Browser would still be a requirement.

AmolosRicha1 commented 4 years ago

Yes this would be nice. Since I use a very complicated setup to make it display on Ipad. I use Rasperry Pi connected to the usb Midi and use a firefox browser with a Plugin to make midi work. The Raspberry Pi auto connects on boot to the wireless and starts a vnc server. Then i VNC over wireless . But it‘s a useful setup since you got a very small linux computer at hand, which enables me to use synths and daws.