Notes get stuck while quickly playing with USB MIDI keyboard controller

[rsta2]

I open this separate issue, because we will probably have to deal with this problem for a while.

Problem description

When quickly playing on an USB MIDI keyboard controller, connected to a Raspberry Pi 1-3 or Zero (Raspberry Pi 4 is not affected), notes can get stuck, i.e. the respective tone is played forever, even after releasing the key. This can be reproduced with continuous voices (e.g. organ) playing rapid glissando for a while on the following keyboard:

• M-AUDIO Keystation Mini 32 Mk3

Workaround

The problem may be reduced or solved with the following option in the file cmdline.txt on the SD card:

usbspeed=full

This switches the whole USB from high speed (480 MBps) to full speed (12 MBps). The most USB MIDI controllers do only support full speed, so this is generally not a problem, but a few other keyboards do require high speed mode to work. The workaround does not work there and cannot applied by default.

[rsta2]

There is a kernel image for the Raspberry Pi 2 attached, which is based on a quick hack. It does use the FIQ (Fast Interrupt Request) for the USB driver. It has a higher priority then the normal IRQ and using the FIQ may result in a more precise timing on the USB. The DWHCI USB controller of the Raspberry Pi 1-3 and Zero requires 8000 interrupts per second (one per microframe) for an accurate timing, which is not easy to fulfill.

@probonopd Can you please try this kernel7.img with the rapid glissando test on the RPi 2 with the M-AUDIO Keystation Mini 32 Mk3? Does the problem remain with this image? I would like to know, if it worth doing the (big) effort to generally support the FIQ for the USB in Circle.

This kernel image is fragile, because of the quick hack. I may even crash, then try again. Serial MIDI does not work with it, because it used the FIQ so far too, and there is only one FIQ supported on RPi 1-3 and Zero. Please test with and without the fast=true option in cmdline.txt. You will not see the bank select and voice change messages on the screen, but the LCD should work.

kernel7.zip

[probonopd]

Thanks @rsta2, using kernel7.zip I can still get stuck notes with and without fast=true. I also get them by e.g., pressing 4 keys at the same time repeatedly for some time.

Cannot seem to reproduce this using my Raspberry Pi Pico tester.

Could it be something else` Maybe the M-AUDIO Keystation Mini 32 Mk3 is errorously sending the same key down message more than once (insufficient debouncing?) under some circumstances?

Can you think of a quick way to log all MIDI messages to a file or serial port?

On the Raspberry Pi 4 without usbspeed=full and without usbpowerdelay=1000 in cmdline.txt I get Canot set device address (19) when I attach the M-AUDIO Keystation Mini 32 Mk3 to either port, and I cannot play any notes. Using only usbpowerdelay=1000, I do indeed get no stuck keys on the Raspberry Pi 4.

[rsta2]

@probonopd Thanks for testing! I did not expect this behavior. So using the FIQ will probably not help for the M-AUDIO Keystation Mini 32 Mk3. I have to think about a logging method and will come back with a solution.

So perhaps we have to add usbpowerdelay=1000 to cmdline.txt in any case. Some devices require a longer time until the power is stable. usbspeed=full does not have a function on the Raspberry Pi 4.

[rsta2]

@probonopd With this patch, you can redirect all screen output to the serial interface at GPIO14 at 115200 Bps 8-N-1. The screen is not initialized with this. Serial MIDI does not work, of course.

serial-monitor.zip

[rsta2]

I'm working on the FIQ support for the USB driver of Raspberry Pi 1-3 and Zero. I guess there is still room for timing improvement. But this will take longer and requires many tests.

@probonopd For now I would like to know, if there is something I could do for MiniDexed (e.g. support multiple Dexed instances, or "EDIT VOICE" menu in the LCD/Rotary Encoder UI)?

[probonopd]

@rsta2 multiple Dexed instances would be tremendous. Eventually this could lead to support for TX816/TX802 style Performance sysex.

Voice editing via the LCD/Rotary Encoder UI would also be nice, but I think we should focus on multiple instances first.

[rsta2]

@probonopd OK, I will work on supporting multiple Dexed instances now. Yes, Performance sysex should be loadable in the end, but for now I will use an .ini file to configure the different instances.

[probonopd]

Here is one more way that I can get stuck notes:

• RPi 2, HDMI
• Keystation Mini 32 MK3
• usbspeed=full in cmdline.txt
• Set up 5 TG TG 1: normal TG 2: OMNI, detune 7, stereo shift halfway to R TG 3: OMNI, detune -7, stereo shift halfway to L TG 4: OMNI, detune 15, stereo shift all the way to R ** TG 5: OMNI, detune -15, stereo shift all the way to L
• Play 6 notes and keep holding them while playing around with Pitch Bend key/wheel
• Repeat a couple of times. Sometimes notes get stuck No fast playing is needed but the Pitch Bend are arriving relatively fast from the MIDI controller.

Note to self: Try to reproduce this issue with something else than the Keystation Mini 32 MK3 (e.g., A non-USB MIDI keyboard plus a USB-Midi converter)

[rsta2]

I think we should not concentrate on the RPi 2 too much. Unfortunately I guess, it's too slow, at least without fast=true. I'm still trying to find a concept for improving the USB driver with using the FIQ. My first approach seemed to work first, but now I was running into problems. It will take longer. I know, that the problem is there, but I do not have a solution at the moment.

[probonopd]

Thanks, no hurries. I just wanted to provide this additional information in case it is helpful for isolating the root cause.

[rsta2]

@probonopd I finished my tests with the modified USB driver, which now can use the FIQ (Fast Interrupt Request) to handle interrupts from the USB, especially the SOF (Start-of-Frame) interrupts, which occur 8000 times per second. The FIQ has a higher priority than the IRQ, which was used before, and can interrupt even running IRQ handlers. I suppose the timing of the SOF interrupts is very precise now.

If this has been the problem before, it may work now with your keyboard controller. It would be great, if you would make another test with the attached kernel image for the Raspberry Pi 3. Please add fast=true to your cmdline.txt and remove usbspeed=full. If this works, please try without a cmdline.txt. Thanks in advance!

kernel8.zip

[probonopd]

Thanks @rsta2. I can still get to stuck notes when playing 6 keys at the same time repeatedly on the Raspberry Pi 3 with fast=true for a while (while having 4 TGs active using the default performance).

[probonopd]

Interestingly I cannot reproduce this issue when using a serial MIDI keyboard attached to a self-made USB MIDI adapter using an Arduino. So it seems that the issue is only with some USB MIDI devices. It would be interesting to know whether my M-Audio Keystation Mini 32 Mk3 is the lonely exception in this regard, or whether some other USB MIDI devices suffer from the same.

If you have any idea how I could test/compare the behavior of the M-Audio Keystation Mini 32 Mk3, please let me know.

[rsta2]

@probonopd Thanks for testing! I don't think your keyboard is the only one, which shows this problem with this USB driver, but I guess much more will work, than will not work. Unfortunately the only method to find out, if some keyboard works, is to take it and try it. The USB MIDI support in Circle is available for five years now, and I heard about problems with only three different keyboards (including yours) in this time.

[probonopd]

@rsta2 thanks for the hard work you keep putting into this, I really appreciate it. Don't feel obliged to spend excessive amounts of time on this keyboard just because I happen to have it. On the other hand, if you think it helps development, I'd be happy to lend it to you, postage paid.

[rsta2]

@probonopd Thanks for your offer! Unfortunately I have currently no idea, what I could change on the USB driver to improve compatibility. If this changes, I will come back on it.

[probonopd]

New hardware to play with: Using a Nektar Impact LX61+ MIDI controller I also have to use usbspeed=full or else I get very many stuck keys on a RPi 2.

[rsta2]

Have you tested this keyboard on the RPi 3 with the USB FIQ test kernel above?

[probonopd]

Yes, unfortunately the stuck notes still occured.

Using a Nektar Impact LX61+ MIDI controller even with usbspeed=full I get stuck keys (very, very occasionally) on a RPi 2 when running 4 TGs plus effects.

[rsta2]

I have a cheap used M-AUDIO Keystation Mini 32 Mk3 here now. I can reproduce the problem. It seems with using the FIQ for the USB driver, the 0x202 errors do not happen any more, but the 0x402 errors (data toggle error) still happen. I'm trying to found out, why? Connected to a Raspberry Pi 4B, there weren't any problems, as expected.

[probonopd]

Unfortunately it's not limited to cheap ones. So my hope is if it can be fixed for the Mini 32 Mk3 then it will also be fixed for others, like the Nektar Impact LX61+.

[rsta2]

This is also my hope. But first the 0x402 errors must be fixed, which is difficult enough. :(

[rsta2]

The problem occurs on a Raspberry Pi 3B with Raspberry Pi OS too. I can demonstrate this with the M-AUDIO Keystation Mini 32 Mk3. When pressing and releasing the MOD button, the keyboard sends B0 01 01 B0 01 02 B0 01 03 ... B0 01 7F B0 01 7E B0 01 7D ... B0 01 01 B0 01 00 B0 01 01 B0 01 02 and so on. When doing this for a while, one B0 01 NN is missing in the row of MOD messages. I also compiled the Raspberry Pi OS on my own and added a printk() message, which is visible in /var/log/messages, when a data toggle error occurs. This happens, when the expected MOD message is missing.

Also see:

https://github.com/NXPmicro/mfgtools/issues/76#issuecomment-453966001

[probonopd]

Wow. So we are tripping over a bug in the Raspberry Pi hardware/firmware?

[rsta2]

Seems so. As I understand the USB 2.0 spec. the problem is not the data toggle error itself. These errors can happen for example with electronic interference, but it should be possible to ignore them without losing data. Currently I do not know, how this is possible on the RPi 1-3.

[probonopd]

Looks like you have spent way too much time on this nasty issue already. Especially if this can be reproduced on Raspberry Pi OS: Do you think we should contact someone from the Raspberry Pi team?

[rsta2]

I think I'm not ready with this yet. At least the usbspeed=full workaround should work. If I remember well, it didn't do this for you. I still need some time to check this. We could open an issue in the RPi "firmware" or "linux" repo, but the problem should be 100% proved, before doing so. I have tested this with my own-built kernel so far only. Still need some time.

[rsta2]

See: https://github.com/raspberrypi/linux/issues/4986#issue-1199763175

[rsta2]

There is a solution for this problem for Raspberry Pi OS. I have to see, if I can implement this for Circle.

[rsta2]

Even with maximum polling speed, I get lost MIDI events with Circle (every 2 minutes or so). This seems to be very critical with these (low-buffered) devices. What we can do is, calling Dexed::notesOff() or Dexed::panic(), when such condition is detected. I also have to add another option for cmdline.txt (e.g. usbboost=true). That's another workaround, but I cannot generally implement this behavior, because it is not appropriate for other applications.

And, because the FIQ is needed for the USB in this configuration, the serial MIDI interface will be less prioritized on the RPi 1-3. We have to test, if serial MIDI is quick enough with the IRQ then, or if we get lost MIDI events via the UART then too.

[probonopd]

Does Linux with dwc_otg.nak_holdoff=0 work in a different way than Circle in this regard?

[rsta2]

It's not possible to copy the exact behavior of the Linux driver. The drivers are much too different. From the info from the RPi Linux repo, I can only read, that I have to poll as quick as possible. That's what I do now in my tests.

[probonopd]

I am really just guessing here, but could it be that they are using interrupts differently to rely less on polling?

[rsta2]

They must poll the device too. When I say "polling", I do not mean, that the Circle driver does polling. It uses interrupts, of course. Otherwise it could not work in the background. But the USB device (the keyboard) has to be requested for data, by sending "polls". That's relatively complicated and cannot be explained in short.

I'm trying to adapt the behavior of their driver, which uses both FIQ and IRQ, in my revised driver. But this cannot be done 100%. The Linux driver is cryptic. I'm glad, that it does work at all with Circle.

The bad thing is, that you cannot see, if an USB MIDI keyboard has small buffers (like the Keystation) or larger ones (like my AKAIpro LPK25). You need to test it. Or use a RPi 4.

[probonopd]

How do you know that the Keystation has small buffers? (I would like to find out for my Impact LX61+.)

[rsta2]

The developer from RPi wrote, that this must be the case.

[probonopd]

I see. Let's go the usbboost=true route then, I'd say.

[rsta2]

OK, I also see no other way at the moment.

[diysynth]

Hello, I just want to mention that I also get stuck notes with minidexed quite often running on my Raspberry Pi 3B with different usb controllers. First I tested MiniDexed with a Yamaha hardware sequencer via an usb/midi adapter and got stuck notes very often. Disabling the midi clock on the output helped a bit, but still notes got stuck. Same problem occurs when using that usb/midi adapter and sending notes from a daw. I also tried connecting a Keystep 37 via usb to the Pi and I got stuck notes especially when using the arp or sequencer. The stuck notes them self wouldn't be the biggest problem. More of a concern for me is that minidexed ist not responding to common note off or midi panic messages. I tried many different ways, but the only way I get rid of the notes is to change the patch/voice. It would be great if minidexed would respond to those note off/velocity 0/midi panic messages and even greater would be to have a midi panic hardware push button, that can be connected to the pi's gpio. Anyway, minidexed is a wonderful project and I am having alot of fun with it. I planed to make some form of opensource enclosure for it. But yeah, those stuck notes are a little bit of a showstopper. I will try this usbspeed command tomorrow.

[probonopd]

Welcome @diysynth. Will open a separate ticket for the All Sound Off (CC 120, value 0) and All Notes Off (CC 123, value 0) MIDI messages.

[probonopd]

@diysynth which exact usb/midi adapter are you using?

[diysynth]

@diysynth which exact usb/midi adapter are you using?

Miditech Midilink Mini

[probonopd]

Interesting. With my self-built one I don't get stuck keys, https://github.com/probonopd/MiniDexed/issues/48#issuecomment-1079971242.

[diysynth]

Interesting. With my self-built one I don't get stuck keys, #48 (comment).

Directly sending MIDI to there serial input or a diy usb adapter? I couldn't sleep before testing it again. Is the usbboost=true option rsta2 was talking about already enabled? I added this to cmdline and I couldn't reproduce the failure again. At least not with reaper. I tested it with arpeggiators on two different midi channels at absurde rates (up to 360 bpm) and the only time I got stuck notes was when I changed the time selection while the arps were running. But that was maybe a reaper problem since it skipped the note off bit or so. Reaper and other daws send a sound/note-off message when pressing the stop button, would be very nice if you could add this midi cc code someday. Vielen Dank!

[probonopd]

Directly sending MIDI to the serial input never produced any stuck notes. Which makes sense since the stuck notes are a USB thing.

When using my selfmade adapter, I also didn't get stuck notes so far. Not all USB devices are prone to produce those.

usbboost=true is not implemented yet.

Reaper and other daws send a sound/note-off message when pressing the stop button, would be very nice if you could add this midi cc code someday.

Some day? I've implemented it, it's waiting for your test :-) See https://github.com/probonopd/MiniDexed/pull/127#issuecomment-1104520055

[diysynth]

Will test it tomorrow evening. Thanks alot!!

[rsta2]

usbboost=true will come with the next release of the Circle framework (around end of May), which is used by MiniDexed. I suppose this will improve the situation very much, but some MIDI events may still get lost with it on the Raspberry Pi 1-3 and Zero. An open question is, if the USB MIDI driver should automatically generate an "All sound off" MIDI event, in case it detects an error on the USB?

[probonopd]

Would probably be better. Ideal would be to have a config option for it.

[diysynth]

So I tested it. Note Off CC and Sound Off CC both instantly mute the sound when they receive a 0. This just makes it so much better do deal with stuck notes. Pressing the stop button on my daw now stops all the sounds from MiniDexed, that's very convenient. Notes still stuck occasionally but mostly when doing things like disabling midi arp plugins while the transport is runnig. Pretty sure that is not MiniDexed fault then. So it works pretty good and stable, at least for me. Thanks alot!

[Kirtai]

Could this be MIDI Running Status again?