keyboard input is finnicky

thenfour commented 1 year ago

a few oddities:

a ding sound is produced when typing in Reaper
32-bit builds running in sandbox in Reaper, no keyboard input is registered
according to netpoet, Ableton Live also has broken keyboard input.

I suppose fixing 32-bit reaper would solve things.

There are clearly multiple problems at work here.

Keyboard input can be handled either via VST messages, or by the native WindowProc. I would assume handling the VST messages is the most reliable, but I have observed that I don't always receive these calls (when precisely?). Handling natively is also a problem because I'm not receiving WM_CHAR messages, even though I do receive WM_KEYDOWN messages. Is Reaper not calling TranslateMessage? Is there something I am missing?

One more issue is that in the Reaper x86 sandbox, my VST window was not receiving any keyboard messages at all; the window never received keyboard focus. Adding a SetFocus() call upon mouse clicks helped this.

thenfour commented 8 months ago

some external resources:

https://forum.cockos.com/showthread.php?t=116894 <-- window user data trick for accepting all keyboard input https://www.kvraudio.com/forum/viewtopic.php?t=565934 <-- the VST keyboard opcodes are not well supported. supported also here : https://forum.renoise.com/t/developing-vst-effect-unable-to-intercept-keystrokes/27693 https://forum.cockos.com/showthread.php?t=179042 <-- the main issue: WM_GETDLGCODE

General info:

* Keyboard input seems like a lot of black magic in VSTs. And it's also important to bridge it properly across Winapi -> VST -> Imgui.
* Different hosts also behave differently. At some point, Renoise was working fine, but Reaper was not sending certain keys (arrows, enter, et al). It casts doubt of a perfect robust elegant general solution.
* Plugin sandbox processes do input differently than native. At some point I noticed that in Reaper, a native plugin is in a "window", while a sandboxed one lands in a child of a dialog.

Observations:

* Renoise was not sending any keyboard messages at all. I was not receiving `WM_KEYDOWN` in my own windowproc.
* Renoise has a "enable keyboard" control which our plugin was hiding (#58 created).
* It's hard to even tell what Reaper is doing. Sometimes I'm getting VST's native `effEditKeyDown`, sometimes not (i think in plugin host i was never getting it).
* At first Reaper was sending `WM_KEYDOWN` but not `WM_CHAR`.
* Reaper was also giving an audible "ding" sound. I don't recall why, but proper keyboard handling fixed it.

there were several issues causing keyboard issues:

the biggest one, `WM_GETDLGCODE`

adding this to our native windowproc,

    case WM_GETDLGCODE:
        return DLGC_WANTALLKEYS;
    }

makes Reaper to send all keyboard input to our window, including WM_CHAR. Apparently within a dialog, this is needed otherwise you don't get all keys.

`SetFocus` was not called

The window had no focus; SetFocus needed to be called in WM_LBUTTONDOWN.

`VstEditor::onKeyDown` was not being called at all

This is unreliable anyway and the solution was to tell the host I didn't use the key

Characters were sometimes double-processed

Handling both native key events and VST's onKeyDown caused this.

Reaper's magic value 0xdeadf00b

        // reaper uses this weird magic value to specify keyboard input should be sent to the window. WHAT
        // https://forum.cockos.com/showthread.php?t=116894
        ::SetWindowLongPtrW(hWnd, GWLP_USERDATA, 0xdeadf00b);

thenfour commented 8 months ago

so finally i think we can mark this closed with latest commit. tests:

* renoise x64 with x64 leveller
* renoise x64 with x86 plugin hosted leveller
* reaper x64 with x64 leveller
* reaper x64 with x86 plugin hosted leveller
* dead keys
* typing in text box
* arrow keys
* modifiers and caps lock
* no "ding" sounds
* no double characters.

all behaving.

thenfour / WaveSabre