search

qmk / qmk_firmware

Open-source keyboard firmware for Atmel AVR and Arm USB families
https://qmk.fm
GNU General Public License v2.0
17.74k stars 37.97k forks source link

[Feature Request] Hi-res scrolling #17585

Open b- opened 2 years ago

b- commented 2 years ago

Feature Request Type

Description

Windows and Linux support a "resolution modifier" report for scrolling, which allows finer scrolling instead of just "ticks." Specifically, one can do smooth per-pixel scrolling. It also seems to use fewer resources, possibly because of the larger descriptors.

Unfortunately, it seems that there's virtually no public code or documentation available. The following is everything I've managed to gather in terms of documentation and examples:

This almost certainly requires MOUSE_EXTENDED_REPORT to be enabled based on my testing.

I started mocking up something here: https://github.com/b-/qmk_firmware/tree/resolution_multiplier

However this is the first time I've ever gotten this close to USB or to HID descriptors.

I haven't tested my above code yet, either, though I will as soon as I get the chance.

Anyway, I'm posting this here to help me keep track of this, and of my progress (or lack thereof). I expect to edit this post frequently.

pandages commented 2 years ago

I think this is a really cool concept. Is it for using a rotary encoder as a mouse-style scrollwheel? Could it potentially be mapped to any keys?

drashna commented 2 years ago

Taking a look at "wheel.doc", I'm not sure how practical this is. Namely, this appears to need custom drivers to support it (namely for registry settings). And those dirvers need to be signed, or the overall security of the system degraded (driver signing disabled, which requires secure boot to be disabled).

It's definitely possible to implement, but I'm not sure that it is worth the effort here.

b- commented 2 years ago

I can say with confidence this works without drivers other than what's bundled with Windows and Linux.

hid-remapper allows one to set a custom scroll multiplier and it works without drivers just fine (tested on Windows 10 and 11, and on KDE Plasma). This what I'm stealing the HID collection tree from, so if I can simply do what it's doing this would work just fine.

b- commented 2 years ago

I think this is a really cool concept. Is it for using a rotary encoder as a mouse-style scrollwheel? Could it potentially be mapped to any keys?

QMK already has scroll wheel support via mouse keys. You can just add MOUSEKEY_ENABLE = yes to your rules.mk and use KC_MS_WH_UP and KC_MS_WH_DOWN as keycodes for mouse wheel scrolling.

My goal here is to add a resolution multiplier for wheel scrolling, as that will allow finer precision than an ordinary scroll wheel allows for (and this works without drivers on Windows Vista and up, or Linux).

My motivation is for DRAG_SCROLL on my Ploopy Classic to work at a finer precision.

Another idea is to put some kind of tiny trackball on top of a mouse, much like Apple's discontinued Mighty Mouse.

I mean, you can already do all of these things with QMK as is, but currently you either need to set the DPI very low (or use an encoder with big notches), because by default on Windows one single scroll wheel notch moves three lines.

By setting the resolution multiplier to 40 (120/3) Windows will instead scroll one line for every wheel notch (or KC_MS_WH_DOWN). Setting it even lower will scroll less than one line per notch, which is supported in many applications such as web browsers and image editors.

b- commented 2 years ago

Microsoft's own mice that took advantage of this had "notchless" scroll wheels whose encoders were much more sensitive than what's standard today.

Example: https://www.cnet.com/reviews/microsoft-wireless-laser-mouse-7000-2008-review/

b- commented 1 year ago

I bought a Microsoft Surface Precision Mouse to see how Microsoft implements hi-res scrolling, and gathered some data here. This is on a Windows 11 machine, with the mouse attached over USB.

First, I installed the relevant drivers for the mouse, and then disabled "flick" scrolling in the new settings panel for the mouse. I also inverted the scroll wheel for preference, but I don't think that's particularly relevant.

Then I unplugged the mouse, and opened the USB port in Wireshark (with the other devices filtered out on the USBPcap level).

Once the capture began, I plugged in the mouse, holding it upside down to prevent any tracking activity. Then I played with the scroll wheel, scrolling down a little bit, then up a little bit, then down a little bit again (varying the speed as I went along). I added comments to the packet capture to mark the end of the initialization handshake, and each scroll direction change.

What I saw that I found particularly curious is that this mouse actually doesn't send a count descriptor in its wheel reports. It instead varies the resolution multiplier with the direction and magnitude of the wheel motion, and that's the only information about the wheel. Still, I'd think it's enough.

Although I also should note that when I tried using the mouse on my Pinebook Pro in KDE Plasma (Wayland), neither Firefox nor Chrome nor native Qt apps had any hi-res scrolling support out of the box. I didn't try testing further.

The spec says the mouse really shouldn't send hi-res scrolling information until the host sends a command to switch into a fancy enhanced mode, so perhaps libinput doesn't have support for this mouse yet to send that command?

I also tried in Chromium and Firefox on Windows Subsystem for Linux, in Wayland, and hi-res scrolling was supported. I think WSL uses libinput as well (but of course Windows is abstracting the input packets), and I know for certain that libinput does normally support hi-res scrolling where allowed.

Attached is my wireshark capture. hrscroll-surface_precision_mouse-with-comments.zip

jfedor2 commented 1 year ago

Hi guys. I think I should be able to help.

As you've already noticed there's some stuff in the report descriptor that's required for this to work. I'd recommend you start with the exact same descriptor HID Remapper uses (the relevant top level collection, you can change the report ID) and only start tweaking it to your needs once you confirm hi-res scroll is working. There's some stuff that's not part of the spec that Windows expects anyway, like the resolution multiplier using 2 bits even if 1 would be enough. Also the logical collection semantics might not be obvious if you're new to HID report descriptors.

The important part is the feature report, which is different from input/output reports. It's a sort of configuration variable that the host can read and write. I don't know if QMK currently uses feature reports for anything.

In our case the feature report will contain two values - the multipliers for horizontal and vertical scroll. In theory there can be multiple levels of the multiplier, but since in practice the operating systems choose the highest value anyway, I think it makes sense to only have two levels, 1x (legacy behavior) and the maximum, which is 120x. This is what it looks like in the report descriptor:

Logical Minimum (0)
Logical Maximum (1)
Physical Minimum (1)
Physical Maximum (120)

What this means is that if the relevant bits in the feature report are set to 0 (which should be the default), the multiplier will be 1x. If they are set to 1, it will be 120x. When it's 120x, it means 120 units of scroll correspond to 1 normal/legacy scroll "tick", giving you higher resolution.

To recap you need:

  1. The relevant parts in the report descriptor.
  2. The ability to receive and send feature reports.
  3. Adjust what you send in the normal input reports in the horizontal/vertical scroll fields, depending on the values set by the host in the feature report.

An operating system that doesn't know about high resolution scroll will not touch the resolution multiplier in the feature report and will expect 1 unit=1 legacy tick. An operating system that knows about high resolution scroll will write 1 (or whatever the logical maximum is) to the resolution multiplier values in the feature report and will then expect 120 units=1 legacy tick (or whatever the physical maximum is).

It's also a good idea to detect USB mounting event and reset the resolution multiplier to the default of 1x, otherwise you run into issues when you reboot from an OS that knows about hi-res scroll to one that doesn't.

I'm not familiar with QMK source code, but I'll be happy to answer any questions you might have about HID so please don't hesitate to ask for clarification on any of this.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged properly or other activity occurs. For maintainers: Please label with bug, in progress, on hold, discussion or to do to prevent the issue from being re-flagged.

b- commented 1 year ago

Hey, thanks for working on this! A lot has gotten in the way of me making more progress, and it's hard to debug this sort of thing when you consider that everything I know about USB and HID I learned from working on this :) but I'll see if I can poke at your branch at some point anyway.

Just a random thought: what if you swap the wheel and AC pan entries in the descriptor? I wouldn't expect that to fix anything, but if it changes the axis upon which we get rogue outputs it might point us in the right direction? (Pun not intended, I promise!)

b- commented 1 year ago

Wow! That's awesome!! Thank you!

I expected it to be about as picky as you're seeing -- that's about what I've experienced with @jfedor2/hid-remapper

Most mouse driver software that implements hires scrolling seems to turn it on and off based on what window your input is going into. Logitech is horribly picky about this and has a very limited whitelist of .exe files that it enables hires scrolling on. Microsoft Intellipoint Mouse and Keyboard Center lets you pick which apps let you do this, though.

Honestly part of my interest in a "hardware" hires scroll thing like this is to help put pressure on Microsoft to better implement their spec :)

iicurtis commented 10 months ago

I don't really understand HID setup, so I am hoping someone can help clarify some things. Looking at the Microsoft wheel.docx, they give the following collection hierarchy:

Mouse Application Collection
    Mouse Logical Collection
        Pointer Physical Collection
            Buttons (Input Report 0x01)
            X (Input Report 0x01)
            Y (Input Report 0x01)
            Logical Collection
                Resolution Multiplier (Feature Report 0x02)
                Wheel (Input Report 0x01)
            End Logical Collection
            Logical Collection
                Resolution Multiplier (Feature Report 0x02)
                AC Pan (Input Report 0x01)
            End Logical Collection
        End Physical Collection
    End Logical Collection
End Application Collection

However, the current QMK hierarchy is noticeably different. Instead of Application -> Logical -> Physical, the QMK hierarchy goes Application -> Physical. 

Mouse Application Collection
    Pointer Physical Collection
        Buttons (Input Report 0x01)
        X (Input Report 0x01)
        Y (Input Report 0x01)
        Wheel (Input Report 0x01)
        AC Pan (Input Report 0x01)
    End Physical Collection
End Application Collection

I attempted to modify usb_descriptor.c to add an additional logical map and the corresponding Resolution Multiplier collections, but I am now realizing this was a bit too naive. My mouse breaks after updating the HID report structure, which probably needs to be reflected elsewhere.

BanchouBoo commented 10 months ago

Someone was working on a PR for this feature but I can't seem to find it anymore, what happened to it?

DejayRezme commented 7 months ago

I just wanted to add my user experience with an apple magic mouse I used a while at work:

It was the best scrolling experience I've ever had! It is a touch mouse of course with a smooth trackpad on top. But you could very smoothly scroll the window vertically 1:1 and it felt like the perfect mouse wheel. No indents or "3 lines at a time". Just smooth fast responsive scrolling. And you could "fling" the window content for fast scrolling and then stop it again like it was a weighted wheel. And you could precisely scroll and lift your finger without the scrolling jerking or drifting. I still miss that mouse... it's one of those little things that you don't know you want until you try it (like a third dedicated mouse button for your ring finger).

So this could also be implemented with two of those "Cirque GlidePoint Circle Trackpads" one for mouse pointer and one for scrolling.

PS: I got part of this functionality back with a firefox extension called "ScrollAnywhere" that uses the middle mouse button (ring finger for me) to grab and drag / flick instead of the super awkward thing windows does. But this is not going to work if your middle mouse click is pressing the mouse wheel down.

thejevans commented 5 months ago

@b- I just got a Ploopy Adept hoping to have precision/hi-res scrolling, and I'm saddened to see it's not an option. I'm fairly green with QMK, but this is a priority for me, so I'm happy to help any way I can. Are you stuck on anything in particular?

b- commented 5 months ago

@b- I just got a Ploopy Adept hoping to have precision/hi-res scrolling, and I'm saddened to see it's not an option. I'm fairly green with QMK, but this is a priority for me, so I'm happy to help any way I can. Are you stuck on anything in particular?

As you may have noticed I haven't touched this in a long while. However, I would love help solving this once and for all!

I want to warn you, though, I was kind of stuck on everything! I understand USB HID descriptors and the like at a cursory level, having learned virtually everything I know about it from trying to solve this. I also am not very experienced in C/C++, either.

That said, the main blockers (at least for me) were that I was getting "weird" cursor movement and it appeared that some input buffer on Windows was getting filled without being properly flushed. The pointer would fly into the corner as soon as I touched the ball, and then the computer would start making short beeps (not alert sounds, but actual beeps) a few times a second.

This happened just from moving the cursor around while hidpi was enabled in the firmware. I think I mostly used a patch from someone else's comment in this thread or something like that, but I recall encountering similar issues when I tried to crudely craft the descriptor and reports myself.

I assume that some of the messages being generated were malformed, and I don't really remember where I left off with troubleshooting that. I have a hunch that the descriptor and reports weren't matching or something, but I'm not sure and I don't really have any backups of what I was trying besides anything remaining on some lingering branch at https://github.com/b-/qmk_firmware.

I know I was looking at Wireshark messages, and I even purchased a Microsoft Sufrace Precision mouse that does support hidpi scrolling to record and compare its own descriptor and reports, but I don't remember where I went from there.

(It doesn't help that I lost my job around the time I stopped working on this and have still been unable to find full time employment since then. But I suppose that gives me more time now to potentially pick it back up...)

If you come up with anything interesting please ping me, either here and/or on the QMK Discord server (my Discord username is @bri9). I'd be more than excited to test with my old Ploopy Classic.

Oh, and I'm sure I already mentioned it more than once in this issue thread, but @jfedor2 has an excellent repository for a different HID device firmware, hid-remapper, with a working and customizable scroll multiplier. This might be a good reference?

Good luck!

evan-goode commented 5 months ago

Hi, I'm also interested in getting this working eventually, though I don't have much time to work on it right now. Like @b- I also tried implementing this feature a couple years ago, but I can't remember what I was stuck on.

I recall these articles by Peter Hutterer (libinput developer) were helpful to me:

Sorry for the noise!

BanchouBoo commented 4 weeks ago

Someone was working on a PR for this feature but I can't seem to find it anymore, what happened to it?

Realized I still had a local copy of the PR so I checked the log to find the username of the person working on it, some light searching led me to this post on reddit. @drashna did anything ever come of this? If desired I could upload it, but wouldn't want to do so without getting permission first.

image

justinjradi commented 3 weeks ago

Hey. I'm not familiar with QMK, but I just implemented this using both ST's USB Library and TinyUSB. It seems to be working fine for the hosts I've been able to test on (Windows 10 and 11) but I'm still kinda confused. Here's my report descriptor for reference:

0x05, 0x01,        // Usage Page (Generic Desktop Ctrls)
0x09, 0x02,        // Usage (Mouse)
0xA1, 0x01,        // Collection (Application)
0x05, 0x01,        //   Usage Page (Generic Desktop Ctrls)
0x09, 0x02,        //   Usage (Mouse)
0xA1, 0x02,        //   Collection (Logical)
0x85, 0x01,        //     Report ID (1)
0x09, 0x01,        //     Usage (Pointer)
0xA1, 0x00,        //     Collection (Physical)
0x05, 0x09,        //       Usage Page (Button)
0x19, 0x01,        //       Usage Minimum (0x01)
0x29, 0x05,        //       Usage Maximum (0x05)
0x15, 0x00,        //       Logical Minimum (0)
0x25, 0x01,        //       Logical Maximum (1)
0x95, 0x05,        //       Report Count (5)
0x75, 0x01,        //       Report Size (1)
0x81, 0x02,        //       Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)
0x95, 0x01,        //       Report Count (1)
0x75, 0x03,        //       Report Size (3)
0x81, 0x01,        //       Input (Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position)
0x05, 0x01,        //       Usage Page (Generic Desktop Ctrls)
0x09, 0x30,        //       Usage (X)
0x09, 0x31,        //       Usage (Y)
0x95, 0x02,        //       Report Count (2)
0x75, 0x08,        //       Report Size (8)
0x15, 0x81,        //       Logical Minimum (-127)
0x25, 0x7F,        //       Logical Maximum (127)
0x81, 0x06,        //       Input (Data,Var,Rel,No Wrap,Linear,Preferred State,No Null Position)
0xA1, 0x02,        //       Collection (Logical)
0x85, 0x02,        //         Report ID (2)
0x09, 0x48,        //         Usage (Resolution Multiplier)
0x95, 0x01,        //         Report Count (1)
0x75, 0x02,        //         Report Size (2)
0x15, 0x00,        //         Logical Minimum (0)
0x25, 0x01,        //         Logical Maximum (1)
0x35, 0x01,        //         Physical Minimum (1)
0x45, 0x78,        //         Physical Maximum (120)
0xB1, 0x02,        //         Feature (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile)
0x85, 0x01,        //         Report ID (1)
0x09, 0x38,        //         Usage (Wheel)
0x35, 0x00,        //         Physical Minimum (0)
0x45, 0x00,        //         Physical Maximum (0)
0x95, 0x01,        //         Report Count (1)
0x75, 0x08,        //         Report Size (8)
0x15, 0x81,        //         Logical Minimum (-127)
0x25, 0x7F,        //         Logical Maximum (127)
0x81, 0x06,        //         Input (Data,Var,Rel,No Wrap,Linear,Preferred State,No Null Position)
0xC0,              //       End Collection
0xA1, 0x02,        //       Collection (Logical)
0x85, 0x02,        //         Report ID (2)
0x09, 0x48,        //         Usage (Resolution Multiplier)
0x95, 0x01,        //         Report Count (1)
0x75, 0x02,        //         Report Size (2)
0x15, 0x00,        //         Logical Minimum (0)
0x25, 0x01,        //         Logical Maximum (1)
0x35, 0x01,        //         Physical Minimum (1)
0x45, 0x78,        //         Physical Maximum (120)
0xB1, 0x02,        //         Feature (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile)
0x35, 0x00,        //         Physical Minimum (0)
0x45, 0x00,        //         Physical Maximum (0)
0x75, 0x04,        //         Report Size (4)
0xB1, 0x01,        //         Feature (Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile)
0x85, 0x01,        //         Report ID (1)
0x05, 0x0C,        //         Usage Page (Consumer)
0x0A, 0x38, 0x02,  //         Usage (AC Pan)
0x95, 0x01,        //         Report Count (1)
0x75, 0x08,        //         Report Size (8)
0x15, 0x81,        //         Logical Minimum (-127)
0x25, 0x7F,        //         Logical Maximum (127)
0x81, 0x06,        //         Input (Data,Var,Rel,No Wrap,Linear,Preferred State,No Null Position)
0xC0,              //       End Collection
0xC0,              //     End Collection
0xC0,              //   End Collection
0xC0,              // End Collection

// 144 bytes

Other than making sure that you follow the correct collection hierarchy in the report descriptor, the key to getting it to work is to use the feature reports correctly. Again, here's the format of the feature report:

typedef struct __attribute__((packed))
{
    uint8_t report_ID;
    uint8_t scroll_resolution : 2;
    uint8_t pan_resolution : 2;
    uint8_t padding : 4;
} ResMultiplierReport;

Since the resolution multiplier usages have a logical range of 0 to 1, I interpret them as really just saying whether the wheels are in high-resolution mode. I haven't played around with them too much, but I think the physical maximums are the actual resolution.

Here's the important part: In addition to preparing for a Set_Report request with the feature report, you also need to configure your device to respond to a Get_Report request for the feature report. The Get_Report request preceeds the Set_Report request. I think this is totally undocumented and I only figured it out by analyzing the Wireshark capture that Bri shared. I highly recommend everyone take a look at the capture themselves.

What I'm still confused about is why? If there was just one Get_Report request, I'd think it's the host just asking for the capabilities of the device, and then the Set_Report requests that follow are just configuring the device. But for some reason every Set_Report request seems to be preceded by a Get_Report request. In all my testing so far, all the Set_Report requests I've seen just echo what my device sent in response to the Get_Report request. How many times have I said Get_Report and Set_Report by now?

Right now, I just have the feature report I respond with hard coded with both scroll_resolution and pan_resolution set. But if and when the host sets values other than what I initially reported, am I supposed to echo that or just respond with the same thing I was sending before?

A couple more notes: