hansmbakker commented 4 years ago

Proposal: RadialController injection input

Summary

There is a preview API to inject HID input. Currently the input types are restricted to

keyboard
mouse
gamepad
pen

It would be great if RadialController (Surface Dial) input could be injected, too. This would enable making non-compatible devices act as RadialController devices via a user-space app instead of writing a driver.

Rationale

Would enable people to write e.g. emulators for RadialController without having to resort to driver development (which is a barrier)
For the usecase of emulating a RadialController device with another hardware device, I added background input to the must haves. This is because if you want to build a "bridge" app for a certain hardware device, that app should be supportive to other apps by running in the background. It should not need to be in the foreground to be able to inject RadialController input.
Windows shows / hides the Wheel page in the Settings app based on whether a Wheel device is connected. To support that functionality, an emulator / bridge app should be able to indicate when the app is connected
I put the request in this repository since there is no good alternative where feedback feels heard

Scope

Capability	Priority
Allows developers to inject RadialController input (button presses & rotation)	Must
Allows developers to call the API when the app is in the background	Must
Allows developers to set the RadialController as present / not present	Should
Allows developers to subscribe to RadialController output events (like vibration commands from apps)	Could
Allows developers to subscribe to RadialController other functionality like firmware updates	Won't

Open Questions

If there is a better place to request this, please let me know. I just do not feel that feedback hub entries with API proposals are responded to, they seem to get lost in the stream of other feedback.

hansmbakker commented 4 years ago

This proposal was moved from https://github.com/microsoft/microsoft-ui-xaml/issues/2548 as requested by @ranjeshj. It was put there originally since the current preview API is in the Windows.UI.* namespace.

shaheedmalik commented 4 years ago

Good idea.

hansmbakker commented 4 years ago

Added some requirements

jonaskuske commented 4 years ago

Would love to see this! It should make it possible to write a plugin for the Logitech Craft keyboard and its rotating crown that would enable support in all apps that don't use logitechs custom API/SDK, but do have support for Surface Dial right?

hansmbakker commented 4 years ago

Yes! That’s the idea.

There are more devices on the market that could act like a surface dial.

The benefit of translating the input to RadialController input is that it is standardized by Microsoft, so this way you can use your device without app developers having to use an app-specific SDK indeed.

Also, this way it is possible to build an emulator for Surface Dial in case you want to test your app for RadialController input.

hansmbakker commented 4 years ago

Developing a windows driver is another possibility (it can be a virtual hid driver in user mode) but it introduces lots of hurdles:

Steep development learning curve for people coming from .Net development
less easy development loop (compile —> install —> install —> debug)
Less easy debugging
driver development tooling is not as user friendly as .net tooling
distribution of drivers has additional requirements. Apps can be distributed easily.

feisu commented 4 years ago

Thanks for your interest – my team works on input devices like the Dial and we’re happy to see the interest here. The output vibration command part is unique for Dial compared to other input types and requires new OS support. We’re looking into a more generic input injection model for Windows, with device-specific helpers delivered here through Project Reunion.

In the meantime, if anyone has a particular Dial injection API shape in mind, please feel free to sketch it out.

hansmbakker commented 4 years ago

The output vibration command part is unique for Dial compared to other input types

@feisu-ms Understood. Is there a way that the input part could be made available before the output part is ready? That way, I believe a part of the people would be helped already.

if anyone has a particular Dial injection API shape in mind

I suppose there of course needs to be an object describing input events, which could be like this:

struct RadialControllerInputData
{
    bool ButtonPressed; //true is down / activated, false is up / untouched
    double Rotation; //delta rotation in degrees, to make it device-independent. Should be signed to distinguish CW/CCW. 

    //RadialController has support for X and Y coordinates (where the controller should be shown on the screen), but personally I never used them. It could be added to be complete.
}

Following the existing InputInjector API that would look something like this:

public void InjectRadialControllerInput(IEnumerable<RadialControllerInputData> input);

weltkante commented 4 years ago

Maybe something like the UWP MIDI API? That one also supports different kinds of data packets, both for input and output.

hansmbakker commented 4 years ago

Added a requirement about mimicking the presence / disappearing of a RadialController.

This is needed because Windows shows / hides the Wheel page (shown below) in the Settings app based on whether a Surface Dial is connected.

feisu commented 4 years ago

The output vibration command part is unique for Dial compared to other input types

@feisu-ms Understood. Is there a way that the input part could be made available before the output part is ready? That way, I believe a part of the people would be helped already.

if anyone has a particular Dial injection API shape in mind

I suppose there of course needs to be an object describing input events, which could be like this:
struct RadialControllerInputData
{
    bool ButtonPressed; //true is down / activated, false is up / untouched
    double Rotation; //delta rotation in degrees, to make it device-independent. Should be signed to distinguish CW/CCW. 

    //RadialController has support for X and Y coordinates (where the controller should be shown on the screen), but personally I never used them. It could be added to be complete.
}
Following the existing InputInjector API that would look something like this:
public void InjectRadialControllerInput(IEnumerable<RadialControllerInputData> input);

That looks reasonable. We may add output support for haptics but agreed that's lower priority. Thanks.

feisu commented 4 years ago

Maybe something like the UWP MIDI API? That one also supports different kinds of data packets, both for input and output.

Good point. Input is mostly HID based. But there are similarity and may be patterns we can leverage.

feisu commented 4 years ago

Added a requirement about mimicking the presence / disappearing of a RadialController.

This is needed because Windows shows / hides the Wheel page (shown below) in the Settings app based on whether a Surface Dial is connected.

Injection devices have its own lifetime that's tied to the lifetime of the injecting app. If no app is injecting Dial, then there's no Dial on the system and thus no Settings for it.

hansmbakker commented 4 years ago

(Also discussed with @qmatteoq)

hansmbakker commented 4 years ago

@feisu

We may add output support for haptics but agreed that's lower priority

To give a bit more context: there are other devices than Surface Dial that have a form of output as well - mine can use a LED matrix to notify the user .

hansmbakker commented 4 years ago

@feisu could you give some indication on next steps / what we can expect?

feisu commented 4 years ago

@hansmbakker This is currently in our back log. It's unclear at this moment when we will be able to get to it. I will update this thread as soon as we have some update.

hansmbakker commented 3 years ago

Is there any update on this?

stevewri commented 3 years ago

@feisu - do you have any updates to share?

feisu commented 3 years ago

Nothing specific to share at the moment. This is still in our backlog. Compared to other input modalities like mouse/keyboard/touch/pen/touchpad, the RadialController device like Surface Dial is relatively less commonly used and thus we are prioritizing accordingly.

hansmbakker commented 3 years ago

@feisu although initially I understand that point, it is a bit of a chicken-egg problem.

Currently the only device that supports the RadialController APIs natively is the surface dial itself. Other similar devices exist, using other protocols.

If you would allow us to emulate such a device (by making existing devices compatible with the RadialController APIs by using a to-be-developed injection API), then more users could make use of apps that already support a surface dial, and then it could also be more interesting for developers to add support in their apps.

feisu commented 3 years ago

@hansmbakker you made a very good point and it's very much appreciated. We will definitely take your comments into account in our prioritization.

microsoft / WindowsAppSDK