Explain Why Interaction Profiles Exists

It seems to be somewhat more focused on using the interaction profiles than explaining the why...

It would be good to have something in there describing why interaction profiles exist / why this pattern is used.

I'm actually a little unsure of the reason that an interaction profile path has to be specified. I understand why it can be; but what of the case where you know you want e.g. /user/hand/left/input/trigger/value, but you don't know what hardware will be used. As it stands it seems like you have to run through all the known profiles, in case one of them is supported. But if a runtime comes back with a new extension profile, you won't find it. There's no way to query which profiles a runtime or device offers, or within a profile to query what paths it offers. Is this a conscious design choice?

This was a conscious design decision, let me try a quick summary here.

It was based on a few assumptions:

VR input devices will vary enough that we can't standardize a correct "abstract device" without harming innovation
Developers should have the power to easily access any specific detail of the exact device they are testing with
Developers shouldn't have to specify the bindings for controllers that they haven't tested with or be forced to try to enumerate all possible devices
Runtimes in the future should have enough information to lie to the app in a way that allows a user to play a game with an input device the game developer has never seen (might not even have existed when the game was compiled)
Only return values to the app that the app expects, therefore avoid "list all availlable devices"-style APIs. This causes major issues as a developer today can write a nice switch statement based the returned list, but that becomes brittle over time as we introduce new devices.
Have the app provide enough information to make it possible for runtimes to create a user rebinding system, similar to what Steam offers. This is why actions need human readable names.

(There's probably a few more)

The fundamental flip is that instead of the apps asking "give me the truth of what is available" the API is designed for the app to tell the runtime "here's the actions that drive my game, and I have tested it on this 3 specific devices, and I suggest you set up the bindings like this:"

For example if you create an application and suggest bindings for /interaction_profiles/oculus/touch_controller your app will work just fine if the user has a /intaction_profiles/facebook/touch_controller_pro, but your app will just be told the user has a touch_controller. (I guess this can be viewed as a compatibility mode). It should also work on a wide set of other controllers. But once a developer wants to utilize unique features of, say, Touch Pro, they can just add bindings for that IP as well. (Say the new haptics)

Now, while it would be possible that every runtime understands every interaction profile which would make an app work if they suggest bindings for a single IP no matter which one, in practice there are some headsets that require you suggest certain device specific interaction profiles. But other than that the development flow was designed to be:

Start your application by only suggesting bindings for the specific input device you are developing and testing with
Incrementally add interaction profiles as you test with new devices and you're not happy with the default bindings or want to access device specific features
Unless forced to avoid suggesting bindings for controllers that you have not tested with
Always call xrSuggestBindings with all the controllers, don't try to dynamically pick the right one. This gives you the best shot at longevity as future runtimes will have more information to do good rebinding
If your application can function with few inputs, use simple_controller as a fallback

Some of this is redundant with the above but I thought I'd post my little writeup.

It also helps to remember, that suggesting bindings for an interaction profile serves as the "I test with this code path" signal
It's not useful to go through and suggest the same approximate thing for every controller, only suggest for what you test: to confirm that you test it (and that your app won't break if we tell you that's the controller you have), and to make sure that the bindings make sense for that controller.
The interaction profile system is striking a balance: As a standards body, we would really rather you provide only generic stuff, and let the runtime map it for your controllers. But, developers want to be able to say for sure how things are mapped on a given controller, don't want to leave it entirely to the runtime. So, we let the app tell us what it would like, on which controllers it cares most about. In turn, we say that we'll map things as best we can on other devices, but we'll never tell you a "current device name" that you haven't affirmatively offered to us.
Ideally, the user is in final control of how actions are mapped, but we won't confuse you if they remap things: it will always look to the application as if suggested bindings are being followed. The only exceptions are very narrow and clearly called out - the localized input name stuff and controller models. (Not many runtimes let you remap manually right now but we hope that will change.)

The underlying principle is, provide the runtime with high-level information as early as possible, so it can make a decision (or let the user decide) how best to map actions to available hardware. (Assume that your action localized names and action set localized names will appear in an input binding GUI like in SteamVR: that's what the intent is, and that is actually what happens when running on SteamVR's OpenXR runtime.) And, because we can't conformance-test apps, make the app tell the runtime what data it expects and won't crash with. We will only tell the app things that it would have received when testing.

(We could have had a "get controller name" function that is open-ended returning any string, not for use in setting up input. Older APIs have had things like this. But somebody would switch on that, checking only the two controllers they have access to, and then when a new controller comes out, their app would break, even though "technically" they weren't supposed to switch on that string and were supposed to expect an arbitrary string there. So now there's an appcompat issue, which means either the app is broken in the future, or everybody has to carry around app-specific compatibility code for that one app if it's important enough. This is not a hypothetical situation, it actually happens with older APIs.)

See also https://github.com/KhronosGroup/OpenXR-Guide/blob/main/chapters/goals_design_philosophy.md which is a more complete writeup on some interaction profile stuff.

Thanks @lionleaf @rpavlik. I think this gives me enough. If I'm to paraphrase, it seems the app-profile relationship here is a one-way graph, intended to prevent situations where a device/app combination without a usable mapping occurs, so it maximizes compatibility at the expense of flexibility.

It's not really that. It is one way, in that the runtime won't give back anything that the app didn't suggest, but it's mostly to maximize compatibility, flexibility, and user choice, at the expense of theoretical-flexibility on the app side that would be inherently untestable. (If you can test all the devices, then suggest for all the profiles and keep updating your app!)

There is the assumption that the runtimes are more likely to be continuously developed and updated more than applications: it's easy to update a runtime to add mappings, it's hard to find the right people and time and rights and... to get a new build of an app.

Having app-specific compatibility code in a runtime really sucks and does not scale. My OpenXR-Guide writeup above talks about it better.

OK, here's what I would expect as the process a dev would follow for designing their actions, etc. Note that interaction profiles are pretty late in the process. I also wrote up a quick example of how to use sub-action paths for a teleport mechanic, since sub-action paths are one of the more complicated aspects of the interaction system, but teleport is a pretty easy example for showing why they exist.

OpenXR App interaction design process:
- Focus first on actions and action sets.
  - Which contexts can a user take actions in? (These start to form your action sets.)
    - Initial examples might be "gameplay" and "menu", but they can become more complex.
    - You can have multiple action sets active at once, so you can "overlay" controls e.g. when the user is holding a certain kind of tool or object.
  - What actions can a user take in your app?
    - Examples might include "grab", "teleport", "interact".
    - You also associate pose tracking of controllers, etc. with actions and action sets, so you might have a pose action for each item that might be held, plus an action for aiming/selecting when nothing is held.
      - There are different pose identifiers: they have unique purposes and are meant to allow easy use across controllers.
      - "grab" pose actions are defined to be useful in rendering things you hold in your hand like a cup or mug.
      - "aim" pose actions are defined to be useful for pointing and aiming.
      - See the description and diagram here: https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#semantic-path-standard-pose-identifiers
      - Note that new standard pose identifiers are added by extensions, also mentioned there. They are not yet universal but were added in response to feedback from developers, to reduce the amount of "if this is the current interaction profile, then use this transform" code required.
    - Each action should be in a single action set, in general. If that same action is needed in multiple contexts, it should be in an action set that can be used as one of multiple active action sets used in those contexts.
    - Do not duplicate actions just because they might be done with either hand: have a single "teleport" action, not one for each hand. (Telling the difference between hands is in the next design step.) This also applies for actions where you do not care which hand performed it: "open menu" perhaps.
      - When you suggest bindings for these actions, suggest both a left and right hand action, if it makes sense. (In some cases, a controller set might only have a useful button on one hand or the other. This is totally fine.)
  - For which actions do you need to know which hand performed the action? This is to determine which actions need "sub-action paths" specified at creation
    - For example, a "teleport" action will cast a teleport target from the same hand that triggered the teleport action.
    - When you create the teleport action, specify it as having subaction paths of /user/hand/right and /user/hand/left. Specify those same subaction paths for a pose action that you will use to cast the teleport target.
  - In cases where you will have multiple action sets active at once, which ones should take priority?
    - This should be pretty clear. e.g. your "gameplay_driving" action set probably has priority over the general "gameplay" action set.
- Only once the actions and action sets are designed, do you think about interaction profiles.
  - Is your set of interactions sufficiently small that you can use the khr/simple_controller?
    - This is the case in a lot of applications, especially where interaction is more physically-based.
    - If this is true, then suggest bindings for simple controller. This will give a great fallback on basically all runtimes.
  - What devices will you be actively testing with and verifying before release?
    - These are the only ones you should suggest bindings for! interaction profiles with suggested bindings are more than just "how I want my input mapped": they are a signal to the runtime that it can safely tell you things as if it were using that device, and your app will operate as intended.
    - What controller will you be developing with primarily? Suggest bindings for that controller first.
    - Later, if you test with other controllers, you can add additional suggested bindings.
Example of using subaction paths at run time for a "teleport" feature, where you aim your teleport while holding a button, then the teleport occurs when you release the button.
- You should have a "teleport" boolean action, as well as a pose action (probably associated with the "aim" identifier). Both these actions should have both left and right hand specified as sub-action paths.
  - You may want to create an action space for each sub-action path for that pose action ahead of time, and store them so you can look them up by sub-action path. Alternately, you can create the action space later, when you know which sub-action path will trigger the teleport, if you want.
- Poll teleport action state each frame, without specifying a sub-action path.
- When it starts being true:
  - Query the action state for each of the sub-action paths, to see which one became true (which hand to use to teleport.)
  - Locate an action space created with that same sub-action path to compute and draw the teleport target. (Time used for xrLocateSpace should be predictedDisplayTime.)
  - C as long as the teleport action state on that sub-action path is true.
  - Continue polling the teleport action state each frame on that sub-action path.
    - While it is still true, continue locating that action space and re-computing the teleport target
    - When it goes to false (is released), use the last computed teleport target to actually teleport.
Why bother with sub-action paths?
- having "teleport_left" and "teleport_right" actions is an alternative, but not a great one:
  - It muddies the semantic meaning: those are actually the same action, just different sides. So representing them as different actions means lower-quality data available to the runtime.
  - It does not scale easily to additional inputs beyond just one for right and one for left
  - it isn't easy to use in a rebinding GUI: the button on the left and right controllers both mean the same thing, they are just associated with different devices (typically). Except that it won't necessarily be devices (this works with controller-less hand tracking too, and anything else people can invent)
    - I believe the related, though not identical, SteamVR Input 2.0 feature is called something like "filter by device". It is intended for the same purpose, but sub-action paths fix some flaws discovered in the API concept used by SteamVR.
  - So if they are the same action just done differently, "teleport" on /user/hand/left is not a separate action but a sub-action. Hence, sub-action paths.
- The exact same (pseudo) code above works on systems with two controllers that can both perform "teleport", systems with only one controller, and systems where only one controller of the two can perform teleport for whatever reason in your input scheme.
  - This means that all systems will use the same tested code path in your app: there is not a "common" vs "less common" (and thus "less tested") code path, there is just one used by all systems.
  - This, deep down, is a major motivation of a lot of the trickiest design decisions in OpenXR: How to make sure that the less common cases, or the future system cases, or the accessibility and user preference cases, go through the same well-tested code path as the most common hardware you used in development.
    - For example, if something might return more than one item, we want to make sure you have a loop there rather than just looking at the first element, so that your code works no matter how many items are returned.
      - If a function might return only multiple values but it only ever returns one when you test it, you'll only look at one value, and you won't be able to test the multiple values case even though the spec says you should be prepared for it.
      - So, we will try to design the API in such an instance so that even the common, simple case returns more than one item, and so the obvious way to handle the output of that function is to loop through the list of items returned, regardless of the size of that list. We want the easy and obvious way to write your code to also be the way that will be successful in the long term across many devices and runtimes. We want your effective codebase coverage of your testing to be high, without having to try lots of devices and runtimes or time-travel into the future.

OK, there, I swear I am done editing this for now. That was stuff I meant to write up for a long time now, and have been saying verbally in a lot of situations. Hopefully that can be turned into some useful text, whether in the tutorial or in some other OpenXR developer docs.

Thanks, Ryan! :D

I'd also add another resource that might be helpful if you haven't seen it. I wrote an example using the Action system with plenty of comments in the Meta OpenXR SDK, you can download it here: https://developer.oculus.com/downloads/package/oculus-openxr-mobile-sdk/

And I'm specifically referring to XrSamples/XrInput/Src/main.cpp once you unzip that package :)

And finally, another resource that might be helpful is looking at how Steam input works, the API was designed to allow runtimes to build a similar rebinding flow as Steam offers today (try opening the binding options from Half-Life: Alyx for instance)

Please post any feedback on Interaction Profile parts of the new Chapter 4 text here.

tentative closing pending feedback

KhronosGroup / OpenXR-Tutorials

Explain Why Interaction Profiles Exists #30