Make mapings action centric rather than input centric

[netpro2k]

While thinking through options for #2 I found myself wanting to organize things by action rather than by input event. It seems to be a more natural way to think about the mappings and lends itself better to a configuration UI in the future.

For example, instead of:

AFRAME.registerInputMappings({
  default: {
    common: {
      dpadleftdown: "action_snap_rotate_left",
      dpadrightdown: "action_snap_rotate_right",
      dpadcenterdown: "action_teleport_aim",
      dpadcenterup: "action_teleport_teleport"
    },
    "vive-controls": {
      menudown: "action_mute"
    },
    "oculus-touch-controls": {
      xbuttondown: "action_mute"
    },
    daydream: {
      menudown: "action_mute"
    },
    keyboard: {
      m_press: "action_mute",
      x_press: "action_mute",
      q_press: "action_snap_rotate_left",
      e_press: "action_snap_rotate_right"
    }
  }
});

You would instead do:

AFRAME.registerInputMappings({
  default: {
    action_snap_rotate_left: {
      common: "dpadleftdown",
      keyboard: "q_press"
    },
    action_snap_rotate_right: {
      common: "dpadrightdown",
      keyboard: "e_press"
    },
    action_teleport_aim: {
      common: "dpadcenterdown"
    },
    action_teleport_teleport: {
      common: "dpadcenterup"
    },
    action_mute: {
      "vive-controls": "menudown",
      "oculus-touch-controls": "xbuttondown",
      "daydream-controls": "menudown",
      keyboard: ["m_press", "x_press"]
    }
  }
});

It's not a huge difference but better enforces the "actions not buttons" way of thinking, makes it easy to see how each action is mapped across devices, and almost directly mirrors what you imagine a configuration UI might look like for it. You obviously still would need ways to index this data by device as well (for binding the events, or displaying info tips on a controller), but this seems to make more sense as the developer facing schema.

Thoughts?

[dmarcos]

I personally prefer to see each controller and how the buttons are mapped in the application all in a single place. It serves as a form of manual and is closer to the way I think. I grab the controller and I think: ok the trigger will do this, the touchpad that... I don’t think of an action and how that action will map on each controller

[netpro2k]

I agree that as a user I think about "what do the buttons on my controller do" but I would argue that we want to nudge developers away from this thinking, because there are a good number (and growing) of input devices a user of your application may end up using. By pushing a developer to think about actions first you help prevent them from designing around how to do things with the particular input device they happen to be using and think more about what actual things a user needs to be able to do in their application (what not how).

It's also worth thinking about how additional metadata would be added to actions later. For example adding a human readable label for a configuration UI. With an action centeric schema this slots in nicely since each action is only represented once:

action_snap_rotate_left: {
    options: { label: "Rotate Left" },
    common: "dpadleftdown",
    keyboard: "q_press"
},

With an input device centric schema you are forced to either duplicate data on each button mapped to that action or more likely creating an additional mapping from actions to action options:

common: {
    dpadleftdown: "action_snap_rotate_left",
},
keyboard: {
   q_press: "action_snap_rotate_left",
},
action_options: {
   action_snap_rotate_left: { label: "Rotate Left" }
}

The later certainly feels a bit more awkward.

Ultimately this does back out in part to personal preference, as you can represent the same information in both ways but this serves as a subtle chance to push best practices.. It's also worth noting the action centric schema more closely mirrors how Unity, Unreal, and Steamworks represent their input system configuration.

[fernandojsg]

When developing it I agreed with @dmarcos that grouping by controller is more easy to understand when you are creating the mapping. But I was thinking also about adding new features to the mapping and one is more descriptive labels as @netpro2k proposed. So I have mixed feelings here. For example we should need to handle left/right hand (or any other number of devices). How should we do that?

action_snap_rotate_left: {
    common: "dpadleftdown@left",
},

common: {
    'dpadleftdown@left': "action_snap_rotate_left",
}

common: {
    dpadleftdown: {
        mapping: "action_snap_rotate_left",
        hand: "left"
    } 
}

Although we should probably fix the issue https://github.com/fernandojsg/aframe-input-mapping-component/issues/2 and then going back to this conversation.

[dmarcos]

This is how I would do it. If both hands emit the same event nothing changes (tangential, but prepending the word action seems redundant):

 common: {
    dpaddown: "snaprotate",
}

if there's different actions for each hand you can replace the string with an object:

 common: {
    dpaddown: { left: "snaprotateleft", right: "snaprotateright" }
}

[dmarcos]

For additional free form metadata you could have an option additional field actions to keep the mappings clean that I expect to be the most important part and something you want to tweak more often:

AFRAME.registerInputMappings({
  actions: {mute: {label: "Shhhh"}, teleport: {label: "Go there"} ... }
  default: {
    common: {
      dpaddown: "snaprotateleft",
      dpadrightdown: "snaprotateright",
      dpadcenterdown: "teleportaim",
      dpadcenterup: "teleport"
    },
    vive-controls: {menudown: "mute"},
    oculus-touch-controls: {xbuttondown: "mute"},
    daydream: {menudown: "mute"},
    keyboard: {
      mdown: "mute",
      xdown: "mute",
      qdown: "snaprotateleft",
      edown: "snaprotateright"
    }
  }
});

[fernandojsg]

In that case maybe adding a new item to clarify the different reserved keywords?

AFRAME.registerInputMappings({
  actions: { mute: { label: 'Shhhh' }, teleport: { label: 'Go there' } ... },
  states: {
    default: {
      common: {
        dpaddown: "snaprotateleft",
        dpadrightdown: "snaprotateright",
        dpadcenterdown: "teleportaim",
        dpadcenterup: "teleport"
      },
      vive-controls: {menudown: "mute"},
      oculus-touch-controls: {xbuttondown: "mute"},
      daydream: {menudown: "mute"},
      keyboard: {
        mdown: "mute",
        xdown: "mute",
        qdown: "snaprotateleft",
        edown: "snaprotateright"
      }
    },
    painting: {},
    menu: {}
  }
});

Instead of

AFRAME.registerInputMappings({
  actions: { mute: { label: 'Shhhh' }, teleport: { label: 'Go there' } ... },
  default: {
    common: {
      dpaddown: "snaprotateleft",
      dpadrightdown: "snaprotateright",
      dpadcenterdown: "teleportaim",
      dpadcenterup: "teleport"
    },
    vive-controls: {menudown: "mute"},
    oculus-touch-controls: {xbuttondown: "mute"},
    daydream: {menudown: "mute"},
    keyboard: {
      mdown: "mute",
      xdown: "mute",
      qdown: "snaprotateleft",
      edown: "snaprotateright"
    }
  },
  painting: {},
  menu: {}
});

[netpro2k]

Having the actions enumerated at the top like this does feel like it provides a lot of the value I liked from the method I initially proposed. And separating the states feels correct as well. Perhaps we could emit a warning if you bind to an action you haven't declared?

I also do like the idea of being able to include additional details with the event on a per mapping basis:

AFRAME.registerInputMappings({
  actions: { 
    action_mute: {
      label: "Mute Mic",
      details: { // all action_mute events include these details
        foo: 1,
        bar: 2
      }
    }
  },
  states: {
    default: {
      vive-controls: {
        menudown: {
          action: "action_mute",
          details: { // action_mute events from vive will have { foo: 4, bar: 2, baz: 3 } as their details
            baz: 3,
            foo: 4
          }
        }
      },
      oculus-touch-controls: {xbuttondown: "action_mute"},
      daydream: {menudown: "action_mute"},
      keyboard: {
        mdown: "action_mute",
        xdown: "action_mute",
      }
    }
  }
});

[fernandojsg]

Cool, so we agree on the actions and states grouping. Regarding the details even though I find it interesting and probably useful, I would leave it out of the first implementation as I said before to try to make it simple as possible and once we get to a point when we agree that there is no way to map an event without this feature or hacking the rest of the code we could include it. But thinking on the current main apps we have (a-blast, a-saturday and a-painter) I can't find a nice use case for that. In any case, having the structure as discussed, it's open to include this kind of things easily in the future, so let's start with the easiest if you agree and we'll keep growing from there ;)

[netpro2k]

Yep I agree, was not suggesting implementing it now, just thinking through potential future use-cases to see how they might slot in.

[fernandojsg]

Thinking again about the use of actions, if we want to use it to set labels for each action, we could eventually want to set the tooltips positions for each label... so we will need to generate another level under each action for each controller

AFRAME.registerInputMappings({
  actions: { 
    action_mute: {
      label: "Mute Mic",
      tooltips-positions: {
        vive-controls: {x: 0, y: 1.2, z: 2.1},
        oculus-controls: {x: 0, y: 1.1, z: 2.1},
        daydream-controls: {x: 0, y: 1.2, z: 0.1}
      }
    },
    action_volumeup: {
      label: "Volume up",
      tooltips-positions: {
        vive-controls: {x: -1, y: 1.2, z: 2.1},
        oculus-controls: {x: -1, y: 1.1, z: 2.1},
        daydream-controls: {x: -1, y: 1.2, z: 0.1}
      }
    }
  },
  states: {
    default: {
      vive-controls: {
        menudown: {
          action: "action_mute",
          details: { // action_mute events from vive will have { foo: 4, bar: 2, baz: 3 } as their details
            baz: 3,
            foo: 4
          }
        }
      },
      oculus-touch-controls: {xbuttondown: "action_mute"},
      daydream: {menudown: "action_mute"},
      keyboard: {
        mdown: "action_mute",
        xdown: "action_mute",
      }
    }
  }
});

Another option is to include that info directly on the mapping itself, as the labels are common to all the actions but the position of the tooltips are specific for each controller:

AFRAME.registerInputMappings({
  actions: { 
    action_mute: {
      label: "Mute Mic",
    },
    action_volumeup: {
      label: "Volume up",
    }
  },
  states: {
    default: {
      vive-controls: {
        menudown: {
          action: "action_mute",
          tooltip-position: {x: -1, y: 1.2, z: 2.1}
        }
      },
      oculus-touch-controls: {
        xbuttondown: "action_mute"
      },
      daydream: {
        menudown: {
          action: "action_mute",
          tooltip-position: {x: -1, y: 1.2, z: 2.1}
        }
      },
      keyboard: {
        mdown: "action_mute",
        xdown: "action_mute",
      }
    }
  }
});

Please note that for example oculus-touch-controls doesn't have a tooltip-position so it will use the default tooltip-position for that button.

But if we have several states and we don't want to duplicate the positions on each one, maybe we'll need a higher level config controllers at the same level of actions and states where we could configure the default positions for each controller's tooltips, and maybe other options in the future.

AFRAME.registerInputMappings({
  controllers: {
    vive-controls: {
      tooltips: {
        xbuttondown: {x: -1, y: 1.2, z: 2.1}
      }
    }
  },
  actions: { 
    action_mute: {
      label: "Mute Mic",
    },
    action_volumeup: {
      label: "Volume up",
    }
  },
  states: {
    default: {
      vive-controls: {
        menudown: {
          action: "action_mute",
          tooltip-position: {x: -1, y: 1.2, z: 2.1}
        }
      },
      oculus-touch-controls: {
        xbuttondown: "action_mute"
      },
      daydream: {
        menudown: {
          action: "action_mute",
          tooltip-position: {x: -1, y: 1.2, z: 2.1}
        }
      },
      keyboard: {
        mdown: "action_mute",
        xdown: "action_mute",
      }
    }
  }
});

I know, we've been talking about keeping things simple and don't overengineering it, but I believe this is an use case we already have in apainter for example and we could add easily to the rest of apps we have that uses controllers.

@netpro2k @dmarcos what do you think?

[fernandojsg]

We could even use these controllers section we could use it eventually for many things to reduce the logic on the app: change the colors depending on the state, change even the controller, the vibrations per action, the sounds it does, etc.

[netpro2k]

I do think tooltips themselves shouldn't necessarily be part of the input mapping, but it does seem like a good idea to be able to store application (or component) specific data on the mappings/actions that things like a tooltip component can query.

Seems like a good set of rules about how this data will be merged so that you can have defaults and overrides is all you would really want to prescribe here, as the number of possible types of metadata one might want to add seems endless.

The first example you provided seems to be the most straightforward and in line with what has already been proposed. I had always envisioned that action data to be largely freeform anyway, with different components pulling data from it. Even the labels in my example would just be a convention used by whatever components make up the config/help screen, but are completely transparent to the input system itself.

[machenmusik]

It seems to me that controller model specific characteristics like positions of tooltips should live closer to the controller components or the controlled models themselves. A little tricky to specify as really you want to show a line from the right point on the model to the tooltip, not just the tooltips alone