integrate controller-cursor component with cursor component

[ngokevin]

Description:

Provide out-of-the-box way for attaching laser cursor to controllers, mapping the events, creating a line.

https://github.com/bryik/aframe-controller-cursor-component

Possible API brainstorm:

<a-camera>
  <a-entity cursor></a-entity>
</a-camera>

<a-entity hand-controls="right" cursor"></a-entity>

The cursor component can perhaps detect when it's attached to a controller or not (tracked-controls), or whether its parent has a camera.

[ngokevin]

cc @bryik

[bryik]

Yeah this would be nice.

Types of cursors:

• gaze-based (Cardboard)

• controller-based (Daydream, Vive, Oculus)

• mouse-based (Desktop)

Mayognaise's mouse cursor is designed differently, but gaze and controller cursors could be integrated somehow.

For tracked controls, the positioning of the laser relative to the model presents a potential complication. Different controllers may require different positioning.

[machenmusik]

so what i do personally is have a separate controller-cursor component that listens for the new controllerconnected event and disables the camera gaze cursor when it takes over (and it uses configurable events like trigger/touchpad/thumbstick for emulating mouse down/up events). not sure it helps to completely merge into same component since controls may need different adjustments?

[ngokevin]

Merging because all of the event names, raycaster code, state is the same. The main thing different for controller is mapping to detect click and option to show a line mesh.

[machenmusik]

if the assumption is that there should only be one active cursor that should jump from camera (gaze) to controller (button press), I understand what you mean (to avoid multiple raycasts etc.) but if the plan is to allow multiple simultaneous, not sure there will be as much benefit. direct mouse click is something else really. maybe just need to allow cursor to respond to controller button events, and create a laser-pointer alternative to the typical ring representation (e.g. you should be able to use laser pointer with gaze if you like)

[ngokevin]

Perhaps make it less magic and explicit with presets:

<a-camera>
  <a-entity cursor="type: gaze"></a-entity>
</a-camera>

<a-entity hand-controls="right" cursor="type: controller"></a-entity>

Based on type, A-Frame can handle listening for VR mode, whether to laser/reticle (or even project).

[cvan]

cc @caseyyee, who's been thinking and working with responsive controller components

[machenmusik]

@ngokevin seems like we're really talking about presets...

<a-entity cursor="fuse: true; fuseTimeout: 500"
            position="0 0 -1"
            geometry="primitive: ring; radiusInner: 0.02; radiusOuter: 0.03"
            material="color: black; shader: flat">

gaze is preset for fuse:true and maybe timeout controller is preset for laser pointer geometry and/or default material, but really we might want preset for ring, cross, dot, etc. as well

would it be better for gaze-cursor and controller-cursor to exist and simply assign properties to cursor dependency as a way to share the code?

[ngokevin]

I might do that. The awkward thing I need to resolve is that our current crop of 3dof controllers primarily have trackpaddown for clicking, and 6dof controllers have triggerdown for clicking. I wish I could set the cursor to just listen to both events, but there's a event collision: 6dof controllers have their own trackpaddown events.

I could do cursor -> gaze-cursor | controller-cursor. gaze-cursor sets fuse, controller-cursor renders a laser. Though a preset property might just be as simple while avoiding API surface / abstraction hell.

presets:

presetName	properties	usage
gaze	fuse: true; upEvents: ; downEvents: ;	under camera
controllerTrigger	laser: true; downEvents: triggerdown; upEvents: triggerup	6dof controllers
controllerTrackpad	laser: true; downEvents: trackpaddown; upEvents: trackpadup	3dof controllers

[dmarcos]

What about gaze-controls and laser-controls as analog of hand-controls ?

[ngokevin]

-controls have meant controllers or code to handle pose. The cursor stuff is more of an attachment onto the controllers for interactivity.

[dmarcos]

I see laser-controls pretty similar to hand-controls. They all use daydream-controls, vive-controls... under the hood? gaze-controls they have also tracking associated to it but it's the headset / camera.

[ngokevin]

The -controls family of components handles effecting the position/rotation of entities and the family of tracked-controls/daydream-controls/gearvr-controls/vive-controls/oculus-touch-controls/hand-controls follow a pattern and build on top of each other.

The cursor components don't necessarily depend on any of those. They just attach interactivity/state on top of events that may come. So I wouldn't want to conflate them. I also don't think we need entirely separate components configure a single component; that's what properties are for, to configure. It can sometimes be hard to follow the abstractions otherwise.

Here's the API I am thinking:

<a-camera>
  <a-entity cursor="preset: gaze"></a-entity>  <!-- Gaze is the default. -->
</a-camera>

<a-entity daydream-controls gearvr-controls cursor="preset: trackpadController"></a-entity>

<a-entity hand-controls cursor="preset: triggerController"></a-entity>

[dmarcos]

I think that at least laser-controls should do more than just interactivity / state. It will position the laser on the correct position based on the controller available and will map the buttons to emit click events on other entities? I don't think it's much different than hand-controls

<a-entity laser-controls></a-entity>

That gives you a consistent laser regardless of the specific hardware or number of DOF.

[ngokevin]

I could get on board with that. That way would be to make the cursor component smarter and do controller (or lack of controller) detection) and automatically set everything. If I had my own custom controller, I could configure the position/rotation of the controller. The slight downside is increased complexity in the component to make things more magic (controller detection and auto-configuration); if it were developer-configured, it'd be simpler, less need for manual testing.

I think -controls naming though should be reserved for components controlling an entity. While it would be coupled to -controls code, the laser wouldn't be controlling anything.

[dmarcos]

Well... laser-controls would control the entity the same as hand-controls, daydream-controls, vive-controls do. The laser-controls I had in mind is pretty similar to hand-controls but in this case the model is just a line (+ the default controller model for each platform) and it scales across 3DOF and 6DOF. laser-controls can make use of another component cursor="type: laser, position: 0 0 0" that can be reused by custom controls

[ngokevin]

OK.

We'll have laser-controls which sets all the proper configurations for each controls we support. If someone does a custom controller, they can configure the cursor/raycaster component. Maybe the cursor can handle drawing the line based on the raycaster's far/origin/direction.

Maybe cursor can default to gaze configuration so we don't have to create another component.

We can also document/share code that will toggle the cursor/controller cursors depending on controllers available if devs want to set both and have everything handled for the "responsive" case.

[ngokevin]

Thinking about this API.

laser-controls

Works with 3DoF and 6DoF.

Configures cursor.downEvents, cursor.upEvents, raycaster.origin, raycaster.direction based on which controller is active.

Configure cursor or raycaster components for further customization.

No properties.

cursor

Laser's length is configured via raycaster.far.

Laser's position/rotation is configured via raycaster.origin and raycaster.direction

laserAlwaysShow if off only shows laser when interacting with objects configured by raycaster.objects

property
fuse
fuseTimeout
downEvents
upEvents
laserAlwaysShow
laserColor
laserEnabled
laserRadius

raycaster

property
far
interval
near
objects
recursive
origin
direction

extra: responsive-controls-toggler

Perhaps provided via documentation or examples

<a-scene responsive-controls>

Creates gaze-controls under camera and laser-controls. Toggles between either way depending on whether controls are active

usage

<a-camera>
  <a-entity cursor></a-entity>
</a-camera>

<a-entity laser-controls></a-entity>

[donmccurdy]

If I understand correctly, this is the chain of components injecting other components:

I'm a bit concerned that the existing -controls components expose nearly-similar APIs without actually sharing any code... if I want to use a custom model, I'm left doing this:

<a-entity vive-controls="model:false"
          oculus-touch-controls="model:false"
          gearvr-controls="model:false"
          daydream-controls="model:false"

That's a bit out of scope for this question of having a laser, so I don't think we need to address it here, but it does make me wish this laser implementation could not add onto that dependency stack.

What if we just called it, say, a laser-cursor component, which depends on cursor, but that does not automatically add all the other controls? Presumably it would have an origin and direction that can be customized by the user and passed on to the raycaster. In terms of getting the origin and direction automatically from the currently-active controls, that sounds like a job for a System and not for a direct dependency between the components IMO.

No opinion on the responsive-controls-toggler proposal for now, since that depends so much on the previous thoughts.

[machenmusik]

What I typically do these days is in a codepen here: http://codepen.io/machenmusik/pen/gWdgMy

You can see a set of components - controller-cursor (translate controller events to down/up/click), controller-model-if-present (think hand-controls abstraction of which controller, but using the native controller model instead of the low-poly hands), controller-with-cursor-if-present (which includes controller-model-if-present and ray representation, as well as disabling gaze cursor when controller connected)

Thoughts welcome

[donmccurdy]

@machenmusik very nice! That looks quite helpful.

Your controller-cursor matches my suggestion on laser-cursor I think, in the sense that it isn't responsible for adding controls itself. Although I hadn't thought of the need to manually call onMouseDown() on the cursor.

[dmarcos]

I'm not fan of components with conditionals in their name. they define logic which is the opposite of declarative

[dmarcos]

I see from the point of view of the user. The same as hand-controls I want a single component laser-controls that gives me a nice laser interface with the model of the platform I'm using so I just care about hooking up the events to my application logic. I'm sure we can have something that will work out of the box for 90% of the cases.

[ngokevin]

I had the same concerns about the towering abstraction stack, but I think of laser-controls as a convenience layer. The responsive-controls-toggler isn't really a proposal for inclusion. Just something we can provide or link to from documentation in case people want to know how.

For custom controls with laser, I see the story as:

<a-entity custom-controls="model: customModel.gltf" cursor="laserEnabled: true" raycaster="direction: 0 -0.10 -1">

AFRAME.registerComponent('custom-controls', {
init: function () {
  // Whatever controls you want to support.
  setAttribute('vive-controls', {model: false});
  setAttribute('oculus-touch-controls', {model: false});
}
});

[dmarcos]

The current implementation of hand-controls is pretty simple and I would use it as a starting point for laser-controls

[dmarcos]

What @ngokevin says :)

[donmccurdy]

As a convenience layer, I would find laser-controls most helpful if it did not BOTH provide behaviors and inject other controls. This comes back to bite users who want to have custom controls, because they now have to re-implement the laser to get that. And it pre-supposes that all future <device>-controls components will be part of A-Frame core.

If the idea (from cursor="laserEnabled: true") is that the actual laser-related logic would go into the cursor component, and laser-controls is just some syntactic sugar with no behavior of its own, then I have fewer concerns. But why not just make it a primitive, then?

[ngokevin]

The actual laser logic would be in cursor/raycaster. The extra behavior needed is to automatically line up the laser appropriate to the controller models + map which button event triggers down/up, depending on which controller is connected.

[machenmusik]

@dmarcos I was not proposing to add names with if-present to core, I just use that to distinguish that they are expected to do nothing until the controller connects - my pet peeve with hand-controls is that they default to visible even if no controllers are present, so you end up with a pair of hand models permanently stuck at origin

[dmarcos]

laser-controls would also position the origin of the raycaster depending on the model. Components that are sintatic sugar are ok to me. A different topic: primitives is something we want to deprecate due to side effects when combined with mixins and confusion when a component is both set on the entity and the primitive (you might have several attributes in the entity that refer to the same thing). As a layer of abstraction they introduce more noise than value over the entity / component model.

[machenmusik]

@ngokevin you can definitely add the controller-cursor behavior into cursor, although then you need to know if it is a controller-cursor or a gaze-cursor, not sure that is worth trying to jam into same component as opposed to having them separate. However, IMO there should be a way to (1) provide custom appearance for ray/line (2) have the ray/line stop at intersection as opposed to tunneling straight through to the full length of the ray/line (and maybe place some intersection graphic) -- really this starts to be like teleport controls (straight laser vs. parabolic, and hit target appearance)

[ngokevin]

@dmarcos Let's not start a different topic in this issue.

@machenmusik Yeah, that's doable. Not much logic needed to differentiate gaze vs controller. You can just attach both types of events and they won't interfere. The laser can be customized through various properties, or really, you can disable the laser and provide your own mesh. And stopping the laser or projecting on the intersection point can be done later too.

[donmccurdy]

The actual laser logic would be in cursor/raycaster.

Ok cool. That, or as @dmarcos said in Slack, if laser-controls uses laser-cursor for the laser behavior, that is similarly fine.

The point on primitives makes sense.

If possible, it would be preferable for laser-controls to not have specific knowledge of each <device>-controls component model for determining offsets, but to get that through some component or system API. Let's not plan on every <device>-controls being in core.

[machenmusik]

@ngokevin w.r.t. Laser customization of appearance - at some point you realize that the entity with controllers and model is related to but distinct from the ray/line indicating the laser with its own position and orientation offsets, they can't be the same entity... If the idea is to force people to make their own mesh, it is complicated by the need to understand the underlying object hierarchy to do it right...

[machenmusik]

As an example, maybe I want the ray to be thin and lower opacity until it is intersecting with something at which point it starts pulsing and appearing to be more solid... which along the lines of cursor could be done as animation if things were structured properly

[machenmusik]

Maybe laser-pointer should just be the ray/line, and cursor can auto-switch if it sees controller connected event, which should never happen if attached to camera to be a gaze cursor?

[donmccurdy]

Brainstorming here:

AFRAME.registerComponent('laser-cursor', {
  dependencies: ['cursor'],
  schema: {
    origin: { type: 'vec3' },
    direction: { type: 'vec3' },
  },
  init: function () {
    var el = this.el;
    var data = this.data;
    el.addEventListener('controllerconnected', (e) {
      if (!data.origin) { el.setAttribute('laser-cursor', 'origin', e.detail.defaultOrigin); }
      if (!data.direction) { el.setAttribute('laser-cursor', 'direction', e.detail.defaultDirection); }
    });
  },
  // ...
});

I don't think I have a good sense of how to allow users full control over the line... seems like they will need to re-implement at least one component there. Ideally we can limit that case to only re-implementing a single component.

[ngokevin]

I don't think we need a separate laser-cursor component. Potential confusion vs. cursor or laser-controls, and not much meat to separate from cursor.

Possible ways to configure a line, just provide your own mesh. You can still offset it with mesh.position or mesh.rotation.

<a-entity cursor="laserEnabled: false" line="radius: 0.1; color: blue; opacity: 0.8"
               event-set="_event: mouseover; line.color: green; line.opacity: 1">
</a-entity>

Alternatively, the default laser mesh is accessible through something like getObject3D('cursorlaser').

[machenmusik]

Cursor/raycaster need to be dormant if no controller at all right? (Else you have stuck raycaster permanently firing from origin)

[ngokevin]

Good point. laser-controls could attach the cursor only once a controller is connected.

[machenmusik]

The big nuisance I see at the moment is having to repeatedly specify the objects property for raycaster - once for gaze and then twice more (once for each controller)

Maybe we need one objects pool and multiple raycaster instances that reference it

[ngokevin]

Not just twice? One for gaze-based and one for controller-based? I don't think it's much work to define twice, but that's also where mixins can come in.

Different raycasters could want to intersect different objects.

[machenmusik]

it's not just defining, you have to gather the list of THREE objects multiple times which is wasteful.

if you can't share raycasters, and you want to be able to select things with either hand, then you need three. if you implement the ping-pong between controllers based on which one was used last -- which is sometimes good and sometimes incredibly annoying -- then you need two

[machenmusik]

so did you guys want to start prototyping the changes in a different branch (feel free to grab whatever you like from the codepen if it is useful), we can see how it feels to use and whether it makes our apps easier to write, and once happy can PR and merge?

[ngokevin]

That's why I wanted to have some toggler-type component somewhere so there's not duplicate raycasters. But the three.js object querying is not as expensive/frequent as the actual intersection tests.

Yeah, I'm working on it at the moment. Diego needs a cursor for his link traversal demo.

[machenmusik]

laser-controls could attach the cursor only once a controller is connected.

once you start down that path, you realize there is a whole bunch of stuff you really only want once a controller (with the correct hand property) is connected.

Yeah, I'm working on it at the moment. Diego needs a cursor for his link traversal demo.

ok, feel free to steal liberally from codepen if helpful. it felt really good / convenient to be able to just say <a-entity controller-with-cursor-if-present="right"></a-entity> -- naming issues aside -- and I'd encourage efforts to simplify the number of -controls currently required to specify as discussed above.

[ngokevin]

Thanks, I'll check it out. I'm not as opposed to having a general utility toggler components like that (A-Blast has a component that toggles the controllers off if not VR), but for now, I think I'll just code in the one event listener into the component.

[donmccurdy]

I don't think we need a separate laser-cursor component.

That's fair. Even if it's laser-controls, the goal should be to let users add their own <device>-controls component without re-implementing the whole stack.

[donmccurdy]

To summarize, in case this is blocking anyone, I think we are agreed on adding optional laser functionality to the cursor component? I assume that is the bulk of the work, here.

Dummy examples of using the laser-controls convenience wrapper would be helpful to understand that API.

[ngokevin]

Working on it at the moment. There's not so much laser functionality, more like laser appearance, which is just creating a line. Doesn't look like there's actually much code to change, perhaps the hardest part is being able to offset the raycaster, which isn't too bad.