[Feature Request] WebXR hands

[yuripourre]

Hello @treblereel do you have any plans to add hand tracking to the WebXR classes?

How the JavaScript specification can be generated? By using WebIDL files?

I tried to add the classes manually but something is not mapped correctly so there are no hands available.

Thank you so much for your efforts! Thanks to you I could successfully make libgdx run into Oculus Quest.

[treblereel]

@yuripourre hmm, i think locally i have updated version of latest WebXR specs ... i have to check it and push.

Very impressive, can you share a demo ? Maybe you can show your work to the community ? Each Thuesday we do a publics calls on twitch so feel free to join (check gitter gwtproject or vertispan/j2cl channels)

libgdx is based on gwt2 ? right ? i totally moved to j2cl and i do new version of three4g with focus on j2cl as well

[yuripourre]

@treblereel

@yuripourre hmm, i think locally i have updated version of latest WebXR specs ... i have to check it and push.

I would really really appreciate!

Very impressive, can you share a demo ? Maybe you can show your work to the community ? Each Thuesday we do a publics calls on twitch so feel free to join (check gitter gwtproject or vertispan/j2cl channels)

Yeah, I can share it there. I tried gitter but is so confusing :)

libgdx is based on gwt2 ? right ? i totally moved to j2cl and i do new version of three4g with focus on j2cl as well

Yes, I had to downgrade a few classes to gwt2 but I am using a also using a mix of elemental2 and gwt classes. Took me a few months but surprisingly it works.

[treblereel]

@yuripourre are you talking about this apis ? https://www.w3.org/TR/webxr-hand-input-1/

[yuripourre]

@treblereel yes, precisely.

Please tell me if there is something I can do to help.

[yuripourre]

I had trouble defining this sequence of XRJointSpaces, I tried JsArray and XRJointSpaces[] (and a lot of combinations) but no matter what I tried they were always null. The hard part is creating a code that tries to populate them and run on a device with hand-tracking capabilities.

Because GWT obfuscates the code is really hard to understand what is wrong, too.

[treblereel]

@yuripourre are you sure it supported ? maybe it's too new ? what is your current setup ?

https://chromestatus.com/feature/5719474055413760

[yuripourre]

@treblereel yes it is new but it is supported by Oculus Quest 2 (the device I am using for the experiments). This example works without issues: https://stewartsmith.io/handy/

[treblereel]

@yuripourre ok, looks like i am good to go!

ps: the demo is using three.js, so ... how do you running your demo it at your Oculus Quest ?

[yuripourre]

@treblereel I wish it was simpler. Right now, I have a libGDX example project with a modified version of your WebXR library that I can compile and serve locally (using my notebook). Then I use ngrok to enable https and redirect to my localhost. Also, I use shorturl.at to shorten the URL so I don't need to type the full ngrok address.

[treblereel]

@yuripourre take a look at https://github.com/treblereel/elemental2-experimental/tree/webxr-hand-input

[yuripourre]

@treblereel it looks good and that was fast! It is very similar to what I did on my experiments, but I believe yours will generate the right types.

How can I generate the java classes from it?

[treblereel]

run build.sh from the root and install webxr and webxr-hand-input from the maven folder.

i have no ar/vr devs so i can't test it ... sorry

[yuripourre]

@treblereel ok! I will try to test until next Sunday. Don't worry I can't thank you enough for making libGDX + webXR a reality. Your code was the ground work I needed to make my libGDX apps run into Oculus Quest!

[yuripourre]

@treblereel I think I will need some assistance with the build, I am close but some jars are missing. The output of sh build.sh is:

DEBUG: [...lots of lines]
WARNING: [...lots of lines]
INFO: Analyzed target //java/org/treblereel/gwt/elemental2/audio:libaudio-src.jar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //java/org/treblereel/gwt/elemental2/audio:libaudio-src.jar up-to-date:
  bazel-bin/java/org/treblereel/gwt/elemental2/audio/libaudio-src.jar
INFO: Elapsed time: 5.081s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
build.sh: line 78: jar: command not found

Seems that the only jar available is the audio. I am trying to understand what is missing, I will update here if I have any progress.

EDIT: I commented all the modules but webxr-hand-input and the jars were generated. I am going to test it now.

[yuripourre]

@treblereel I just hit the first blocker.

I am trying to port this example:

// This method expects a JsArray but XRHand extends JsIterable<String>
// So the line below cannot be called
frame.fillJointRadii(inputSource.hand.values(), radii)

I will keep trying and update here if I had some progress.

[yuripourre]

@treblereel awesome news, it's working!!!

I had to update XRHand and XRFrame (I also moved the code from WebXRHandInputXRFrame to XRFrame). I believe you know a more elegant way to update them.

The code I mentioned now looks like this:

if (!frame.fillJointRadii(inputSource.getHand().values(), radii)) {
  console.log("no fillJointRadii");
  return;
}

JsIteratorIterable<XRSpace> values = Js.uncheckedCast(inputSource.getHand().values());
if (!frame.fillPoses(values, refSpace, transforms)) {
  console.log("no fillPoses");
  return;
}

XRHand.java

@JsType(isNative = true, namespace = JsPackage.GLOBAL)
public interface XRHand extends JsIterable<JsMap.JsIterableTypeParameterArrayUnionType<XRHandJoint, XRJointSpace>[]> {
    XRJointSpace get(String key);

    @JsProperty
    double getSize();

    JsIteratorIterable<XRJointSpace> values();
}

XRFrame

@JsType(isNative = true, namespace = JsPackage.GLOBAL)
public interface XRFrame {
  XRPose getPose(XRSpace space, XRSpace baseSpace);

  @JsProperty
  XRSession getSession();

  XRViewerPose getViewerPose(XRReferenceSpace referenceSpace);

  boolean fillJointRadii(JsIteratorIterable<XRJointSpace> jointSpaces, Float32Array radii);

  @JsOverlay
  default boolean fillJointRadii(XRJointSpace[] jointSpaces, Float32Array radii) {
    return fillJointRadii(Js.<JsIteratorIterable<XRJointSpace>>uncheckedCast(jointSpaces), radii);
  }

  boolean fillPoses(JsIteratorIterable<XRSpace> spaces, XRSpace baseSpace, Float32Array transforms);

  @JsOverlay
  default boolean fillPoses(XRSpace[] spaces, XRSpace baseSpace, Float32Array transforms) {
    return fillPoses(Js.<JsIteratorIterable<XRSpace>>uncheckedCast(spaces), baseSpace, transforms);
  }

  XRJointPose getJointPose(XRJointSpace joint, XRSpace baseSpace);
}

Thank you so much for your help and for your effort. I am planning to post something in gitter soon. I want to opensource my libGDX port but I am still figuring out the best way to do it.

[treblereel]

@yuripourre always welcome ! join me in getter and follow our channels https://gitter.im/gwtproject/gwt and https://gitter.im/vertispan/j2cl