Real-world understanding

[bialpio]

One of the fundamental aspects of Augmented Reality technologies is the ability for the application to gain access to information about the user’s environment through various real-world understanding APIs in order to create immersive experiences for the users. Hit test is one example of these but other capabilities exist and are in use in native AR applications.

I’d like to propose creating a repository specifically for exploring / incubating APIs that provide access to world geometry, such as planes or meshes. We could then create explainers and incubate possible APIs related to such world understanding that could eventually be adopted by WebXR.

[NellWaliczek]

Moved issue to appropriate Repo for discussion.

[blairmacintyre]

A number of have been interested in this, so I would support creating this repo.

Specifically, a proposal for a starting point is that the API expose a forest of meshes into browser if the underlying platforms supports it. Hololens/ML/etc could expose detailed meshes; ARKit/ARCore could (currently) expose their planes as meshes.

Points we need to discuss:

• how this fits with privacy/permissions. A starting point could be that any geometry that the browser would allow hit-testing against could be passed in. As tech improves, will we need to limit the accuracy and resolution (i.e., at some point, the meshes will be dense and highly detailed)
• is a simple representation sufficient? What about planes without boundaries, for example? Faces such as ARKit's face tracker that report a well-defined mesh (or blend shapes)?

[johnpallett]

I can commit to help move this conversation forward (meaning I'm willing to help coordinate issues, write docs, and facilitate discussion.) @TrevorFSmith do you have enough information to create a repo for this?

@blairmacintyre I agree privacy should be an input to the discussion of this API, but to avoid duplication I'd prefer that the analysis of threat vectors and mitigations remain in the Privacy & Security repo. For example, geometry detail levels are covered here, which would be a good reference for the new API exploration but IMO shouldn't need to be duplicated in another repo. I think this is also what you meant, can you confirm?

[TrevorFSmith]

Hey, folks. In the interest of balancing progress with our quickly expanding list of feature incubation repos I've written up the criteria that need to be met before creating a repo. The goal is to make sure we're making progress without spreading out our attention across dozens of barely used repos.

If you haven't read the README before then now's a good time take a look as it explains all of this. If you have read it lately then skip to this section to see the repo creation criteria.

I suspect that pieces and parts of this topic will end up in repos but it's probably not helpful to create a repo for a general topic like "real-world understanding". The proposals repo Issues (here) are a great place to discuss general topics like that.

Feature incubation repos are meant to be used to get to the bottom of specific open question like "what is the API for hit-testing for WebXR" so let's use this Issue to figure out what specific goals we have in the real-world understanding topic, recruit people to do the work, and then create repos as needed.

[johnpallett]

Thanks @TrevorFSmith - re-phrasing according to the criteria:

The goal: Explore / incubate APIs that provide access to world geometry, such as planes or meshes.

This scope seems sufficiently narrow that a definite end point is possible (particularly, as @blairmacintyre points out, since several OS-level APIs already provide APIs for this capability.)

In the discussion above @blairmacintyre and @bialpio contributed to the goal definition, which represents two organizations.

I agree a repo named "real world understanding" doesn't reflect the goal precisely. "Accessing real world geometry" might be better.

I can be a feature advocate for this.

@TrevorFSmith does this meet the criteria for creating a repo?

[TrevorFSmith]

Thanks @johnpallett for narrowing in on a more specific goal.

It sounds like we need a "real-world-geometry" repo with the goal of "defining an API for the WebXR Device API to provide sensed geometry that is registered to coordinate spaces". Is that right?

Are there topics that should be excluded in the interest of focus and clarifying what success would be? As a somewhat random example, would work in this repo exclude the ability to get camera data and running in-page SLAM?

[johnpallett]

@TrevorFSmith I'm not sure we need to be specific about 'registering geometry in coordinate spaces' in the text for the goal (it could be interpreted as suggesting a particular API shape.)

I'd suggest the original text be used as the goal: "Exploring / incubating APIs that provide access to world geometry, such as planes or meshes; creating explainers and incubating possible APIs related to such world understanding that could eventually be adopted by WebXR." That goal includes success criteria (explainers and an API incubation).

I agree that camera data being used for in-page SLAM is out of scope; IMO the goal already excludes that (the API provides access to world geometry, not camera data).

[TrevorFSmith]

Ok, I've created a repo: https://github.com/immersive-web/real-world-geometry

We're trying to get more specific in feature repos about who is the "feature lead" who will take on the political work of pulling together the right people and finding consensus. Right now, that's @johnpallett.

As usual, if you have comments or suggestions about process then let me know.

I'm going to close this issue since conversation going forward should happen in the new repo Issues.