Geospatial data, Geolocation and WebXR

[blairmacintyre]

What will the relationship between WebXR and geospatial data be.

It seems that WebXR cannot entirely ignore geospatial positioning, as geospatial content will be a major use case for mobile AR (at least eventually).

The web already has a geolocation API, but it is not sufficient for these purposes: it gives position but not orientation, is of very poor quality and not synchronized with the WebXR frame data. The deviceorientation API cannot be relied on for orientation: it is of very poor quality, was never standardizes (and is potentially going to be removed from existing browsers) and is also not synchronized with the WebXR frame data.

ARKit offers the option to have it's local coordinate system be aligned with geospatial orientation (e.g., Y up, Z south, X east). This provides a possible direction for how geospatial might be handled: have the WebXR API expose a property that says if the coordinate frame can be aligned with geospatial EUS coordinates, and provide a way for the developer to request this. Crude/simple geospatial positional alignment (between the user and the local coordinates) is easier, if you are guaranteed to have the local device coordinates aligned with EUS: each time a geospatial value is received from the local coordinates, an estimate of the geospatial location of the origin of the local coordinate frame can be updated (based on error values, etc). It won't be any better than the error of the geolocation API, but can be stable (because the local coordinates are used for locating and rendering content, not the very-slowly-changing geolocation values).

In theory, if a platform also provided better estimates of geoposition (somehow), it could simply provide those improved values through the geolocation API, benefiting both geospatial AR and other applications that use location. Permission to access location would always be done through the current web permissions approach.

[anssiko]

We're incubating a next generation Geolocation API called Geolocation Sensor (based on the Generic Sensor API) and would love to get your WebXR-inspired geospatial positioning use cases and requirements considered in the design of this new API.

@blairmacintyre, I opened an issue to capture your use cases and requirements: https://github.com/WICG/geolocation-sensor/issues/12 Please chime in :-)

The Geolocation Sensor joins the growing family of Generic Sensor-based APIs that bring consistency to and modernize the API surface of the web platform. We have two Geolocation Sensor polyfill implementations (example) and are planning to start Chromium experimentation in the near future after we've nailed down the feature set for the first experimentation. The spec is currently in incubation in W3C Web Incubator CG and is expected to advance to formal standards track in Device and Sensors WG in April 2018 (new charter).

[blairmacintyre]

@anssiko I'll chime in over there, thanks!

[blairmacintyre]

I would like to "bump" this issue again. As we start to think about requesting capabilities during session creation in WebXR, I would like to propose that we can request that the platform geoaligns the local coordinate system if it can. After the session is created, the app should be able to check if the coordinate frame is geoaligned.

Platforms that don't support geoaligning their coordinate frames can simply ignore the capability request, and signal that the frame isn't aligned.

[janerivi]

Hi I am the founder of the https://open-arcloud.org/ . The TL DR AR-cloud description is a 1:1 digital map/twin of the physical world stored in the cloud that enables a shared programmable space attached directly to our physical suroundings that enables multiuser AR and persistent and universally consistent placement of virtual assets in the real world

One of the things we hope can help bring this to reality is a standard definition of geographical position and orientation that can be understood across platforms and applications. What we call "GeoPose". Each AR-goggle, AR-smartphone or AR-content could have a GeoPose at any given moment.

Obtaining GeoPose of an XR-device could be achieved by matching sensor data from the device with the 1:1 map in the cloud through something like a "GeoPose" cloud service. Once the device has its GeoPose it can display geospatial data and assets that are anchored to a GeoPose.

A bit more about that here: https://open-arcloud.org/standards

please join our slack channel and chime in on how you think a GeoPose should be defined. https://join.slack.com/t/open-arcloud/shared_invite/enQtMzE4MTc0MTY2NjYwLWIyN2E4YmYxOTA4MWNkZmI5OGQ4Mjg2MGYzNTc4OTRkN2RjZGUxOTc4YjJhOTQ0Nzc3OWMxYTA3ZDMxNGEzMGE

My hope of course is that WebXR will support using device GeoPose and do the proper transforms for assets that are ancored to GeoPoses or assets that are described by geospatial coordinates.

[blairmacintyre]

Thanks @janerivi. I agree that we need geospatial support.

I'm curious why you are trying to invent new standards. What's wrong with all the work being done in the geospatial community (XYZECEF, LLA, etc), and using approximations like WGS84 (for example)? It would seem if you create new ways of representing geospatial data, all existing data becomes problematic.

[janerivi]

@blairmacintyre I would really like this to be in sync with what the geospatial community is developing. I have no intention of creating a new standard if one is already in existance that meets the requirements.

However only the 3d-position Latitude, Longitude & Atlitude is not sufficient for AR, Robotics & self driving cars - the spatial orientation (yaw,pitch,roll or quaternion ) relative to a global coordinate system is also needed. That is what is the intention of the GeoPose proposal. If such a thing already exists I'd be happy to learn about it and see if it is functional for AR. If it is then I would be very happy to promote the use of that standard, if not I hope to reqruit help in developing the standard.

[boblaublaw]

Just chiming in with the result of some searching, but unfortunately I see no notion of orientation/pose in the following standards:

• GeoJSon
• OGC Simple Feature access. Part 2 of this standard formalizes the text and binary representations of these primitives, which makes standards-based SQL implementations possible.

Unclear to me if ISO 19107:2003 includes pose.

[blairmacintyre]

@janerivi you are right, LLA is wholly inadequate, which is why geospatial systems don't generally use LLA, except to convert to other formats like ECEF.

If you take a look at cesiumjs.org (for example) you can see a full-fledged geospatial system. geospatial poses are represented internally as ECEF (FIXED in their terminology) matrices: essentially, meters relative to the center of the earth and using the principle axes and equator as the coordinate system. Converting to/from (many) other formats is relatively straightforward. (They also support an INERTIAL frame that is used to represent satellites around the earth)

The simplest way to manage geospatial orientation in an AR system is essentially what Apple has done with ARKit; orient the coordinate frame around some point on the earth, such that XYZ == EUS (east, up, south). This means the local coordinate system is a tangent plane to the earth (or, more likely, the WGS 84 approximation to the globe that is used in LLA).

The essence of my request is to do this: allow a programmer to request that the local coordinates are oriented to align with EUS at the origin of the local system. Once that is done, geospatial position is easy to do using the web APIs. If other systems (such as an AR Cloud like thing) is used by an application, aligning the coordinates out of the cloud with the AR system is straightforward.

I do not think that the web standards can possibly standardize on any given AR cloud system. I also have huge issues with the whole idea of a global AR cloud; it seems to be a horrific privacy nightmare. So, while a given application might want to use that, it probably isn't going to make sense to assume something like that in the web.

[boblaublaw]

I do not think that the web standards can possibly standardize on any given AR cloud system. I also have huge issues with the whole idea of a global AR cloud; it seems to be a horrific privacy nightmare. So, while a given application might want to use that, it probably isn't going to make sense to assume something like that in the web.

I can't speak for @janerivi, but as someone who has taken to using the term "AR Cloud" quite a bit lately, I can offer a suggested interpretation: "AR Cloud" is like regular "cloud" in that it describes a variety of protocols, services, organizations, etc. It's not a singular service, app, company or protocol - just a handy catchall for all new services and technology specific to providing interoperable AR. Viewed in this light, I think we must assume this will take place on the web.

[blairmacintyre]

I’m all-in on cloud-based AR services; but I’ve never heard anyone use “AR Cloud” to refer to that; I’ve always heard it be used to refer to the “massive model of the world used to do localization, persistence, etc”.

[boblaublaw]

You're not crazy, people are certainly focused on relocalization, but I think that has more to do with the immediate availability of some apparent solutions there than a holistic view of what is needed. It isn't like the name "cloud" refers to localization solutions. (In fact, it does refer broadly to global internet services.)

I completely agree that not enough attention is being paid to the privacy and security implications surrounding AR. It is my hope that airing these issues in the same venues will attract interested parties to the problem.

[janerivi]

Regarding privacy, Open AR Cloud is going to make a «Privacy manifesto for the shared programmable, geoaligned space (aka the AR cloud)». The privacy issue is one of my main motivations for founding the Open AR Cloud initiative. A promising observation is that his is a shared concern an one that many bright people have been looking for solutions to for a long time. One example is 6d.ai who made a huge point during a presentation during Augmented World Expo about their discriptors used for aligning multiple users through the cloud containes no data that could possibly be reverse engieneered to detect individuals. I am pretty sure they would be eager add their name in support of a privacy manifesto. It is a selling point to a public that is now much more aware of the detrimental consequences of having everyones privacy infringed.

Also a major concern of people in the AR cloud scene is that we desire a decentralized AR cloud rather than a version controlled by a single entity like Google, Facebook, Amazon, Microsoft, Tesla or a superpower like USA or China. Individuals should be able to controll the «AR cloud / AR Edge» in their home or in their garden. A community should control their public spaces etc. Lots of intelligent people are putting their minds to work looking for ways to help that be a reality.

Regarding the tangential plane. Yes we used that feature in Norkart in our prototype app last year when we did manual geoaligning using the Google Tango platform. To my knowledge all the major AR platforms creates coordinate systems that has two of Its axis tangential to the earth surface plus a vertical axis. I would think it would be a good idea to define the orientation part of the pose as something that is defined relative to one axis in norht/south direction and one axis in east west direction both of wich are horizontal or tangental to the earth surface as well as to the vertical axis. Computatuonally it would probably be best to use a quarernion as almost all 3D graphics systems use them for pose because of the convenient matrix operations.

The only place where that would be sort of messy would be very close to the north and south pole.

Best regards, Jan-Erik Vinje

fre. 15. jun. 2018 kl. 23:36 skrev Joe Boyle notifications@github.com:

You're not crazy, people are certainly focused on relocalization, but I think that has more to do with the immediate availability of some apparent solutions there than a holistic view of what is needed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/immersive-web/proposals/issues/5#issuecomment-397748661, or mute the thread https://github.com/notifications/unsubscribe-auth/AAz1ScDrn_IMz8pDrt6TDwcGu4wTi0A9ks5t9CjfgaJpZM4Rhq5v .

[janerivi]

It was supposed to say «couldn’t possibly be reverse engineered»

lør. 16. jun. 2018 kl. 00:28 skrev Jan-Erik Vinje j.e.vinje@gmail.com:

Regarding privacy, Open AR Cloud is going to make a «Privacy manifesto for the shared programmable, geoaligned space (aka the AR cloud)». The privacy issue is one of my main motivations for founding the Open AR Cloud initiative. A promising observation is that his is a shared concern an one that many bright people have been looking for solutions to for a long time. One example is 6d.ai who made a huge point during a presentation during Augmented World Expo about their discriptors used for aligning multiple users through the cloud containes no data that could possibly be reverse engieneered to detect individuals. I am pretty sure they would be eager add their name in support of a privacy manifesto. It is a selling point to a public that is now much more aware of the detrimental consequences of having everyones privacy infringed.

Also a major concern of people in the AR cloud scene is that we desire a decentralized AR cloud rather than a version controlled by a single entity like Google, Facebook, Amazon, Microsoft, Tesla or a superpower like USA or China. Individuals should be able to controll the «AR cloud / AR Edge» in their home or in their garden. A community should control their public spaces etc. Lots of intelligent people are putting their minds to work looking for ways to help that be a reality.

Regarding the tangential plane. Yes we used that feature in Norkart in our prototype app last year when we did manual geoaligning using the Google Tango platform. To my knowledge all the major AR platforms creates coordinate systems that has two of Its axis tangential to the earth surface plus a vertical axis. I would think it would be a good idea to define the orientation part of the pose as something that is defined relative to one axis in norht/south direction and one axis in east west direction both of wich are horizontal or tangental to the earth surface as well as to the vertical axis. Computatuonally it would probably be best to use a quarernion as almost all 3D graphics systems use them for pose because of the convenient matrix operations.

The only place where that would be sort of messy would be very close to the north and south pole.

Best regards, Jan-Erik Vinje

fre. 15. jun. 2018 kl. 23:36 skrev Joe Boyle notifications@github.com:

You're not crazy, people are certainly focused on relocalization, but I think that has more to do with the immediate availability of some apparent solutions there than a holistic view of what is needed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/immersive-web/proposals/issues/5#issuecomment-397748661, or mute the thread https://github.com/notifications/unsubscribe-auth/AAz1ScDrn_IMz8pDrt6TDwcGu4wTi0A9ks5t9CjfgaJpZM4Rhq5v .

[blairmacintyre]

I'd like to revisit this issue and move it to a repository to discuss. A more specific proposal is this.

In recent discussions of session creation and flow https://github.com/immersive-web/webxr/issues/330 there is a notion of requesting features on a session. Assuming a flow like that will be exposed, I would like to propose we add a geoAlignment feature:

xrSession.requestGeoAlignment()

This feature would request that the underlying session aligns it's coordinate system such that XYZ = EastUpSouth (a standard geospatial frame of reference).

Devices that can do this would succeed, others would fail.

There are two reasons to fail:

• the underlying platform doesn't support this, and the browser is unable to simulate it. ARKit supports this directly for example, so supporting it on such a platform is as simple as setting a flag. Other platforms could use the built in motion APIs to determine an offset between the underlying geospatial frame and the platform frames, and attempt to provide this capability. Or, they could choose not to.
• the hardware is incapable. To support this, a device must have an electronic compass, or must use some sort of global tracking system that is aligned with a geospatial frame of reference. For example, if a virtual positioning system (using pre-set visual mapping) was used by the platform, the map could be aligned with a geospatial without the device needing to do the alignment.

The feature can expose geospatially aligned coordinates without provide geospatial position to the web application. This mean that we can request this without asking the user for location permission (assuming they have already given the browser location permission).

If the web page also requests geospatial location access, it will be trivial to create geospatial AR applications. (Crudely) aligning the origin of the WebXR coordinate frame (or the coordinates of an anchor) with a geospatial pose from the web geolocation API is trivial for a web framework to do, and if the WebXR coordinate frame is aligned with the earth, applications can easily display geospatially aligned content. Math libraries (such as the underlying libraries in cesium.js) exist to make this easy (we used these libraries for this very purpose in argon.js).

[TrevorFSmith]

@blairmacintyre A few questions:

What should happen when the compass value rapidly changes, such as when the device moves near a magnet or large metal structure?

Would there be compass change events to let authors know that they should take measures to prevent nausea?

Would the UA change the orientation of the coordinate system or is it one-and-done at session start?

[blairmacintyre]

What should happen when the compass value rapidly changes, such as when the device moves near a magnet or large metal structure?

The same thing that happens now: the OS will do the best it can to ameliorate it. But, by integrating with the XR platform (e.g., WebXR) more can be done that can be done by reading the compass independently: the platform has compass, accelerometer AND visual data, so it can distinguish between more than can be done separately.

Would there be compass change events to let authors know that they should take measures to prevent nausea?

There won't be nausea. Even with ARKit, the anchors don't move when the compass is disturbed, the underlying system updates the coordinate frame and anchors AT THE SAME TIME. So, the Z axis may rotate around Y (for example), but the visual scene remains stable because the TOTAL system can distinguish between different sensors: the visual features and inertial sensors aren't changing so there is no swim.

That's the whole reason I keep pushing for this. Without leveraging the platform, and trying to use the sensors at the web level, this is much harder.

Would the UA change the orientation of the coordinate system or is it one-and-done at session start?

It would change over time, but is essentially indistinguishable from other changes to the underlying platform. Each frame, the platform hands the the system the head pose and anchors in the current coordinate system. If the coordinate system changes it's relationship to the world, it doesn't matter. It's no different if that is caused by the system re-orienting w.r.t. the compass, etc., or because it changed it's understanding of the planes or structure of the world.

That's the beauty of it all.

[TrevorFSmith]

@blairmacintyre Ok, thank you for the summary.

[johnpallett]

+1 to investigating geoAlignment, we think this is worth creating a repo for further discussion.

[cwilso]

+1 to investigating, but I'd keep this in incubation for now.

[NellWaliczek]

@TrevorFSmith Can we get a repo opened for this work?

[cwilso]

We should indeed open a repo. Also, the README.md for that repo (and any new proposal, maybe?) should clearly define the scope it's trying to cover. I think it's more than just requestGeoAlignment() but it would be good to scope the involvement with the other efforts above?

[blairmacintyre]

I'd like to have a repo, although honestly the first step is just "requestGeoAlignment()" ... there are a bunch of other things related to geo that we can talk there.

In fact, as we develop libraries to help devs create geospatial applications on top of webxr, we could discuss them (or even put them) there ...

[TrevorFSmith]

I agree that we should create a repo, but got caught up on some meta work about process.

I lean toward creating a geo-orientation repo specifically for talking about just that feature and (hopefully) moving it toward acceptance by the working group. Then we can retire that repo and the work can continue in the WebXR spec.

For the other aspects of geo like global location we should start a new, separate Issue for each aspect and work it through the same process: Issue -> repo -> submit to appropriate WG (which may or may not be WebRX) -> the WG's process for acceptance.

So, how about I create a repo named "geo-alignment" and tag Blair to write a scope limiting README?

Please thumbs up if that works for you, or comment if you'd like something different.

[blairmacintyre]

I don't see a reason to limit the discussion. Sure, ignore my suggestion about long term storage of libraries, that was silly.

But as the discussion above shows, separating this feature from the broader question about how to effectively create geo-spatial AR systems doesn't seem plausible. How can we know the feature is adequate or correct if we can't demonstrate it in use?

I don't think we need to make any decisions or recommendations about things like how WebXR might align with the current geoposition API; all of that "work" can be done in a JS library that builds on those two APIs, with the only difference being that apps that only use the geolocation API wouldn't see added precision without knowing about and managing WebXR. But, that's not really important here, it's really a discussion for the folks working on the geolocation spec. Independent of that, we should work through how this would work, and can even test it, to make sure this simple is sufficient.

After all, the feature itself is trivial; the only real question in and of itself is how to expose it, which is really bound up in the current "session creation" discussions where we're talking about how to request and check for specific capabilities.

[cwilso]

I'm not thinking this has to only be the narrowest possible slice, but I would like to see some delineation of scope in the readme, even if it grows over time.

[TrevorFSmith]

My concern is that "geo" is a sprawling topic and in the short term we're mainly trying to figure out whether a small slice, geo-alignment of WebXR coordinate systems, is something that we can nail down in a short period of time. Making a single-purpose repo just to nail down geo-alignment and then submit it to the WG acceptance process (which cwilso is putting together) seems much easier to manage than a general purpose "geo" repo where anything related to geospatial is fair game.

Blair, I understand that you consider geo-alignement easy and essentially solved with a single call, but I don't think that's a widely shared belief, mostly because people haven't seen it in action. If you're interested in geo-alignment making its way into WebXR in a timely manner, I highly recommend carving it off from the rest of geo.

On Wed, Sep 12, 2018 at 2:13 PM Chris Wilson notifications@github.com wrote:

I'm not thinking this has to only be the narrowest possible slice, but I would like to see some delineation of scope in the readme, even if it grows over time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/immersive-web/proposals/issues/5#issuecomment-420799926, or mute the thread https://github.com/notifications/unsubscribe-auth/AACH291Su96_VPmU4yWsd9_mE6jJCNiyks5uaXkGgaJpZM4Rhq5v .

[blairmacintyre]

We are probably saying the same thing, frankly. I'm also not interested in engaging in wide ranging discussions about geo, since I've already "done that" (we have extensive geospatial AR support in argon.js, I've been down this path multiple times).

My view (which I suspect you implicitly agree with) is that we'll end up discussing some of this, as it relates to "should we do this geo-alignment thing."

So, calling it geo-alignment, and then restricting the discussion to "geo things related to the geo-alignment feature" is what we want.

[TrevorFSmith]

Ok, that sounds like a plan.

On Wed, Sep 12, 2018 at 2:34 PM Blair MacIntyre notifications@github.com wrote:

We are probably saying the same thing, frankly. I'm also not interested in engaging in wide ranging discussions about geo, since I've already "done that" (we have extensive geospatial AR support in argon.js, I've been down this path multiple times).

My view (which I suspect you implicitly agree with) is that we'll end up discussing some of this, as it relates to "should we do this geo-alignment thing."

So, calling it geo-alignment, and then restricting the discussion to "geo things related to the geo-alignment feature" is what we want.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/immersive-web/proposals/issues/5#issuecomment-420805661, or mute the thread https://github.com/notifications/unsubscribe-auth/AACH232b5RiAu1FmqtPkHGsCDAkmJotSks5uaX3ZgaJpZM4Rhq5v .

[TrevorFSmith]

I have created a new repo to discuss geo-alignment: https://github.com/immersive-web/geo-alignment/

New conversation about that feature should happen in that repo's Issues: https://github.com/immersive-web/geo-alignment/issues