search

MozillaReality / webvr-spec

*** MOVED TO https://github.com/w3c/webvr ***
https://w3c.github.io/webvr/
Other
158 stars 19 forks source link

Linking between VR web pages #22

Closed brianchirls closed 8 years ago

brianchirls commented 8 years ago

I seem to recall some discussion of enabling linking directly from one VR web site to another without having to stop presenting. The spec as it is doesn't seem to handle that.

This is a potentially tricky case, so I understand wanting to hold off on implementation until more progress is made in the field. But has there been any discussion about what this might look like in an API? Are we confident that the API as it currently stands leaves room for that in the future?

AlbertoElias commented 8 years ago

From what I understood, once there has been a user interaction to start presenting in the HMD, you can move to different VR capable sites without having to take the headset off.

brianchirls commented 8 years ago

So how does the destination web page know that it's already presenting when it starts running?

And what happens if that destination web page doesn't know how to handle VR?

cvan commented 8 years ago

So how does the destination web page know that it's already presenting when it starts running?

And what happens if that destination web page doesn't know how to handle VR?

Last summer, we prototyped a standalone VR backwards-compatible browser based on Firefox's Graphene project (which is Desktop B2G, similar to Electron). You have an app manifest, an index.html, and your scripts which have access to special APIs (which you can use obviously to create your own application – or browser! – in HTML+CSS+JS). Check out the MDN docs, if you're interested. It's quite similar to Chrome's Content API for its webviews.

Check out this short video demoing our Horizon prototype browser.

À la the meta[name=viewport] tag, we added a meta[name=viewmode] tag (see code in Firefox here). This is really only useful for Graphene-based apps which can listen for mozbrowser-specific events such as mozbrowsermetachange.

Unlike the viewport tag, viewmode does not block. This meant that we wouldn't know exactly when a page loaded whether it would be a full VR experience, a 2D classic/mono experience, a 2D experience that also has a button to enter VR mode, a 2D-to-3D VR responsive experience, a 2D+3D+mobile+desktop responsive experience, etc. It gets complicated.

And you want to know which type of experience it is before you render the next page. Do you project a non-VR site as a 2D plane? That's what we did with Horizon. That isn't doable in content with pure WebGL in Chrome, however.

We looked into the Navigation Transitions spec (good critique blog post here) or writing an extension or chrome-level code in Firefox that would allow a kind of hijacking/pausing of the navigation. Navigation Transitions isn't fully spec'd or implemented in Chrome nor Firefox, so that seemed premature to count on that.

Chrome versions 31-38 used to rely on <meta name="mobile-web-app-capable" content="yes"> to identify whether a web site was a "web app" that could be added to the homescreen. (As I'm sure you know, Safari for iOS have their own Apple-specific meta tags, which was the inspiration for this mobile-web-app-capable one.

Anyway, Chrome for Android 39 stopped using the mobile-web-app-capable meta tag. Instead, "web apps" are defined as sites with a manifest and a Service Worker.

That's when I proposed using the Web App Manifest for VR-specific metadata. It seems like a good idea because you can reference a single manifest for all documents on your server (e.g., suppose you have a VR world that contains a different document per level). Huge plus would be not having to propose and invent a new <meta> tag for each new WebVR feature we want to support.

Example: Michael Blix at Samsung had proposed a meta[name="vr-background"] (à la [meta[name="theme-color"]]). This would be a perfect case for using a key in the manifest.

Since the Web App Manifest spec has a non-normative section suggesting to use proprietary keys, the browser could use that information to identify the viewmode of the site and then figure out the projection. Or if a 2D page is encountered, the user agent could decide to open that in a new tab and display a notification message in the headset. It's really up to the user agent to decide how to handle "legacy" content. But parts of this probably ought to be spec'd, at the very least as non-normative text in the spec.

The Manifest spec is explicit that manifests should be fetched asynchronously by the browser. So obviously fetching and reading the manifest metadata would slower than synchronously parsing a meta tag.

If you're interested in some of the pros/cons of the approaches the Mozilla VR was doing WebVR prototyping and prepping for the WebVR API overhaul for v1.0, check out this Google Doc.

And, heh, I just found a short summary I wrote a few months ago that mostly summarises what I said above.

Anyway, we stopped working on our Horizon browser prototype because of a bunch of technical limitations:

We should loop in @caseyyee from the Mozilla VR team who has done a lot of thinking with me on the browser and link traversal UX prototyping. Also, we should loop in @mkeblx from Samsung. The Samsung Internet Browser for Gear VR has to handle all of these things in the very near future, as they're looking to support first-class legacy 2D and WebVR experiences.

kearwood commented 8 years ago

This probably belongs as non-normative text, as the behavior can vary from one browser to another without affecting the WebVR API, but I can share our design related to link traversal and permissions:

Calling Navigator.GetVRDisplays triggers the UX prompts for requesting entering VR...

brianchirls commented 8 years ago

If the site is already in WebVR or is identified as a VR site linked from another VR site, the returned promise resolves without any user interaction.

I assume in this case it resolves with the devices, right?

I'm still not clear on how this solves the problem. Assuming my site has been linked from another VR site and is already presenting, how do I know that so I can start rendering in stereo? When that promise resolves very quickly with a list of devices, how can I tell the difference between being already in VR and the user having selected "accept always"?

kearwood commented 8 years ago

I assume in this case it resolves with the devices, right? Exactly

I'm still not clear on how this solves the problem. Assuming my site has been linked from another VR site and is already presenting, how do I know that so I can start rendering in stereo? When that promise resolves very quickly with a list of devices, how can I tell the difference between being already in VR and the user having selected "accept always"?

Would a flag living along side VRDisplay.IsPresenting and VRDisplay.IsConnected help here? I suppose it would need to be something that returns "true" in this case.

brianchirls commented 8 years ago

Yes, I imagine something like that would work. I'd want to try it out to be sure, of course. There are always unexpected quirks on different devices and scenarios. On Mar 24, 2016 3:33 PM, "Kearwood Gilbert" notifications@github.com wrote:

I assume in this case it resolves with the devices, right? Exactly

I'm still not clear on how this solves the problem. Assuming my site has been linked from another VR site and is already presenting, how do I know that so I can start rendering in stereo? When that promise resolves very quickly with a list of devices, how can I tell the difference between being already in VR and the user having selected "accept always"?

Would a flag living along side VRDisplay.IsPresenting and VRDisplay.IsConnected help here? I suppose it would need to be something that returns "true" in this case.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/MozVR/webvr-spec/issues/22#issuecomment-200984678

mkeblx commented 8 years ago

Calling Navigator.GetVRDisplays triggers the UX prompts for requesting entering VR...

Can you clarify thoughts in regards to prompt display in case of third bullet point above. Any site calling getVRDisplays to query WebVR capabilities would show dialog right at page load potentially, even if no immediate intent to present/enter fullscreen. Is there a good way for site being able to determine if any vrdisplay connected (without getting data access) without calling getVRDisplays and having prompt displayed? Would ideally like to avoid that for just that one piece of info.

cvan commented 8 years ago

@mkeblx à la gamepadconnected and gamepaddisconnected for navigator.getGamepads, vrdisplayconnected and vrdisplaydisconnected events will be emitted without your having to call navigator.getVRDisplays to query for the display. I never actually spec'd this but I have an issue on file to: https://github.com/MozVR/webvr-spec/issues/15#issuecomment-191038325

should definitely spec that soon.

cvan commented 8 years ago

Calling Navigator.GetVRDisplays triggers the UX prompts for requesting entering VR...

I don't understand why we need UX prompts for entering VR. and calling navigator.getVRDisplays doesn't necessarily mean the user is going to enter VR. calling VRDisplay.requestPrompt() does though.

IMO, we should talk to @marcoscaceres, et. al. about adding WebVR to the Permissions API enum so we can use that API.

and until the Permissions API is supported, in the meantime, follow the same security model used by Pop-ups, Pointer Lock, Fullscreen API, etc. - that is, require an "engagement gesture" on the first entering of VR:

see https://html.spec.whatwg.org/#allowed-to-show-a-popup

mkeblx commented 8 years ago

Great, forgot about those, that'd work. Will comment on issue for specing out.

I think permission prompt on getVRDisplays more to do with tracking data a la geolocation API, etc. so does make sense there in cases where will return a VRDisplay. But only when actually at point of requesting that access (or ahead of time with Permissions API). Or should we view pose data as same as same as device orientation data w.r.t. permissions?

Or are there two different permissions: data access, and ability to present/go fullscreen?

cvan commented 8 years ago

I think permission prompt on getVRDisplays more to do with tracking data a la geolocation API

in Chrome, Geolocation permissions (since 43) have been controlled by the Permissions API. see https://developers.google.com/web/updates/2015/04/permissions-api-for-the-web and https://googlechrome.github.io/samples/permissions/

Firefox implementation isn't done yet, but we should plan on using the Permissions API regardless.

But only when actually at point of requesting that access (or ahead of time with Permissions API). Or should we view pose data as same as same as device orientation data w.r.t. permissions?

FWIW, push and midi already have specific keys. I think we should do the same for vr: add specific keys for queryDisplays, allowRequestPrompt, etc. (proper naming TBD, obviously).

thoughts?

cvan commented 8 years ago

full pseudo-code example would go something like this:

var canQueryDisplays = false;
var canRequestPresent = false;  // Can enter VR based on a user-initiated gesture (e.g., `click`).
var canRequestPresentOnLoad = false;  // Can enter VR immediately on document load (see `NOTE` below).
var vrDisplay;

/*

// If you want to test for the permissions explicitly …

navigator.permissions.query({name: 'vr', queryDisplays: true}).then(function (result) {
  canQueryDisplays = this.state === 'granted';
  result.onchange = function () {
    canQueryDisplays = this.state === 'granted';
  };
});

navigator.permissions.query({name: 'vr', requestPresent: true}).then(function (result) {
  canRequestPresent = this.state === 'granted';
  result.onchange = function () {
    canRequestPresent = this.state === 'granted';
  };
});

// NOTE: We may not ever want this to be a toggleable permission.
navigator.permissions.query({name: 'vr', requestPresentOnLoad: true}).then(function (result) {
  canRequestPresentOnLoad = this.state === 'granted';
  result.onchange = function () {
    canRequestPresentOnLoad = this.state === 'granted';
  };
});

*/

function successQueryDisplays (displays) {
  console.log('`navigator.getVRDisplays` access granted');

  if (displays) {
    vrDisplay = displays[0];
    console.log('VR display found and using:', vrDisplay);
  }

  return vrDisplay;
}

function errorQueryDisplays (err) {
  console.warn('`navigator.getVRDisplays` was blocked for this reason: %s', err);
}

function requestPresent (vrDisplay) {
  return vrDisplay.requestPresent({source: webglCanvas});
}

function errorRequestPresent (err) {
  console.warn('`VRDisplay.requestPresent` was blocked for this reason: %s', err);
}

function successEnterVR () {
  enterVRButton.style.display = 'none';
  console.log('`VRDisplay.requestPresent` access granted and entered VR on document load:', vrDisplay);
}

function errorEnterVR (err) {
  enterVRButton.style.display = 'block';
  console.log('Could not enter VR for this reason: %s', err);
}

window.addEventListener('vrdisplaypresentchange', function () {
  if (vrDisplay && vrDisplay.isPresenting) {
    enterVRButton.style.display = 'none';
  } else {
    enterVRButton.style.display = 'block';
  }
});

// Query for a VR display.
navigator.getVRDisplays().then(successQueryDisplays, errorQueryDisplays)
  // Try to enter VR immediately if we can.
  .then(requestPresent, errorRequestPresent)
  .then(successEnterVR, errorEnterVR);

enterVRButton.addEventListener('click', function () {
  if (!vrDisplay) {
    console.warn('Cannot enter VR because no displays found');
    return;
  }

  if (vrDisplay.isPresenting) {
    console.warn('Cannot reenter VR because already presenting');
    return;
  }

  requestPresent(vrDisplay).then(successEnterVR, errorRequestPresent).catch(errorEnterVR);
});
mkeblx commented 8 years ago

Permission API makes sense, plus pseudo code LGTM.

mkeblx commented 8 years ago

@kearwood: If the user declines, the promise is rejected and the site can not access the VRDisplay objects or use their attributes in an exploit to track the user.

-

@cvan: First, do we ever plan on adding a permission to call navigator.getVRDisplays()? I'd prefer no, at least for this first version.

Is queryDisplays permission required? One one hand it definitely is exploitable tracking data, on other hand so is mouse position data. Also lets page determine if and what model of device you own. You can also imagine other exploits when system/chome UI is showing.

Obviously it would simplify things if not required.

mkeblx commented 8 years ago

Also currently spec Security Concerns says:

Content does not need to request user permission to present to the VR HMD

This could be clarified unless intend no permission required (just user gesture required perhaps). I kind of don't like having two permissions. (And anyway if did would probably have a UA setting to ignore for all sites for one/both).

cvan commented 8 years ago

I vote no to introducing a permission to query the displays (i.e., to call navigator.getVRDisplays()). If we need one later, we can add one (and have it use the Permissions API).

This could be clarified unless intend no permission required (just user gesture required perhaps).

That was my initial suggestion. Introducing a permission (in the Permissions API sense) seems a bit much, introduces a dependency on the Permissions API (which only Chrome has shipped to release), and introduces a ton of user experience implications that I don't think we've properly thought through with link traversal. Once we add it, it's going to be next to impossible to remove.

Per my earlier comment, I vote for this simpler approach to gate on a user gesture for the initial user request of entering VR:

follow the same security model used by Pop-ups, Pointer Lock, Fullscreen API, etc. - that is, require an "engagement gesture" on the first entering of VR:

see https://html.spec.whatwg.org/#allowed-to-show-a-popup

toji commented 8 years ago

FWIW I also vote no on introducing a permission here, but ultimately that will be up to the security/privacy people on the Chrome team and not me. :( In the meantime I won't add one voluntarily and I'll push back if one is requested.

cvan commented 8 years ago

Moved to w3c/webvr#30