URL Scope to which the manifest applies

[annevk]

Given that it still seems likely we are keeping scoped service workers, manifests should adhere to the same URL scope somehow.

[marcoscaceres]

Related: https://github.com/slightlyoff/ServiceWorker/issues/137#issuecomment-31288138

[marcoscaceres]

Deferring to v2 or later.

[annevk]

How can you defer this?

[marcoscaceres]

It can be defined as an extension to this spec in the SW spec.

[annevk]

How would that work?

[benfrancis]

See also, a thread on Mozilla's dev-b2g mailing list https://groups.google.com/d/msg/mozilla.dev.b2g/r6zAZfEBAI8/qvSmGpAPAJMJ

This is also important for navigational purposes (see #142).

The spec defines properties like "display" which define characteristics of the window in which the bookmarked app is opened, but under what circumstances should those options be applied?

Once a browsing context has been created it can navigate to any URL. Does the "standalone" or "fullscreen" display mode still apply if I navigate outside the app to an unrelated web page from another origin? What defines "outside the app" exactly?

My conclusions so far from that thread are that by default the scope of an app is the origin of the manifest. But there are cases (like multiple apps per origin) where you want to be able to carve the origin up into multiple apps, or into app content and general web content.

In the thread I suggest a manifest property called "scope" for this purpose, which contains a list of strings (URLs with wildcards or regex, but relative to the origin of the manifest URL like the start_url) which define the scope of the app. Two apps with overlapping scope can not be installed.

{ "name": "Google Calendar", ... "scope": ["/calendar_", "/some_other_parts_of_theapp/"] }

Whenever a URL is navigated to it is matched against the list of "bookmarked" origins. If there is a match then it is matched against the scope property of the corresponding manifest(s) if present. If there is still a match then the URL is opened in the app defined by that manifest, with the properties defined in the manifest. If there is no match then the URL is opened inside a normal browser window.

This is similar to the "urls" property in the manifest of Chrome Hosted Apps, except that it is a path relative to the origin rather than an absolute URL which can be from any origin. This is because allowing multiple origins may be open to abuse where an app registers to handle URLs from another origin. To prevent that I believe Chrome Hosted Apps require developers to prove that they own a given domain name, which is a centralised tricky process that requires the manifest be hosted by the app store. By restricting the scope at maximum to the origin of the manifest we can already deduce that the author of the manifest has a certain level of ownership over or access to that origin.

If no scope is defined, should the scope be the full origin of the manifest, or restricted to just the start_url?

I had not considered how this might interact with scope of ServiceWorkers so I'm interested to find out more about how that works.

[marcoscaceres]

Ok, so my plan is to pilfer "URL scope" from Service Workers and use it to set the boundary of the web application.

The boundary is the extent to which members declared in the manifest apply to the top-level browsing context* (display, orientation, navigation, etc.). The scope member will accept a "path expression" which sets the boundaries of the application in some URL space relative to the application's origin. When the scope value is missing, it defaults to "/*" as it does in service workers.

The default behavior will be to open anything outside the scope in the browser. This allows the user to navigate seamlessly withing a URL scope, as it only affects the top-level browsing context.

The URL scope does not affect where resources are loaded from within HTML. That is controlled via CSP.

[benfrancis]

I like this approach, particularly how it mirrors the feature in Service Workers.

I have some questions...

"The default behavior will be to open anything outside the scope in the browser. "

1. What if the URL being hyperlinked to is outside of the scope of this app, but inside the scope of another app? Should the current top level browsing context switch to using the properties of the manifest of the other app, or should it create a new browsing context for the other app (or re-use an existing one)? I would suggest the latter, but perhaps this should be left down to the user agent?
2. What if the user follows a hyperlink to a URL inside the scope of this app from outside the scope of this app, like on a web page from another origin? Should the user stay inside their current browsing context and switch to using the properties of this app's manifest, or should a new browsing context be created (or re-use an existing one) for this app? I would suggest the latter, but perhaps this should be left down to the user agent too?
3. Does this manifest property have any influence on the way that calls to window.open() are handled with URLs inside and outside the scope of the app? Or is that left down to the user agent?
4. Should there be a mechanism by which an app can override the default behaviour and force an off-origin URL to open inside the current browsing context, or in a child of the current browsing context? An example use case for this might be third party authentication. Or should off-origin navigations always exit the current display mode (e.g. exit full screen and display browser chrome so that the new origin is visible in a URL bar)?
5. Should multiple apps be allowed to have overlapping scopes? If so, when navigating to a URL which falls under the scope of both apps, should the user be presented with a choice as to which app to use? Should it always use the current foreground app if the navigation matches its scope?
6. Should apps be able to specify multiple "path expressions" for the same app, for more complex cases?
7. Should the scope be limited to the origin of the manifest? I would suggest yes, but there may be existing web apps which span multiple origins.

[marcoscaceres]

What if the URL being hyperlinked to is outside of the scope of this app, but inside the scope of another app? Should the current top level browsing context switch to using the properties of the manifest of the other app, or should it create a new browsing context for the other app (or re-use an existing one)? I would suggest the latter, but perhaps this should be left down to the user agent?

When I said "open in the browser" I mean leave it up to the UA. Definitely, if you have an app installed that own some URL space, then it should just open in that app with the manifest applied.

What if the user follows a hyperlink to a URL inside the scope of this app from outside the scope of this app, like on a web page from another origin? Should the user stay inside their current browsing context and switch to using the properties of this app's manifest, or should a new browsing context be created (or re-use an existing one) for this app? I would suggest the latter, but perhaps this should be left down to the user agent too?

UA specific - the UA may progressively apply the manifest at that point or ask the user.

Does this manifest property have any influence on the way that calls to window.open() are handled with URLs inside and outside the scope of the app? Or is that left down to the user agent?

No - it does not influence window.open(). But we should fix window.open() in HTML to better address the oauth and overlay browser use cases.

Should there be a mechanism by which an app can override the default behaviour and force an off-origin URL to open inside the current browsing context, or in a child of the current browsing context? An example use case for this might be third party authentication. Or should off-origin navigations always exit the current display mode (e.g. exit full screen and display browser chrome so that the new origin is visible in a URL bar)?

Should window.open() IMO.

Should multiple apps be allowed to have overlapping scopes? If so, when navigating to a URL which falls under the scope of both apps, should the user be presented with a choice as to which app to use? Should it always use the current foreground app if the navigation matches its scope?

We can't stop the overlap. It's the same problem service workers have. We should behave the same - and that's still not fully clear how it will work.

Should apps be able to specify multiple "path expressions" for the same app, for more complex cases?

Probably. I'm on the fence ATM. It sucks that the only choice someone has is to either "/*" or to put everything inside a single directory "/myapp/". OTOH, it does make everything much easier to deal with - but might be too limiting. I need to play with a few apps to understand more how they are laid-out.

Should the scope be limited to the origin of the manifest? I would suggest yes, but there may be existing web apps which span multiple origins.

The scope is bound to the origin of the document doing the installation (which is not necessarily the origin of the manifest). "path expressions" force this by default.

CSP 1.2 will include a manifest-src which dictates where the manifest can be acquired from. But the origin of the app is still bound to the origin of the Document that referenced the manifest. See: http://manifest.sysapps.org/#content-security-policy

[digitarald]

I created bug 1038833 to track this feature for Gecko as this is blocking or impacting many app developers.

[marcoscaceres]

Ok, discussed with @mounirlamouri.

The default manifest scope is that URL of form which the page is bookmarked.

If the page is bookmarked from:

http://example.com/index.html

Then the following are in scope:

http://example.com/index.html/whatever
http://example.com/index.html/more/whatever

Use case: allows bookmarking http://www.w3.org/Mobile/IG/ without needing to add a scope attribute. Allows navigation of /Mobile/IG/<**whatever**>

If the manifest is present and has a scope prop., the scope property is a URL that is same origin to page being bookmarked.

{"scope": "/news"}

So, bookmarking: http://www.bbc.com/news/world-us-canada-29842472 is in scope.

The following is not in scope: http://www.bbc.com/weather/

If the Document is http://www.bbc.com/weather/3413829, but the scope is "/news/" and there is no start_url, the scope is "http://www.bbc.com/weather/3413829". That is, the start_url must be in scope, or else the scope gets dropped on the floor and replaced Document's URL.

[marcoscaceres]

Rough notes above.

[benfrancis]

Interesting.

If the Document is http://www.bbc.com/weather/3413829, but the scope is "/news/" and there is no start_url, the scope is "http://www.bbc.com/weather/3413829". That is, the start_url must be in scope, or else the scope gets dropped on the floor and replaced Document's URL.

So what happens if the document is at http://www.bbc.com/weather/3413829, the start url is http://www.bbc.com/weather and no scope is defined? That would make the start url out of scope.

If for some reason a weather forecast did point to a manifest for a BBC News app with a scope of "/news" as in your example, why would you want to install an app with the name "BBC News" and a BBC news icon, with a scope of http://www.bbc.com/weather/3413829?

Why be inconsistent with the way scope works for Service Workers? Surely a better default scope would be "/" as it is with Service Workers? At least that way the start url will always be in scope by default.

The solution described above seems to be optimising for single page apps being the most common, but your research indicates that isn't the case.

Here are some other examples:

https://www.facebook.com/john.smith/posts/358817860958628 would have a scope of: https://www.facebook.com/john.smith/posts/358817860958628 rather than a scope of: https://www.facebook.com/

https://twitter.com/johnsmith/status/527965998847373312 would have a scope of: https://twitter.com/johnsmith/status/527965998847373312 rather than a scope of: https://twitter.com/

http://www.yelp.com/search?find_desc=cheese&find_loc=San+Francisco%2C+CA&ns=1 would have a scope of: http://www.yelp.com/search?find_desc=cheese&find_loc=San+Francisco%2C+CA&ns=1 rather than a scope of http://www.yelp.com/

If you're going to use a path other than "/" as the default scope then a better choice would seem to be the provided "start_url". However, for the very common case of http://foo.com/index.html as a start url, http://foo.com/bar would be out of scope, which seems wrong.

This leads me to the conclusion that "/" is the most sensible default so that by default the whole origin is in scope, as with Service Workers. If the developer wishes to carve a single origin up into multiple apps (e.g. http://google.com/calendar and http://google.com/maps), then the scope property allows them to do that.

If you want to bookmark http://www.bbc.com/news/world-us-canada-29842472 then just bookmark it. Why would I want a standalone app which can only view one news story? The user agent could optionally choose to open that bookmark inside the BBC News app if the BBC News app is also installed with a scope of http://www.bbc.co.uk/news.

[marcoscaceres]

On October 31, 2014 at 3:14:43 AM, Ben Francis (notifications@github.com) wrote:

So what happens if the document is at http://www.bbc.com/weather/3413829, the start
url is http://www.bbc.com/weather and no scope is defined? That would make the start
url out of scope.

Start URL becomes the scope. 

If for some reason a weather forecast did point to a manifest for a BBC News app with a scope
of "/news" as in your example, why would you want to install an app with the name "BBC News"
and a BBC news icon, with a scope of http://www.bbc.com/weather/3413829?

Rationale for author/developer decisions is not really in scope for the discussion. I.e., we can't really correct for developer error - we can only work within the authors intent. 

Why be inconsistent with the way scope works for Service Workers? Surely a better default
scope would be "/" as it is with Service Workers? At least that way the start url will always
be in scope by default.

Service workers are outside the scope of this discussion. This is to do with navigation scopes, and not related to service workers.

The solution described above seems to be optimising for single page apps being the most
common, but your research indicates that isn't the case.

Actually, that's a matter of perspective. We start with the default case where nothing is defined and work our way from there. The most logical thing to do in that situation is to work with the information the UA has.  

Here are some other examples:

https://www.facebook.com/john.smith/posts/358817860958628 would have a scope of: https://www.facebook.com/john.smith/posts/358817860958628 rather than a scope of: https://www.facebook.com/

Correct. That's Facebook's choice. We don't make decisions for Facebook. 

https://twitter.com/johnsmith/status/527965998847373312 would have a scope of: https://twitter.com/johnsmith/status/527965998847373312 rather than a scope of: https://twitter.com/

Correct. That's Twitter's choice. We don't make decisions for Twitter. 

http://www.yelp.com/search?find_desc=cheese&find_loc=San+Francisco%2C+CA&ns=1
would have a scope of: http://www.yelp.com/search?find_desc=cheese&find_loc=San+Francisco%2C+CA&ns=1
rather than a scope of http://www.yelp.com/

Correct. That's Yelp's choice. We don't make decisions for Yelp. 

In all cases above. the service in question could just add:

{
   scope: "/"
}

The counter example is: http://www.w3.org/Mobile/IG/

Where defaulting to "/" makes no sense - defaulting to "/Mobile/IG/" does. 

If you're going to use a path other than "/" as the default scope then a better choice would
seem to be the provided "start_url".

There is no guarantee of a start_url. But yes, then start_url can override the scope iff it's outside the "scope". 

So: 

{   "scope": "/bar/"   "start_url": "this_one_trashes_your_scope.html" }

 However, for the very common case of http://foo.com/index.html
as a start url, http://foo.com/bar would be out of scope, which seems wrong.

That would again be developer error. It's easy to fix with just adding a scope property. 

This leads me to the conclusion that "/" is the most sensible default so that by default
the whole origin is in scope, as with Service Workers.

Again, all sites seem to disagree with this conclusion. See:  http://www.w3.org/Mobile/IG/ http://bbc.com/news/ (where it clearly doesn't make sense to drop to "bbc.com")

And so on... 

If the developer wishes to carve
a single origin up into multiple apps (e.g. http://google.com/calendar and http://google.com/maps),
then the scope property allows them to do that.

Agree. 

If you want to bookmark http://www.bbc.com/news/world-us-canada-29842472 then
just bookmark it. Why would I want a standalone app which can only view one news story? 

You are confusing the manifest with installation. It's just metadata. What the UA does with it is something different. The manifest can be used in bookmarking just as much as it can be used in installation. 

The user agent could optionally choose to open that bookmark inside the BBC News app if
the BBC News app is also installed with a scope of http://www.bbc.co.uk/news.

Linking/link handling is currently outside the scope of this discussion - this is to do with navigation. We will deal with that in a separate bug. 

[kenchris]

I agree with @marcoscaceres

The basic idea as I got it, is:

if start_url is undefined, start_url is determined from the "bookmarked" url. if scope is undefined, scope is determined from the start_url (which might be determined from the bookmarked url)

[marcoscaceres]

So, I spoke with @annevk and he pointed out a bit of an issue with the path based scope model: essentially, foo.com/app1/ will be able to access foo.com/app2/'s stuff (idb, localStorage), because same origin. This might not be a huge issue, as assuming that the entity that controls the domain knows not to trash over their apps' settings.

We have a complicated issue where some want sites to use one origin per application, but the situation is that many don't or can't - and motivating sites to change their infrastructure/setup to accommodate manifests will be extremely challenging.

Using Suborigin may help, but adds a dependency that again makes this really difficult to deploy: http://www.chromium.org/developers/design-documents/per-page-suborigins

@mounirlamouri can you check what the Chrome security folks think?

[benfrancis]

essentially, foo.com/app1/ will be able to access foo.com/app2/'s stuff (idb, localStorage), because same origin.

Is this something which the Web Manifest specification needs to solve now, or a wider issue around the definition of "origin" on the web?

The current properties of Web Manifest are more related to presentation than to a permissions model. Perhaps the "scope" property of Web Manifest should focus on navigation scope rather than permissions scope, until such a point in the future when Web Manifests are used to define permissions.

Chrome Hosted Apps have a path based permission scope which is something I think Mozilla Open Web Apps lack, but this does not effect the fact that multiple apps on the same origin have access to the same IndexedDB databases. Mozilla Open Web Apps have separate "data jars" per installed app and key some permissions on origin + manifest URL, but have a restriction of one app per origin and don't currently enforce a navigation scope for app windows so a permission granted to an app is effectively granted to its whole origin.

Permissions scope is a thorny issue. Defining a navigation scope to which a certain display mode and default orientation applies seems a lot less contentious.

[marcoscaceres]

Started looking at site structures here in Alexa's top 500: https://etherpad.mozilla.org/url-scope

I'm hopeful we can do the top 500 over time, but it will require a coordinated effort with multiple people. Ben and I will try to do at least the top 100... my eyes are already bleeding from the yahoo homepage: ask me about which celebrity is trending this week! ASK ME! GO ON!!! I DARE YOU :dancer:.

A thing that we haven't added to the discussion is that, because the default display mode is "browser", if the developer doesn't actually specify a display mode, the scope doesn't matter because the application is just opened in the browser.

Hence, a "scope" (and its default) only comes into play with the display mode is explicitly not "browser".

[annevk]

I think if anything you want to align the security model with service workers, since down the line this will have the same power, no?

[marcoscaceres]

So, looking a various sites, I find that one cannot reconcile the same origin policy with navigation (or site structures in general). That is, if we try to enforce a structure or scope by default, we are breaking the Web.

tl;dr: it seems, to me, that Chrome's unbounded (or infinite?) default scope is the most sensible given how sites are made. That is to say, if no scope is defined, then a user can navigate freely from origin to origin within from within a site (same as happens today in a web browser) - this irrespective of view mode with sensible behavior being that the url bar is displayed if the user leaves an origin:

Consider, Web Apps can:

Expand multiple sub origins

And across multiple independent domains!! The following are all the same "app"!!!

Can span multiple independent origins (the above is taobao.com, tb.com, and tmall.com!).

The same applies with sites like the BBC News:

Can work only across one set of paths, but not across others.

For example, Evenote's applications scope would be either /Home.action or "/". However, if it was "/", then "/products" would be "in scope", which wouldn't make sense.

[annevk]

You won't break the web and if you want to associate things like service workers through manifests you need a more sensible policy than that.

[raucao]

And across multiple independent domains!! The following are all the same "app"!!!

I think your definition of the word "app" here is a bit too broad. I would consider this a set of mobile web sites, instead of a single discernible app/program, which you add to your device/homescreen and use as such.

[marcoscaceres]

@annevk, this still needs to work without service workers, tho. Looking at Alexa's top 100, it seems like a non-starter to try to impose any form of structure or the manifest will add no value to a significant percentage of existing content. I'm all for locking it down for service workers, but feel we should provide a flexible path of migration.

@skddc For instance, in toaboa.com (tmall), the navigation, breadcrums/menus, and aesthetics are made to feel like a consistent whole. The fact that a user is bouncing from one domain to another doesn't seem to matter with regards to what constitutes the boundaries of the "app". This is further evidenced by the fact that you can "install" (add to home screen) the app in iOS, and you have a consistent and cohesive application experience without ever returning to the browser (even though the app is split across multiple domains). See below:

Here is the site icon:

Here is the site start page, when opened from the icon above (ju.taobao.com):

Here is the site navigated (detail.m.tmall.com):

Clearly, this is the same "app" running across multiple domains.

[marcoscaceres]

(fixed URL in second image in comment above to ju.taobao.com)

[annevk]

@marcoscaceres it's not really clear to me how you're providing security guarantees without having security boundaries.

[marcoscaceres]

@annevk that's kinda exactly the point, we are not initially providing anything and not making any security guarantees. We are deferring to the user agent to provide the UI to inform the user when a boundary is crossed (as Chrome does today, by showing the URL bar if the user leaves the origin from which they started navigating from). Again, like this:

It is, of course, totally valid to argue that showing the URL bar might be completely inadequate and insufficient. However, it gives a starting point on which to build a security model from, and can serve as a "stick" that breaks the "app" experience (e.g., throwing the application out of fullscreen mode on crossing and origin boundary). "Good" developers will want to avoid that, for instance, by not crossing domain boundaries.

[annevk]

It sounds like you're assuming a particular UI, such as no address bar. Doesn't seem like a great idea. And you can't leave things like this up to user agents if you actually want developers to use this. They need something they can rely on.

[marcoscaceres]

By default, the spec assumes the page is opened in the "browser" (using system convention for URL handling). However, the spec allows web apps to be opened in one of four display modes:

• fullscreen: no chrome.
• standalone: status or address bar.
• minimal-ui: ability to bring in navigation buttons.
• browser: what we know and love.

This is controlled by the display member. This provides some guarantees about how the site is displayed "by default" as the user navigates from one page to another - but also gives the UA control to intervene if user expectations are violated (e.g., if the site crosses an origin boundary).

[benfrancis]

I'm part way through a study [1] of URL scope for web properties from the Alexa Top 500, Chrome Web Store and Firefox Marketplace. So far I've looked at the top 20 from Alexa, 10 from Chrome Web Store and 30 from the Firefox Marketplace.

I haven't reached any final conclusions about how scope should work, but I thought it might be helpful to share my findings so far.

Some conclusions so far:

• Many web sites do not translate well into an app-like experience. Some of the most popular web sites are portals to content spanning multiple separate web sites with different origins. This is particularly the case for popular web sites in Asia which span multiple origins and often open every hyperlink in a new tab. This type of content might be better suited to being used in the browser.
• Web properties which have been submitted to the Chrome Web Store and Firefox Marketplace tend to be a bit more self-contained and more often restricted to one origin.
• Logging into a web app often requires navigating to another origin.
• Web apps will often share an origin with a web site so that /foo and /bar might be part of the app but /about or /products are not.
• Both Chrome OS and Firefox OS currently have an unbounded scope for apps which means that their browsing context can be navigated to any URL. This often results in a bad user experience where the user navigates away from the app but stays in the same browsing context with no browser chrome, leaving them stranded without a back button.

Regarding the previous proposal of using the start_url or Document URL as a default origin:

• Using the Document URL as a default scope is probably a bad idea. This would mean that the scope of the app would vary depending on which URL it was installed from, which makes the scope unpredictable. The scope should be something more static.
• Using the start_url as a default scope is probably also a bad idea. The Start URL of a web app may change over time and restricting the scope of an app to just the start URL will provide a bad user experience in the majority of cases. Most web apps consist of more than one web page.

Regarding the proposal to set an unbounded scope for apps by default:

• Defaulting to an unbounded scope is the "least surprising" thing to do because this is generally how browsing contexts behave. A browser tab can be navigated to any URL.
• However, this doesn't always provide the best user experience. Users can navigate away from the app in standalone mode and get stranded without a back button.

Regarding the proposal to use the root of the web app's origin as the default scope:

• This will be too broad for some web apps, and too narrow for others.
• Web content which is intended to be used like an "app" is more likely to be restricted to one origin.
• Must provide a way to temporarily navigate to another origin in order to log in.

Other observations:

• A single URL prefix is not sufficient to define the scope of a web app in a large number of cases. Web apps share URL spaces with web sites and other web apps in complex ways.
• It may be necessary to have a whitelist of multiple URL prefixes as an array. In some cases an additional blacklist of certain URL prefixes would also be helpful.
• In some cases it might be helpful to also specify a whitelist of URL prefixes which are not part of the app but should open inside the context of the app (e.g. for third party authentication).
• Service Workers do not need to share their scope with the Web Manifest from which they are referenced. The service_worker property can specify a separate scope for the Service Worker.
  1. https://docs.google.com/document/d/1fOsQWOOVuKyqO7cXZoKmxZGQ9FLgLMwmCRw3OEqIKrQ/edit?usp=sharing

[marcoscaceres]

Great summary, @benfrancis. Just some minor things: the "open pages in new tabs" pattern for Asian sites seems to be a desktop only thing (which I guess makes sense). For their mobile/small-device equivalent, when they have one, sites work similarly to what we experience in US/UE sites.

Generally, I think I'm in agreement with your initial conclusions. At the very least, it seems that "unbounded" by default (with UX security behavior like Chrome's) poses the least number of hurdles to adoption of this technology.

Question is: should we prioritize adoption over security? As long as installation is left in control of the browser, then I think it's for us to do so as we continue to work out the heuristic and UX for web app installation. We can then also work out if we need white/black lists to define actual navigation scope.

[kenchris]

I don't think that the point should be to turn any site into an app with as little effort as possible. Quite the opposite.

If people want to create an app like experience we should make it possible for them to do so, ie. give them the right tools.

Default should be what gives the best security and a real app experience. People should invest in creating nice app experiences and not do the bare minimum. When some show what is possible, others will follow. If everyone just does the bare minimum, people will keep saying that the web isnt compelling in comparison with native offerings.

If some links etc ends up opening the browser (and closing it again for logging in) that is not the worst thing in the world, but we should make it possible for the web app author to mitigate these issues, even if that has to be done in the manifest or on their servers.

[benfrancis]

It seems there are a couple of directions this could be taken in. One is to try to accommodate all existing web content with a complex, flexible solution. The other is to go for a simpler solution which is not backwards compatible with some of the web.

A complex solution might be an unbounded scope by default and a manifest which looks like:

{
  "start_url”: “http://foo.com/",
  "scope": {
    "include": ["/foo", "/bar"],
    "exclude": ["/baz", "/qux"]
  },
  "stay_in_app": ["http://norf.com"]
}

The scope has both a whitelist and a blacklist and there's an additional list of URLs which are not part of the app but should stay in the context of the app (e.g. third party authentication). This is very flexible but I think the danger is that the matching rules start to get quite complex for the user agent to carry out on every top level navigation and we risk creating a complex declarative list like AppCache which is unwieldy to maintain.

At the other extreme, the default scope is the root of the domain and the manifest looks like:

{
  "start_url": "/foo",
  "scope": "/foo"
}

Scope is a single URL prefix. This would be simple for new web apps to use, but may provide a bad experience for a lot of existing content where web sites and web apps share URL spaces in complex ways. For example https://evernote.com/products/ would be forced to open inside the Evernote app, and users of GMail might be kicked out of the app into the browser when they want to log in.

If we were starting with a clean slate then my preference would be that the default scope is the root of the origin and the scope of the app is defined by a single URL prefix (or perhaps a whitelist array).

Unfortunately if we turned this feature on overnight in Firefox OS and Chrome OS for existing apps it would create a bad user experience for many existing apps which were not built to this pattern. In fact in the case of Firefox OS which has separate data jars for each app it might actually break apps which navigate to a separate origin in the top level browsing context for authentication purposes. It would also create friction for a lot of big name web properties to adopt Web Manifest because it would either require them to add in workarounds in their apps or to make big infrastructure changes in order to use a manifest.

So, should Web Manifest be backwards compatible with all web content, should it be a clean slate for new web content built with a clean URL scope, or should it take a middle ground?

An example of a middle ground might be:

{
  "start_url": "http://foo.com/bar",
  "scope": ["/bar", "/baz"],
  "stay_in_app": ["http://qux.com"]
}

which incidentally is basically what Jonas proposed in https://groups.google.com/forum/#!topic/mozilla.dev.webapi/yqLffXgokuI

I think we could also make the root of the origin the default scope as in Jonas' proposal above, but to do this for existing apps we would need to give developers advance warning.

I've added examples of this "middle ground" manifest format to the document at https://docs.google.com/a/tola.me.uk/document/d/1fOsQWOOVuKyqO7cXZoKmxZGQ9FLgLMwmCRw3OEqIKrQ/edit#heading=h.854z5nn814fy

[kenchris]

It's kind of more complicated. Why wouldn't a scope be needed for external domains? Maybe

"scope": ["/bar", "/baz", "http://qux.com"]

which would be the same as

"scope": ["http://foo.com/bar", "/baz", "http://qux.com"]

We should probably enforce some CSP rule if we allow other domains , or ignore them otherwise.

I still think it would be good to show some sign when moving to another domain (maybe ignoring subdomains). It would still more or less feel like one app and still it would discourage the use of multiple domains somewhat.

[benfrancis]

Why wouldn't a scope be needed for external domains?

One potential use case for scope is that an installed app could capture navigations to URLs within the scope of the app and load them in the app instead of in a browser tab. This would allow deep linking inside a web app.

You don't want evilapp.com/manifest.json to claim that facebook.com is part of its scope and then capture all of the user's navigations to Facebook. How do you safely prove that another origin is part of the same app? As I understand it the Chrome Web Store uses a centralised process using CNAME records to do this, but that doesn't scale very well to the rest of the web. In my opinion, the maximum scope of a web app should be a single origin.

However, you still might want goodapp.com to use facebook.com/login for authentication. The idea of the "stay_in_app" property is that it can be used to enumerate third party URLs which are not part of the app itself but are used by the app for something like third party authentication. The app will not capture navigations to those URLs, but if the user is already in the context of the app and then navigate to one those URLs they will stay in the app rather than being kicked out to the browser.

Jonas explains this more thoroughly in the thread linked to above.

I still think it would be good to show some sign when moving to another domain (maybe ignoring subdomains).

I agree that if an unbounded scope or a stay_in_app URL causes an app window to be navigated away from its original origin, that the user should be made aware of the new origin they just entered. I don't think subdomains can be assumed to be part of the same app though because many shared hosting services used subdomains to separate sites.

[marcoscaceres]

On November 17, 2014 at 10:09:22 AM, Ben Francis (notifications@github.com) wrote:

Why wouldn't a scope be needed for external domains?

One potential use case for scope is that an installed app could capture navigations to
URLs within the scope of the app and load them in the app instead of in a browser tab. This
would allow deep linking inside a web app.

True. But we've not experimented with this enough to know if scopes are the right thing here. 

You don't want evilapp.com/manifest.json to claim that facebook.com is part of its
scope and then capture all of the user's navigations to Facebook.

Exactly. Hence the same origin restriction. 

How do you safely prove
that another origin is part of the same app? As I understand it the Chrome Web Store uses
a centralised process using CNAME records to do this, but that doesn't scale very well
to the rest of the web. In my opinion, the maximum scope of a web app should be a single origin.

Agree. 

However, you still might want goodapp.com to use facebook.com/login for authentication. 

window.open()? 

The idea of the "stay_in_app" property is that it can be used to enumerate third party
URLs which are not part of the app itself but are used by the app for something like third
party authentication.

Why is window.open() not sufficient? What am I missing? 

The app will not capture navigations to those URLs, but if the
user is already in the context of the app and then navigate to one those URLs they will stay
in the app rather than being kicked out to the browser.

As above. 

Jonas explains this more thoroughly in the thread linked to above.

I still think it would be good to show some sign when moving to another domain (maybe ignoring
subdomains).

I agree that if an unbounded scope or a stay_in_app URL causes an app window to be navigated
away from its original origin, that the user should be made aware of the new origin they
just entered. I don't think subdomains can be assumed to be part of the same app though
because many shared hosting services used subdomains to separate sites.

Agree. For example, GitHub pages.  

[benfrancis]

Why is window.open() not sufficient? What am I missing?

A lot of existing web content navigates the top level browsing context to another origin for authentication. There are web apps which span multiple origins for other reasons too (e.g. both http:// and https:// URLs or different origins for different parts of the app). The stay-in-app property is a proposal of how to be backwards compatible with this behaviour in-app, whilst not allowing malicious apps to capture outside navigations from other origins (I think capturing/deep linking is a key use case we should be designing for).

The Manifest spec could enforce that all top level navigations for an app should come from a single origin, and therefore intentionally not be backwards compatible with some content, but that should be a conscious decision.

I agree that window.open() could offer an alternative to the stay-in-app property if app developers author their apps that way. But I actually think the default behaviour for window.open() or a target=_blank hyperlink to a URL outside of the app's defined scope should be to open outside the context of the app (e.g. in the browser). The stay-in-app property would be one way to override this default, another way would be to pass a special argument to window.open().

[benfrancis]

To throw something else into the mix...

The scope of Service Workers is "restricted by the path of the Service Worker script unless the Service-Worker-Scope: ... header is set." [1]

We could follow the example of Service Workers and make the maximum scope of a manifest be the path of the manifest file (i.e. the same directory). This would prevent an app on a shared hosting site (e.g. http://example.com/~jack/manifest.json) from registering to manage URLs of another app (e.g. http://example.com/~jill/index.html). We could even go as far as to make this the default scope, though that would again not be backwards compatible with some content. I believe this is basically how Amazon web app manifests handle scope [2] and it's a pretty neat solution.

1. https://infrequently.org/2014/12/psa-service-workers-are-coming/
2. https://developer.amazon.com/appsandservices/community/post/Tx1VQ8CDCS8T3O2/Creating-and-Updating-your-Web-App-Manifest

[marcoscaceres]

Filed bug 27653 on HTML, as supporting scope would require interacting with HTML's navigate algorithm.

[marcoscaceres]

@benfrancis, I'm not a fan of binding scope to the location of a resource (because resources can, and should be able to, move and redirect independently of the navigation scope).

I think keeping Chrome's current behavior + and scoping to a single URL will work ok for now. I'm preparing a PR based on that.

[benfrancis]

I still can't help but think we're missing an opportunity by not making the default scope of an app the origin, rather than unbounded... maybe something worth extra investigation as a follow-up.