Handleing capabilities with Manifoldjs

[boyofgreen]

I want to put something out there for feed back from this community. Manifoldjs is trying to figure out how to manage it's capabilities. Specifically, we are allowing users to add in cordova plugins to make those APIs available in the platforms that support. We have two options, via the command line or via the manifest. The manifest would be prefixed of course, much like our whitelist management is:

"mjs_cordova": {
"pluginMode": "server",
"baseUrl": "/js/cordova",
"match": ["http://shiftr.azurewebsites.net/profile", "http://shiftr.azurewebsites.net/users/*"]
}

We may collapse this with the current whitelist value, since we already manage scope (and security for windows):

  "mjs_access_whitelist": [
      {
          "url": "https://bancocontoso.azurewebsites.net", "apiAccess":"all"
      }
  ]

Q1: Does anyone predict capabilities needs within the manifest directly that we can align to?

Next, we are considering enabling developers to be able to add js files to a page via the manifest, that wouldn't be added to the page when it's not an app. This makes it easier to add js that consumes app only APIs. It would look something like this:

"mjs_custom_scripts": [
    {
      "source": "js/barcode-scanner.js"
    },
    {
      "source": "http://mydomain.com/js/photo-capture.js",
      "match": ["/pictures/*", "/user/profile"],
      "platform": ["android", "ios"]
    }
  ]

Q2: Is this opening Pandora's box? It's a big ask from developers, but I don't know if it will lead to bad habits in the future.

[kenchris]

Adding some people who might have comments @anssiko @marcoscaceres @r0b5t4

[kenchris]

Regarding keys like "apiAccess" - according to http://w3ctag.github.io/design-principles/ (JSON keys Lowercase, underscore-delimited) it should be "api_access"

I actually find the name "apiAccess" quite confusing, as I thought it means which APIs you allowed, but it seems to be the other way around:

"none", "all", "allowForWebOnly".

I am a bit confused what these actually do. You already give a url (which is all web?), so how does that work with allowForWebOnly?

Does "all" actually mean that url is basically a wildcard match? Like all urls with this as base can access those windows apis?

[kenchris]

Personally, I am a bit afraid of the custom script stuff, especially if others repackage it, and I think there needs to be a requirements like the manifest MUST live on the same server as the start_url.

Now, as devs don't submit a URL to the stores but use Manifoldjs, it also means that people can modify the tool quite easily - so I don't know how to get around that easily.

I would also leave out "platform" from custom scripts. People can already do checks in the JavaScript file themselves via the userAgent string and that is much more flexible. I don't think that we want to add anything else like that to the platform.

It is unclear when the script will be loaded, maybe the key name could make that more clear. Is it always deferred to after load? "when": "auto" or "ahead" or "defer"? Maybe this is unneeded.

https://developer.mozilla.org/en-US/docs/Web/API/WorkerGlobalScope/importScripts

Maybe we should name it "mjs_import_scripts".

"mjs_import_scripts": [
  { 
    "src": "js/barcode-scanner.js"
  },
  { 
    "src": "http://mydomain.com/js/photo-capture.js",
    "target": ["/pictures/*", "/user/profile"]
  }
]

btw, we use "src" already in the manifest spec.

"mjs_import_cordova_plugins": [
  { 
    "name": "org.apache.cordova.camera"
    "target": ["/pictures/*", "/user/profile"]
  }]

You could make "name" be "src" instead, and just cordova add them automatically if just a name (and not a url) is used. People can use file urls for local plugins then.

[anssiko]

@boyofgreen some questions to make sure I got this right:

• Is mjs_access_whitelist about relaxing the same-origin restriction for the given domain(s)?
• Does mjs_custom_scripts effectively inject and run the given script on the page a la Greasemonkey?

[boyofgreen]

Thanks for the input, @kenchris originally the idea was we were meeting the needs of the window platform and being able to turn on the API access, since that was the only platform that had it. We now are looking at adding api access for other platforms so it seems like you all might agree it needs to be treated differently than the access uRLs. This is what we are thinking right now. separate them into different values:

mjs_extended_scope: This is an array, which we really think is what scope should be in the spec. many web apps contain multiple domains, here is an example: http://www.foxnews.com/manifest.json however we don't want to messy the waters by tying this to api access.

msj_api_access: this will be an array of urls that have API access. maybe we can allow a wildcard here to indicate that all urls in the extended scope should get api access. Open to thoughts.

@anssiko we don't want these to be confused with CSP, and i think we need to address CSP directly to encourage developers to utilize it. This was just about identifying scope. Unfortunately in current version for iOS / Android you needed to list domains (or URLs) that you were pulling any files from. I think we will be able to fix this in next version so it works the same on all platforms. @kenchris @anssiko For msj_custom_scripts it is really similar to a grease monkey solution, difference being it's for the app builder, not the end user. mjs_import_scripits is a better name, we'll use that. I also agree that we need to remove the platforms, that seems to encourage browser detection as apposed to feature detection. I don't want to be an encourager of anything that looks similar to it. src will also be used for refs. What do you think?

[kenchris]

@marcoscaceres could you look at this scope thing?

[marcoscaceres]

Really sorry I haven't gotten to this yet :( it slipped passed me a few days ago and now trying to catch up.

Sent from my iPhone

On 24 Nov 2015, at 8:33 PM, Kenneth Rohde Christiansen notifications@github.com wrote:

@marcoscaceres could you look at this scope thing?

— Reply to this email directly or view it on GitHub.

[marcoscaceres]

(sorry, Jeff, for the huge delay ... I'm currently working on an internal project and have been neglecting my spec duties :()

On November 11, 2015 at 3:19:27 AM, Jeff Burtoft (notifications@github.com) wrote:

I want to put something out there for feed back from this community. Manifoldjs is trying
to figure out how to manage it's capabilities. Specifically, we are allowing users to
add in cordova plugins to make those APIs available in the platforms that support. We
have two options, via the command line or via the manifest. The manifest would be prefixed
of course, much like our whitelist management is:

"mjs_cordova": {
"pluginMode": "server",
"baseUrl": "/js/cordova",
"match": ["http://shiftr.azurewebsites.net/profile", "http://shiftr.azurewebsites.net/users/*"]  
}

I personally don't have an opinion about this. 

We may collapse this with the current whitelist value, since we already manage scope
(and security for windows):

"mjs_access_whitelist": [
{
"url": "https://bancocontoso.azurewebsites.net", "apiAccess":"all"
}
]

Q1: Does anyone predict capabilities needs within the manifest directly that we can
align to?

My prediction and hope (wish?) is that we won't ever require such a mechanism. I've argued why here: https://marcosc.com/2014/12/why-manifests-and-permissions-dont-mix/

(tl;dr: we are working on the Permissions API which should afford developers with a more sophisticated way of doing permission requests within their applications. I worry that the above leads to "up-fron't" permissions requests, which Android has already switched away from, iOS has never embraced, and doesn't match how we do permissioning on the Web) 

Next, we are considering enabling developers to be able to add js files to a page via the
manifest, that wouldn't be added to the page when it's not an app. This makes it easier
to add js that consumes app only APIs. It would look something like this:

"mjs_custom_scripts": [
{
"source": "js/barcode-scanner.js"
},
{
"source": "http://mydomain.com/js/photo-capture.js",
"match": ["/pictures/*", "/user/profile"],
"platform": ["android", "ios"]
}
]

Q2: Is this opening Pandora's box? It's a big ask from developers, but I don't know if
it will lead to bad habits in the future.

I'm personally not a fan of this for a few reasons: 

1. we have a fairly robust way of including scripts on the web, including the ability to include them dynamically. 
2. it adds more indirection: e.g., when should the script actually load? after other scripts, before, after the user installs, etc. 
3. Should the scripts be loaded in order, sync, or async? 
4. there would be limited recovery opportunity, as you could not access the events of the script loading machism (i.e., there is no script tag, so you lose all the value you get from it)  

I'm all for Manifoldjs to experiment with this, however. I think we can learn a lot from how developers would use something like this. For instance, we could in the future add a conditional media attribute to the script element to only load if the display-mode: browser matches one of the conditions in the spec. That would allow special scripts to be loaded as needed.

Hope that makes sense! 

[marcoscaceres]

On November 24, 2015 at 7:33:56 PM, Kenneth Rohde Christiansen (notifications@github.com) wrote:

@marcoscaceres could you look at this scope thing?

As I mentions elsewhere, we really need to drag in Web Apps Sec into the discussion... particularly folks from Google, if this is going to go anywhere. They are really driving the whole "progressive apps" things right now, so their buy-in to allow multiple domains (an even "scope" support) is critical. 

cc @slightlyoff, @mikewest  

[mikewest]

I don't have enough context to understand the discussion in this thread. :) Would someone mind summarizing the proposal? In particular, I don't understand what "access" is being granted to these other origins. What powers are associated with being in this list?

[slightlyoff]

I have a few questions:

1. it seems like this is about an extension point for a project that doesn't really need anything from this group other than a blessing for a key name for the extension point. Am I misunderstanding something?
2. the notion of non-origin security (the scope conversation) is much larger than something we can decide here. It's asking for a wormhole across the web's single safe composition mechanism and I think the bar for standardizing it will need to be much higher; at a minimum, we need to see why postMessage, <iframe>, and foreign fetch aren't sufficient for collaboration.

[marcoscaceres]

@mikewest wrote:

I don't have enough context to understand the discussion in this thread. :) Would someone mind summarizing the proposal? In particular, I don't understand what "access" is being granted to these other origins. What powers are associated with being in this list?

Imagine this scenario - it's the simplest one and most applicable (the access being granted is to a particular "display mode", which controls what browser chrome is shown to the user):

1. A progressive Web App called "News" is actually composed of a set of domains (a.news..com, b.news.com, news-is-awesome.com, etc.)
2. When the web app is launched from the home screen, it is displayed "full screen" (or in the display mode a developer has specified in their web manifest).
3. As the user navigates from a.news.com to b.news.com to news-is-awesome.com, the browser should stay in "full screen".

Today, the Web Manifest spec only allows the "scope" of a web app to behave similarly to a Service Worker scope - as such, it is limited to same origin. This is not acceptable in a lot of cases, because it is common for a single "app" to span multiple domains when navigated.

As such, we are asking: could the web manifest contain a list of origins that define the "scope" of the application: that is, so long as the user stays within this set of URLs, they are "within the scope of the web application" - but once they leave that scope, whatever is defined in the web manifest no longer applies.

For example, going from news.com to google.com would cause the browser to leave full screen mode and return to normal browser mode... but hitting the back button might return the user to fullscreen mode (as they've again entered into the scope of the web app).

Hope that helps!

[marcoscaceres]

@slightlyoff wrote:

I have a few questions: it seems like this is about an extension point for a project that doesn't really need anything from this group other than a blessing for a key name for the extension point. Am I misunderstanding something?

Yes, this is one part. And easy to solve :)

the notion of non-origin security (the scope conversation) is much larger than something we can decide here. It's asking for a wormhole across the web's single safe composition mechanism and I think the bar for standardizing it will need to be much higher; at a minimum, we need to see why postMessage, <iframe>, and foreign fetch aren't sufficient for collaboration.

Agree. Yes, this is about a larger discussion about sites that are composed of multiple origins. See the navigation problem I described. It could be hacked around using a fullscreen iframe, sure... but that has its own problems.

Question is, when/where should we have that discussion? I've been delaying discussing it at Mozilla as we are mostly interested in supporting the basic stuff for now.

[marcoscaceres]

(fixed comment above)

[boyofgreen]

@marcoscaceres I love the blog post. I agree with everything you are saying, however at the same time Manifoldjs is "pollyfilling" the gap between where we are today and where we want to be when progressive apps are a way of life. two action items:

1. We can remove the notion of script injection. I was already leery about this, and I feel like it creates a separate layer of code that pulls us away from a progressive app model. Again, encouraging feature detection, not browser detection.
2. API access.I've gone back and forth between making this something you have to do command line, vs something that you can just include in the manifest behind an mjs_ flag. I'm working on something that might allow this to go away completely and make a more modern experience (much like you blog post encourages). stay tuned.

@slightlyoff @marcoscaceres Scope is a tough topic. I understand now wanting to break the navigation scope, as it has great implications. However Marcos's example is a valid, we see that with fox news, which use the "extended scope" to include their different news domians, but when chrome picks up the manifest, it isn't the same experience they intend, because of the different domains. Interesting enough I feel like Gmail is another example. I would want "mail.google.com" to be the app, but I would want "contacts.google.com" to also be part of the app, however I wouldn't want "www.google.com" to be part of the app. i don't think i can do that with today's spec. Thoughts?

[kenchris]

The way Android would handle something like contact in gmail, would be to use an intent (Contact UI pane/overlay shown on top similar to Chrome Custom Tabs). Another way would be to use a custom UI component (possible soon with custom elements and foreign fetch).

Maybe we need something like the former

[marcoscaceres]

Closing this issue, as it's now a duplicate of #114