HTTP-based solution for loading manifests

[marcoscaceres]

Explain how the manifests can be used together with Web Linking.

[annevk]

Please don't. I don't think we want to keep support for that header around.

[marcoscaceres]

Ok. I got the impression @sicking wanted to continue to support that: https://twitter.com/sickingj/status/409938936216055808

[annevk]

We can't even get other browsers to support it for style sheets. Doesn't seem worth reviving.

[marcoscaceres]

Removed it from spec. Thanks @annevk!

[sicking]

I think the situation is different for manifests than for stylesheets. For manifests it is much more commonly going to be the case that there's existing content that people want to add a manifest to. Doing that by editing each and every HTML file or HTML-outputting server side script seems likely to in many cases be much harder than adding a config setting that adds a link header.

This was never really the case with style sheets as we never had large quantities of content where people wanted to apply a stylesheet without otherwise altering the markup content.

[marcoscaceres]

That still doesn't address Anne's concerns about other UA's actually supporting the header. Encouraging people to use the header (by including this in the spec as an example) may lead to some users not getting a good cross browser experience.

Obviously, the spec cannot forbid people from using Link, specially for targeting Gecko - but I'd be reluctant to include this in the spec unless other implementers were also willing to support it.

[sicking]

Well, have you asked any US's how they feel about implementing link-header support for manifests? I'm pretty sure Gecko would have no problem doing so. With all due respect to Anne, I don't like it when we're guessing at what implementors are willing or not willing to implement. Asking is better.

[marcoscaceres]

I agree - and that's fair. I'm checking what support is like as we speak - and also downloading HTTP Header archive data to see usage in the wild. I know that the other potential implementers monitor this, so hopefully they will chime in.

[marcoscaceres]

Ok, browser compat is indeed fairly poor:

• Chrome - not supported
• Safari iOS - not supported
• Firefox - Supported
• IE 11 - Not supported

The test consists of 2 files:

<?php 
header('Link: <http://192.168.0.15:8888/style.css>; rel="stylesheet"');
?>
<h1>Test</h1>

And style.css:

body{
   background-color: green;
}

[marcoscaceres]

In the October 31, 2013 data set, occurrences of Link appear on 2292 individual sites (with a total occurrence of 2362). That's roughly 2.9% of sites in the top ~78,000 sites.

However, it's grossly overrepresented by sites running wordpress (or I presume that is the case as routing happens through a wp.me URL shortening service).

An example of typically what I see:

Link: <http://wp.me/2C6pF>; rel=shortlink

Sites just using wp.me appears 1164 times out of 2362 = 49%. And rel=shortlink appears 64.44% of the time - or 1522 times. rel="canonical" make up the other significant occurrences - 682 occurrences 28.9%. And the rest are rel="alternate" - 99 occurrences, and rel=icon, with 67 occurrences

There are a few other bits and bobs, the they are not statistically significant.

I've posted the data as gists:

• sites
• data

The data was gathered by running the following query:

find ./ -name "*hdr.txt" | parallel grep -l Link: >> linkheader.txt
find ./ -name "*hdr.txt" | parallel grep Link: >> linkheader_raw.txt

I don't know what consumes these HTTP headers - but it doesn't seem to be browsers.

I tried searching the Web to find some information - but could not find much stuff of value. What I found was that an article saying that the Link: header support was added to WordPress in version 3.0.

[annevk]

I don't think we should use Link for manifests if we don't expect we'll keep supporting it for style sheets and other features currently supported through the link element (such as style sheets). In that case I'd much prefer to support a dedicated Manifest header rather than a Link header that looks generic, but actually isn't.

[sicking]

Marcos: I don't think stylesheet support for link: is indicative of if manifests would be supported through link:. It's unlikely that the same codepath would be used. The way to find out if manifest would be supported for link: is to ask implementors exactly if they would support manifests through link:.

I guess I don't have a strong opinion on if we use manifest: or link:. However the simplicity and intuitiveness of using <link rel=manifest> and link: rel=manifest is pretty nice. After all, almost everyone has jumped to the conclusion that external manifests should be done through <link rel=manifest>.

[annevk]

Why are the considerations different from <html manifest="">?

Supporting Link only when convenient seems pretty bad and not at all intuitive.

[marcoscaceres]

@sicking so I agree that asking implementers if they agree to support this is important - but I also think we should wait to see if developers actually want to use this (evidently, they are not using it for stylesheets - at least not in my dataset).

Gecko would be free to add support for this using Link: and see if people use it. And that would make a stronger case for adding a dedicated Manifest: header if need be to the platform.

It needs to be set that if any implementer doesn't end up adding support for this, then the feature becomes quite useless (as a dev, you would need to include the HTML equiv. for legacy user agents and for any non-supporting browser... or you would have to do UA sniffing on the server, which would suck even more).

[marcoscaceres]

Call for comment sent. Reopening while people have a chance to comment.

[marcoscaceres]

FWIW, if we really want to make this easy for devs without needing to modify any HTML or server side stuff, we could consider doing it with a standardized resource name for manifest.

There is already precedence for this:

• iOS automatically searches for icons for "add to homescreen" at the root of the site.
• Windows 8 automatically searches for configuration document (browserconfig.xml) at the root of the site:

From their dev docs:

When you put browserconfig.xml in the root of your site (for example, contoso.com/browserconfig.xml), more than one webpage can use the same tiles and notifications. You'll have the same behavior regardless of which webpage the site is pinned to.

[mnot]

It's up to a particular use of linking to figure out whether or not the Link header is an appropriate way to convey the link; basing the decision for manifests on how well it's supported for stylesheets -- given that it was never specified to be used with the "stylesheet" relation -- seems very strange to me.

If it makes sense to use the link header for manifests, use it. It really depends on what you want to do, though.

[marcoscaceres]

@mnot the stylesheet thing is a red herring; iiuc, what Anne is saying, is that there has been little interest from other browser vendors to implement rfc5988. As such, Mozilla is the only UA currently using it in a limited way (for stylesheets).

From doing the little survey, what I was trying to see was if people were using it for other things (not just stylesheets) to get an indication if developers were aware of it.

If it makes sense to use the link header for manifests, use it. It really depends on what you want to do, though.

Well, in terms of making sense: what we are trying to do is express the same semantics as <link rel="manifest" href="foo">. So, logically, it sounds like a good idea (as Jonas argued).

However, the problem of dealing with legacy and non-supporting user agents still stands - i.e., devs would still have to do some interesting XHR gymnastics to get this header value to pollyfill on the client side - or will need to do UA detection on the server side to send Link: to the right clients and HTML to everyone else that doesn't support Link: (or Manifest:).

Using <link rel="manifest" href="foo"> is also fairly easy to pollyfill, as you just XHR the href value, parse the JSON, and append out the appropriate meta tags to the DOM.

Another potential problem is: "Well! if you support Link: <foo>; manifest, then surely you support Link: style; rel="stylesheet""... and so on - partial implementation of RFC5988 would not be great. So that can also be a potential source of confusion and fragmentation.

[marcoscaceres]

I wrote up a quick prollyfill to see how this would work:

https://gist.github.com/marcoscaceres/7906456

The only annoying bit is parsing the value Link:.

This can be done naively, but it could all go to hell if you have a Link: that contains multiple items:

function parseLinkHeader(linkHeader) {
    var urlStart,
        urlEnd,
        manifestURL;
    //way too simple parser
    if (linkHeader && linkHeader.indexOf('rel="manifest"')) {
        urlStart = linkHeader.indexOf('<') + 1;
        urlEnd = linkHeader.lastIndexOf('>');
        if (urlStart > -1 && urlEnd > -1) {
            return linkHeader.substring(urlStart, urlEnd);
        }
    }
}

[sicking]

That sounds dine with me.

[mnot]

FWIW, here's one in Python that I wrote a while back: https://gist.github.com/mnot/210535

Google finds a few more (haven't checked their correctness deeply): http://bill.burkecentral.com/2009/10/15/parsing-link-headers-with-javascript-and-java/ https://github.com/wombleton/link-headers https://npmjs.org/package/parse-links https://npmjs.org/package/parse-link-header

[marcoscaceres]

Thanks! hehe, I had a feeling my indexOf("<") was a wee bit ... um... ":sweat_smile:"

I'll update my code to use the first one. At least we can then have something to play with.

[marcoscaceres]

Ok, adapted this one: http://bill.burkecentral.com/2009/10/15/parsing-link-headers-with-javascript-and-java/

@mnot, is there a test suite somewhere for Link:?

[marcoscaceres]

The adapted code is here, btw: https://gist.github.com/marcoscaceres/7906456

[mnot]

Unfortunately, no. If you come up with test cases, I'm happy to review, and would be willing to start a test repository.

[marcoscaceres]

So, if we do this, we would need to add what @mnot suggested on the mailing list:

...while the Link header is generic, and indeed link relations (whether it's "stylesheet", "manifest" or something else) are also generic, a particular application of linking (e.g., "should my browser load a manifest?") ought to specify what link serialisations it expects to be used, to improve the chances of interop. The ones chosen have a lot to do with how they're expected to be authored and consumed. Of course, it doesn't make much sense to prohibit other serialisations, because it might be useful for some people, and anyway that's not how the Web works.

I'm still inclined to initially standardize a reserved resource as per #96 (it addresses Jonas' use cases, it seems like a lot less work than having to send a HTTP header as you don't need any server-side tech - and can be used with services like GitHub pages... though I just tested ".well-known" and it doesn't work :frowning: ... filed a bug via email to them). At a minimum, this spec can provide guidance for any UA that wants to support Link.

[sicking]

On Mon, Dec 16, 2013 at 6:39 PM, Marcos Caceres notifications@github.com wrote:

I'm still inclined to initially standardize a reserved resource as per #96 (it addresses Jonas' use cases, it seems like a lot less work than having to send a HTTP header as you don't need any server-side tech - and can be used with services like GitHub pages... though I just tested ".well-known" and it doesn't work ... filed a bug via email to them). At a minimum, this spec can provide guidance for any UA that wants to support Link.

I think using just a .well-known URI solution would not be good. In FirefoxOS we currently have a restriction of only supporting one hosted app per origin. This is something that a lot of developers has complained about.

It's also causing us some problems in the design of various APIs as we can't use the application origin as an indication of which developer the app is coming from and allowing apps from the same origin to have more trust in each other. This since all apps are on separate origins and so from a platform perspective they all come from different developers.

I'm thinking that simply using is the simplest solution. We can punt on inline and http-header based manifests for now if all implementors are ok with that.

/ Jonas

[marcoscaceres]

I think using just a .well-known URI solution would not be good. In FirefoxOS we currently have a restriction of only supporting one hosted app per origin. This is something that a lot of developers has complained about.

Sorry, I was not very clear. I was suggesting the .well-known/manifest.json in addition to <link rel="manifest">. That covers both the single domain case, which is common, but also (via <link>) allows authors the freedom to have multiple applications per origin.

It's also causing us some problems in the design of various APIs as we can't use the application origin as an indication of which developer the app is coming from and allowing apps from the same origin to have more trust in each other. This since all apps are on separate origins and so from a platform perspective they all come from different developers.

I'm thinking that simply using <link rel=manifest> is the simplest solution. We can punt on inline and http-header based manifests for now if all implementors are ok with that.

I'm ok with that. I have the spec text ready but haven't added it to the spec yet - so I can just stash it for later.

@mounirlamouri, @kenchris, @anssiko? => please see https://github.com/w3c/manifest/issues/96 and let us know what you want to do.

[marcoscaceres]

@mnot, in case we come back to this later, can you suggest some text that would meet your requirements as per https://github.com/w3c/manifest/issues/98#issuecomment-30721845. Or point me to a another spec so I can see how to spec what you suggest (it was not completely clear from your email, so a little more guidance would be appreciated)?

I want to at least capture a draft of the required spec text here in case we add it later.

[sicking]

On Tue, Dec 17, 2013 at 3:46 PM, Marcos Caceres notifications@github.com wrote:

I think using just a .well-known URI solution would not be good. In FirefoxOS we currently have a restriction of only supporting one hosted app per origin. This is something that a lot of developers has complained about.

Sorry, I was not very clear. I was suggesting the .well-known/manifest.json in addition to . That covers both the single domain case, which is common, but also (via ) allows authors the freedom to have multiple applications per origin.

Ah, ok. I think that's reasonable but obviously more complex than just having a single mechanism based on . I don't have strong opinions either way.

/ Jonas

[marcoscaceres]

Closing this for now until we get more interest from other implementers.

[BigBlueHat]

I found this while in search of content on using rel="import" (HTML Imports link relation) with the Link: HTTP header.

As a developer, I'm wanting (at least the option) to return a "pure" HTML document and use Link: headers to load any "application" / "navigational" content and styling. Reason being that I'd like to be able to later change those Link: headers when I redesign without changing the body content of those HTML pages.

Imagine that blog post you wrote back in 2000 not being tangled up in it's own antique template, but being only itself and having orthogonal (header) references to how it should be rendered, browsed, and interacted with.

To @mnot's point:

Of course, it doesn't make much sense to prohibit other serialisations, because it might be useful for some people, and anyway that's not how the Web works.

I intend to use them in this way regardless of their UA support via JS shims. rel="manifest" isn't as easy to shim as rel="import" or even rel="stylesheet" (which I'd also like to be using fwiw), but the desire is the same.

Also, here's the Chromium bug that needs fixing to get RFC5988 into Blink (at least): https://code.google.com/p/chromium/issues/detail?id=58456

Thanks for listening. :smile_cat:

[marcoscaceres]

There is a test suite for Link: here: http://greenbytes.de/tech/tc/httplink/

[marcoscaceres]

I'm happy to add support for this in Gecko - @sicking also wants this. @mounirlamouri, any plans to support the Link: http header in the Blink implementation?

[igrigorik]

@marcosc crbug.com/58456 ... what has been missing so far is a compelling use case to prioritize that bug over other dev work.

[marcoscaceres]

So, compelling are:

1. supporting the header would mean that sites would not need to change any markup to add a manifest (handled in HTTP layer).
2. if we support it in Moz (which we don't yet), it would mean we would have another layer of interop.

[stevesouders]

Additional compelling use cases for better web performance:

1. Loading stylesheets early is important, but in some cases the time for the backend server to stitch together the HTML that contains the LINK HTML tag can take a fair amount of time. Also, some CMSs don't flush ANY markup until the entire HTML document is complete. In both of these situations, the server could return a LINK: rel=stylesheet response header immediately which would allow the browser to fetch it early so the page would render faster.
2. Intermediaries (proxies, CDNs) could observe this header and proactively fetch resources to their intermediate cache so that it's ready when the browser later requests it.
3. The Resource Hints proposal ( http://w3c.github.io/resource-hints/ ) includes rel=preload. The best way to send a LINK rel=preload hint is as a HTTP response header - this gets it to the browser sooner than the alternative of including it in markup.

[mnot]

@marcoscaceres case 1 seems compelling to me, in terms of this request.

@stevesouders are you speaking about the manifest in your case 2, or using other link relations?

Again "generic support for the Link header" is a red herring -- RFC5988 does not mandate (or even imply) that just because a link rel=foo can occur in HTML, that relation can also appear in a link header. It's up to foo (or things that use it) to say that.

"Developers get confused by that" isn't a good argument either; they can learn (and have) that the header is different than what's in HTML.

[marcoscaceres]

@mnot, I'm still wanting guidance from your here about what, if anything, I should add to the spec.

Ideally, I would like to add something here: https://w3c.github.io/manifest/#linking

[marcoscaceres]

@mnot, context: https://github.com/w3c/manifest/issues/98#issuecomment-30803909

[mnot]

I think it's just a matter of updating "5.1 Linking to a manifest" as appropriate. If you take a stab at it, I'm happy to review; I can also do a pull request, but the QoS on that might be quite bad...