Client side include feature for HTML

[TakayoshiKochi]

Spun off from HTML modules discussion

There are certain amount of interest that including HTML snippet into an HTML document, without using JavaScript. That would be similar to <iframe>, but more lightweight and merged into the same document.

It would work as a naive way to have your header and footer sections defined in one place.

---

(Edit by @zcorpan: also see https://github.com/whatwg/html/issues/3681)

[TakayoshiKochi]

I personally do not buy this much (sorry!), as we have enough primitives (fetch, DOM APIs, maybe Custom Elements) to realize a equivalent feature very easily. Other than ease of use, what is the benefit of having this in the platform?

[mathiasbynens]

How would this differ from HTML Imports?

[TakayoshiKochi]

HTML Imports load a HTML document, via <link rel="import" href=...> and its contents are never rendered without DOM manipulation via script. The document is stored in $(link).import property. HTML Imports have more, like <script> is executed etc.

This idea is about inserting HTML snippet in HTML document. e.g.

main document

<include src="header.html"></include>
Awesome contents
<include src="footer.html"></include>

header.html

<h1>Welcome!</h1>

footer.html

<footer>Copyright 2017 by me</footer>

will result in

<h1>Welcome!</h1>
Awesome contents
<footer>Copyright 2017 by me</footer>

[rianby64]

I was always dreaming to see this feature in the browser. This tag, as exposed by @TakayoshiKochi should allow to put some HTML content in the DOM in a simple way. I think the <include> tag should stay and not be replaced.

I would like to propose the following:

<include id="my-include" src="an_URL.html"></include>

And the event could be:

var included = document.querySelector('#my-include');
included.addEventListener('load', e => {
  // ...
});

included.loaded.then((included_) => {
  // here you see that
  // included_ === included
  // and this promise is ready once the HTML code
  // from included.src has been fetched and appended to the DOM
});

Reproducing the behavior from document.currentScript I found easy to use document.currentInclude, so if a script is executed inside an <include> then it should know where it is.

So, an include has a small set of features

• load event
• loaded promise
• currentInclude (or a better name)

Hope this idea will be useful.

[rianby64]

There are some questions around this tag that I'd like to expose too.

• How to resolve the src path if it's relative?

• What about if src changes. The fetched content should change too?

• If an include contains some tags?

  <include src="an_URL.html">
  <div class="preloader">...</div>
  </include>

  After fetching, the innerHTML content should be replaced?

• In which order should be dispatched the load event? from most nested up to the top?

[domenic]

I don't think we should do this. The user experience is much better if such inclusion is done server-side ahead of time, instead of at runtime. Otherwise, you can emulate it with JavaScript, if you value developer convenience more than user experience.

I'd encourage anyone interested in this to create a custom element that implements this functionality and try to get broad adoption. If it gets broad adoption we can consider building it into the platform, as we have done with other things like jQuery -> querySelectorAll.

[rianby64]

@domenic I tried to develop this idea as a custom element for my projects, and found that it's possible to achieve HTML import, but there are some things that made that solution hard to debug. For instance, beforescriptexecute was removed or even not implemented. Because of that I was forced to turn all my scripts into "inline" scripts.

I'll keep on spreading the word with more cases about how to split the code into small pieces without using extra JS effort.

[Yay295]

What's the actual purpose of this? As domenic mentioned, you can already do this quite easily server-side, so why do we need an HTML element to do it less effectively?

[rianby64]

Personally, I found this feature very useful in my projects. But, this is only my personal opinion. And, what @domenic said sounds fair. The only thing that I'd like to repeat is the absence of beforescriptexecute event, that forces me to turn all the scripts into inline scripts. All other primitives are enough to implement this functionality into a custom element.

I'll be happy to share with you @Yay295 or anybody else my experience with this feature, the pros and cons, but that chat should be outside this issue.

[brandondees]

I think it would be quite useful for any cases where we want DRY html authoring but not the burden of running code server side or requiring JS. It's actually what I naively expected html imports to do at first.

The use cases may be relegated primarily to the realm of small, static-only websites but I think it's a huge advancement for those cases. Simple static-only sites are a large number of websites, or sites that probably should be purely static but cannot be for reasons such as requiring server side rendering to DRY shared fragments such as header/footer, etc. I'm thinking of all the shared web hosting site builder tools and a large number of wordpress sites (a security/maintenance nightmare for typical site owners in my experience) and things along those lines. These kinds of sites are typically owned/maintained by the least tech-savvy operators and are therefore likely under-represented in these kinds of platform-advancement discussions. I'm aware that dynamic rendering or static build tools can get the job done, but those are inaccessible tools to a majority of simple website owners (again, in my personal experience).

The JS-free aspect gets back into the philosophy of progressive enhancement including "site basically works without scripting enabled" and I think that's still important, personally, particularly when we have Brave browser picking up steam with JS disabled by default for security/privacy purposes.

I may try to take a stab at faking this using a custom element backed by fetch, but it wouldn't fill the same gap IMHO and would merely be a demonstration for illustrating the convenience it can provide to the page authoring experience once it's all set up.

[brandondees]

I might also comment that I would expect client side includes to do something efficient with caching based on server headers or whatever, minimizing the UX cost of the extra round trips after first load (and I would presume we could also use link rel=preload etc. to great effect for load time beyond the first page). With http/2 implemented appropriately the UX cost of this feature should go away entirely.

[grepsedawk]

I want to jump in and mention that PHP (Personal Home Page) was literally created to solve this problem "In the most simple way possible". This could be simply done on the browser/markup level so much easier. Imagine if the client could cache the entire HEADER and FOOTER and only need to d/l the main content... Sounds like pretty dang powerful feature to me!

[rianby64]

HTML import feature is what big frameworks offer indirectly. I think, if we've this feature then we've more possibilities to write nice things in a simple way. If HTML imports will be present right into the browser then I'll feel that it is a complete framework.

[AshleyScirra]

Further to @brandondees' point, I think I'd point out that offline-first PWAs using Service Worker very much encourage a client-side approach. For example in our PWA (editor.construct.net), despite it being a large and complex web app, we generate virtually nothing on the server side. This is the obvious way to design something that keeps working offline, because everything is static and local, and there's no need for a server to be reachable, especially if all the server is doing is a trivial substitution of content that could easily be done client side. So I think there are actually some significant use cases where you might want to process an include client-side, and "just do it on the server" doesn't cover everything.

[TakayoshiKochi]

FYI, there was a same discussion happened at https://github.com/w3c/webcomponents/issues/280

[brandondees]

I've implemented my own very quick-and-dirty demonstration here https://github.com/devpunks/snuggsi/pull/109 to begin experimenting with the pros/cons this feature might have, and we're attempting to keep track of other related efforts for reference as well. @snuggs took it beyond the most basic proof of concept and appears to have brought it close to general production-readiness.

I had a discussion recently with a colleague whose initial impression was that this concept merely re-invents server-side includes, which should otherwise be easy enough to work with for most content authors, but I think there are some significant subtle differences still. It's not clear to me why server side includes have not been well leveraged in commonly used website building tools, and I think the reasons boil down to a lack of accessible (read: free) and user-friendly (enough for non tech-savvy users) authoring tools supporting that technology, and lack of standardization. There can be performance benefits from automatically leveraging client side caching of partial documents, which is something I was always baffled by the absence of since I first began learning web dev. New page loads for a given site can retrieve primarily only the portions of the document that are unique, without the need to re-transmit boilerplate sections such as header, navigation, footer, sidebars, etc. without even getting into how the same kinds of benefits also apply when using web component templates.

[TakayoshiKochi]

Oops - sorry about closing accidentally.

I had not been sure about the advantage of client-side processing against server-side include (including PHP's include(), which sounds popular but I don't have any data), but PWA (especially, using service worker to save client-server roundtrips) story in Ashley's https://github.com/whatwg/html/issues/2791#issuecomment-311934876 sounds one of the good reasons of having client-side processing of HTML being okay.

[snuggs]

Indeed @TakayoshiKochi we created a super simple <include- src=foo.html> iteration utilizing the DOMParser. Methinks this is how polyfills are (not doing a good job of) handling HTMLImports.

I'd encourage anyone interested in this to create a custom element that implements this functionality and try to get broad adoption. If it gets broad adoption we can consider building it into the platform, as we have done with other things like jQuery -> querySelectorAll.

I concur with @domenic. on providing a sound iteration/adption/developer ergonomics being worked on in this pull request.

The algoritm was as simple as follows. Also works with nested dependencies due to custom elements lifecycle reactions:

  Element `import-html`

  (class extends HTMLElement {

    onconnect () {
      this.innerHTML = 'Content Loading...'
      this.context.location = this.getAttribute `src`

      let headers = new Headers({'Accept': 'text/html'})

      fetch (this.context.location, {mode: 'no-cors', headers: hdrs})
        .then (response => response.text ())
        .then (content => this.parse (content))
        .catch (err => console.warn(err))
    }

    parse (string) {
      let
        root = (new DOMParser)
          .parseFromString (string, 'text/html')
          .documentElement

      , html = document.importNode (root, true)

      , head = html.querySelector
          `head`.childNodes

      , body = html.querySelector
          `body`.childNodes

      this.innerHTML = ''
      this.append ( ... [ ... head, ... body ] )
    }
})

Any caveats to DOMParser would be great. Especially older versions of IE.

Hope this helps @TakayoshiKochi

/cc @brandondees

[rianby64]

I was thinking last days since @TakayoshiKochi opened this issue. And found really interesting how to integrate this feature include with Worker, <link>, <iframe> and so on, also don't forget to take in count CORS... Looks too hard to achieve the goal of HTML import in a simple way. If <base> could be more flexible, then this feature could be done "as we have enough primitives".

[Yay295]

Ignoring the fact that that code doesn't work, at all, you're really overthinking it. Here's a complete HTML test page. Just change the source to include.

<!DOCTYPE html>
<html>
    <head>
        <script>
            class include extends HTMLElement {
                connectedCallback() {
                    fetch(this.getAttribute('src'), {mode: 'cors', credentials: 'same-origin'})
                        .then(response => response.text())
                        .then(text => this.outerHTML = text)
                        .catch(err => console.warn(err));
                }
            }

            customElements.define('include-html', include);
        </script>
    </head>
    <body>
        <!-- Include the partial HTML. -->
        <!-- If the included HTML has includes, they will be included too. -->
        <include-html id="test" src="to_include.html" />

        <!-- No problems here either. It just logs an error if this happens. -->
        <!-- script>document.getElementById('test').remove()</script -->
    </body>
</html>

This should be a void element in my opinion. There's nowhere to put any nested elements except after everything, so you might as well just put them outside the include instead.

p.s. "Any caveats to DOMParser would be great. Especially older versions of IE." is irrelevant considering custom elements currently only work in WebKit browsers.

[rianby64]

@Yay295 Nice. But how to execute scripts that are present in src="to_include.html"?

The concept of HTML import should be more than just pasting static HTML, right?

[AshleyScirra]

I think the intent here is just to paste DOM content in to another document. HTML imports are a different feature.

[rianby64]

Ok @AshleyScirra . You're right. As consumer, If paste DOM content in to another document then I expect to see scripts, links, workers et al and other inclusions parsed and executed. Hope this feature will gain broad adoption.

[snuggs]

Ignoring the fact your code doesn't work ...

1. @Yay295 the code works fine. Was a snippet from the pr that was clearly referenced in the previous comment. Spared you the details.
2. This code is also intended to be used as a polyfill of sorts for the crappy implementation of webcomponentsjs polyfill that currently breaks for reasons outside of this thread. Therefore to be clear we PERSONALLY need a bonafied Document not a string.
3. tried your method but ran into a few issues on different (ancient) platforms. Have you tried (with scripts) more than just your browser @Yay295 ? Just curious.
4. was the fastest past could think of that runs external scripts and styles. DOMParser is fairly "ancient" based off spec. (thanks for the refactor tho 😎 will add it to our pull request if HTML Imports keels over)

/cc @brandondees @pachonk

[snuggs]

@rianby64 Have to understand that from our tracking/observation of the insane multitude of issues related to imports and includes. I feel there's a lack of understanding of what the respective terms even mean. At least for my simple brain. CSS is a great example of import when include is actually what's happening. I've seen the community fracture over the last x years from the former of the two with module imports. Add the 💩 show from the w3c for more confusion. Seems like they need tons of help but not too versed in backstory and much duplication.

I Feel understanding the definition is the first step. And I know i'm not the only one in confusion piecing the bigger picture together relative to subresources of type text/html. Now having to keep an issue track for the issues that round up the issues for the following 3:

1. HTML Imports - May have been before its time but feel it's a great spec. Possibly PTSD happening.
2. HTML modules - So far everything looks great on paper. More JS heavy.
3. HTML Includes - Place fetched content in DOM (AND execute scripts/dep resolution/etc. if need be)

No. 3 must be able to run scripts no differently than loading images. As that's how the browser is intended to work today. What I learned is instead of figuring out implementation details I learned to appreciate connectedCallback CER. Without that would be difficult to do dependency resolution.

As an aside. Many of the convos happening respectively are people saying similar things but not knowing what to call said thing. #CATAT (Call A Thing A Thing) - @tmornini I hope i'm not the only one that has empathy for the first day developer which we all were once. And one of the reasons PHP was even made is because everyone had "includes" CSS/JS but ironically no HTML.

My takeaways are clear(er) now. It can be done. It doesn't require any change of any specs. It's a simple implementation. All that said have to use JS still (unfortunately but no longer a concern for our authors). Maybe just the "first day developing as a kid" in me who thought <frameset> was an amazing way to separate my concerns before I even knew javascript.

Lastly, CERs are RAD :metal:

Thanks for input. Just trying to figure out where/who/what(org) to contribute to when/how. And most importantly learn along the way from some bright people. The code is the easy part ;-)

🎉 Happy Friday

[rianby64]

@snuggs , Thanks a lot for these 3 definitions.

No. 3 must be able to run scripts no differently than loading images. As that's how the browser is intended to work today.

Can't figure out which part of the HTML standard states that restriction :confused:

What I wanted to point out is the importance of other parts of HTML, like Workers, links with styles and so on. Let's suppose the browser supports HTML include. What can be included? If the main concept is (as stated @AshleyScirra )

to paste DOM content in to another document

Then the first idea I'll try is to put scripts, links and many other things inside that include, and expect that the addresses of all these things are being resolved from the include's base address. Unfortunately, this can't be achieved by using the current primitives fetch, DOM APIs, Custom Elements, etc...

So, workers can be included? If so, then last question: Try to change the baseURL of a worker before load it.

[AshleyScirra]

Then the first idea I'll try is to put scripts, links and many other things inside that include, and expect that the addresses of all these things are being resolved from the include's base address.

I'm not sure this is a good idea, actually. It means you could end up with two identical-looking scripts that actually load from different URLs:

<script src="script.js"></script> <!-- was in document originally -->
<script src="script.js"></script> <!-- was included from subfolder, loads different script -->

The easiest solution is probably just what @Yay295's polyfill did, essentially setting the outerHTML so DOM content is pasted in place in the main document. That uses the same base URL, but will still load scripts, images etc.

[snuggs]

@AshleyScirra thanks for that baseURL nod. Hadn't even thought of that edge! 🙏

[rianby64]

@AshleyScirra , what about the restrictions? Look at this question, please.

But how to execute scripts that are present in src="to_include.html"?

[AshleyScirra]

Doesn't inserting a script tag by assigning outerHTML already download and execute it?

[rianby64]

No, it doesn't. Please, consider this restriction.

When inserted using the innerHTML and outerHTML attributes, they do not execute at all.

So, what I want to understand is if HTML include should execute scripts shipped inside or not... and not only scripts. Links, workers, images and so on...

[AshleyScirra]

Oh, I didn't know that. Well, I agree that a HTML include feature should do that. Perhaps the polyfill could be modified to insert DOM elements instead. If it just fetches a document then does appendChild for each of the root-level elements, that should execute scripts, right?

[rianby64]

No. Neither that way won't work. A good candidate that allows you to execute scripts after appendChild is Range.createContextualFragment. But the problem with createContextualFragment is that all references of scripts inside the documentFragment will be resolved against the first baseURL. And <base> can't be changed once defined.

[snuggs]

Oh, I didn't know that. Well, I agree that a HTML include feature should do that. Perhaps the polyfill could be modified to insert DOM elements instead. If it just fetches a document then does appendChild for each of the root-level elements, that should execute scripts, right?

@AshleyScirra I proposed this earlier today and got lashback for overthinking. Suites our needs just fine though. Again this works with infinitely nested <include->s. https://github.com/devpunks/snuggsi/pull/109#issuecomment-316874026 Trick was to document.importNode on the documentElement from the import.

@rianby64 Range.reateContextualFragment never thought of this. What does this do? would this strategy be considered a "hack"? Have any code examples? Curious.

[AshleyScirra]

I think we're being derailed in to talking about how to implement the polyfill, rather than focusing on what the spec should say about this feature. Perhaps someone should make a series of test cases for what the feature is expected to do. Then anyone can write a polyfill that passes those tests, and the particular manner the polyfill does that is just an implementation detail that doesn't need to be discussed here.

The key here is to write tests, not polyfills, since they define what is expected of the feature. Writing the polyfill first can accidentally enshrine quirks of the polyfill in to the spec, since if it gains wide adoption it becomes very difficult to change it. It also provides browser makers with a way to verify their implementation is compliant if it ever becomes a real standard.

[rianby64]

@AshleyScirra good point. Let's come back to the main thread. As a consumer, I suggest that HTML include should allow to paste DOM content which can hold scripts, workers, links, images and other HTML includes. And the address of every pasted content must be resolved against the HTML include's address. That's what I'd be happy to see in this feature.

[snuggs]

Ok @AshleyScirra pretty simple. HTML include should bring nodes over (anything that inherits from HTMLElement & Text/Comment nodes honestly). And act as if they were within that element in the first place when the parser encounters the elements.

I do believe <link> and <meta> believe it or not are ok to be in <body> these days last i checked.

That's the spec i'd love to have. Resolving the baseURL was something didn't think about but a must for sure.

[AshleyScirra]

I think due to my previous comment, URLs in the included document should either resolve against the URL of the main document, or change any src, href etc. attributes when inserting to the main document, so elements don't need a hidden base URL. Changing attributes upon insertion is more complex though (you'd have to consider every attribute in all of the HTML spec as well as any future additions), and it would be simpler to just resolve against the main document URL since that's more like a copy-paste of HTML content.

Another problem with changing the base URL might be that any included script that makes a fetch, will do so against the base URL of the main document, not the included document. So if the base URL is changed for subresource requests, that becomes inconsistent with script fetches. So my vote is to not change the base URL.

[AshleyScirra]

Here's another example to illustrate the difficulties of changing the base URL:

<img src="image.png">

<script>
fetch("image.png")
</script>

Obviously these both ought to fetch the same resource. However if you include that HTML file with an altered base URL, they fetch different paths, because the script runs in the context of the main document.

[rianby64]

@AshleyScirra , as far as I understand you, looks like the HTML include tag should be something that appends a content once fetched from an URL. This point is important to define, because as pointed @snuggs there are three different concepts, and spec should choose one of them.

[tmornini]

@domenic said:

The user experience is much better if such inclusion is done server-side ahead of time

It doesn't seem relevant to me what's possible to do on the server.

There are no HTML servers, only HTTP servers, so the HTML spec should be completely indifferent with respect to what's possible in the server, as that's an entirely separate protocol.

[snuggs]

@AshleyScirra @rianby64

Using this along with importable document as a remote example. Had to enable CORS to get <include-> to work. Afterwards all resources <img> <link> & <script>.

The following is a smoke server I threw up just as a proof of concept with two documents within two different domains. Was speaking to @brandondees about this and realized at any given moment an author can (and should) use explicit absolute urls when leaving outside domain. If i'm not mistaken.

I know you'd rather not get into implementation details but doesn't render real tests cases as useless IMHO. Each layer imports a script that states where it's being called from.

Utilizes about 3 layers of hierarchy with <include- > dependencies. Not sure what's linking to what as i'm currently lost in inception but a demo's worth a million words IMHO. @AshleyScirra I'm sure you can test your baseURL theory out.

Left <include- src=...> as a wrapper in and didn't replace .outerHTML to understand placement.

Pardon the bright colors. Red background comes from a <link rel=stylesheet> resource nested three <include- >s deep in hierarchy. https://snuggsi.now.sh/examples/include-

What other use cases besides we shall see.?

[Yay295]

To add another complication, includes should probably be blocking. They could then use async and defer for asynchronous loading, similar to how script elements work.

[brandondees]

that's a really key point @Yay295 we'll have to work on that part still I think.

[rianby64]

IMHO, I think, instead of trying to develop a YA import/include tool we should rethink our approaches and start using <link rel="import">. It should be implemented in all browsers...

[snuggs]

@rianby64 i couldn't agree with you more. And we continue to use them. However sluggish as guaranteed to be waiting until at least DOMContentLoaded. There of course is <link rel=pre* mechanisms but still embelishments around not being able to hook in to constructor and connectedCallback if i'm not mistaken. At least that's our current situation and firing order based off current webcomponentsjs polyfill that leaves much to be desired.

We would also not be able to block with polyfill as @Yay295 pointed out would be a nice feature. I'm not saying these have to be our only constraints nor do I like it. But gotta respect the reality they won't ship across the board "ever". And not sure can change that on the FF side.

I fear FF will never implement HTML Imports based on this great summary provided by @annevk. The irony is it was believed in 2014 we would be to concensus by now with (imports, modules, & includes) however much is still the same place. I wonder how Anne feels 3 years later but probably has PTSD from the plethora of people asking like broken records years ago.

Both ES6 modules and service workers open up resource dependency management to web developers. And while ES6 modules is mostly intended for JavaScript, we want to see what kind of dependency systems will be built on these two systems in libraries and frameworks before committing to a standardized design. Especially before committing to one that is not influenced by them. - Anne van Kesteren

On reading again for the first time in a few years there wasn't a No. More a Let's see what you build and if people use before consideration. Just like anything that is considered for the spec. We do have a head nod towards using Service workers and even the polyfill we use doesn't do this for imports let alone includes. What I am thankful for is getting a clearer understanding and separation of needs with the three terms. I feel the people who just wanted include were getting drowned out (and confusing to) people who are focused on the trajectory of ES6 modules / HTML imports. Which is more ancillary than relative to this include discussion.

That said I really really like HTML imports. I also think it came to soon and got some unnecessary flack. I also think there should be a more robust HTML Imports polyfill and something we are working on. I do also realize although imports doesn't have the adoption we'd like doesn't mean it doesn't solve problems very similar to what we are discussing. Like the ability to get a sub document into a master document. I realize there are ways to do this as we are showing here. But still somewhat of a hack (conceptually) and requires author to know Javascript. Feels like kicking the can down the wrong alley with HTML modules but that's outside the scope of this discussion.

[rianby64]

You can get the desired include by enveloping a link rel=import inside a custom element. I haven't tried but I'm sure this idea will work.

[snuggs]

Correct! I covered that use case in my example above @rianby64. However the convention selects default <template> within imported doc not all immediate .childNodes within the document. What I believe we are discussing is all immediate .childNodes are included as well.

[rianby64]

Your best pollyfill won't support workers. link rel=import should.

[snuggs]

To circle back to the relevant topic @rianby64 I am uncertain the underpinnings of the import implementation but surely some details are relevant to this discussion in regards to async defer etc.