Declarative Shadow DOM

[mfreed7]

I would like to re-open the topic of declarative Shadow DOM. This has been discussed in the past, here on WHATWG, in W3C here and here, and in WICG. The last substantive public discussion was at the Tokyo Web Components F2F, where it was resolved not to proceed. I would like to revisit that decision.

I think declarative Shadow DOM is an important feature that is missing from the Web, and is something that we should try to implement. The primary motivating use case for declarative Shadow DOM is Server Side Rendering (SSR), which is practically difficult or impossible to use in combination with Shadow DOM. There are also other compelling use cases such enabling scoped styles without requiring Javascript. The rationale behind the prior decision not to proceed with this feature was largely a) implementation complexity and b) lack of developer need.

To address these points, and to explore the topic further, I've written up an explainer, here:

https://github.com/mfreed7/declarative-shadow-dom/blob/master/README.md

I believe this document captures most of the details of the motivation, required features, contentious points, and prior history. But I would love to hear your thoughts and feedback so that this proposal can evolve into something implementable and standardizable. I'm hoping we can use this thread as a discussion forum.

As a quick summary of the proposed syntax, this HTML:

<host-element>
    <template shadowroot="open">
        <style>shadow styles</style>
        <h2>Shadow Content</h2>
        <slot></slot>
    </template>
    <h2>Light content</h2>
</host-element>

would be parsed into this DOM tree:

<host-element>
  #shadow-root (open)
    <style>shadow styles</style>
    <h2>Shadow Content</h2>
    <slot>
        ↳ <h2> reveal
    </slot>
  <h2>Light content</h2>
</host-element>

[emilio]

One interesting question about the proposal is how does it affect all the other weird html parsing things like table fixups and what not. If I have <template shadowroot="open"><td>Foo</td></template>, what is the final dom?

(Other than that kind of stuff, I agree that this is something worth addressing)

Also, it seems a bit weird/unfortunate that you are forced to have a template for every shadow host, when I assume the common thing for a given component is to always have the same shadow root... But I don't have a great solution for that off-hand, maybe you should be able to reference a template from the host by ID? Something else?

cc @hsivonen @smaug---- @edgarchen

[emilio]

Well I guess the parsing insertion may or may not be much of an issue, as you don't have to insert the template contents in the parent, but instead goes directly into the shadowroot...

[annevk]

There's still a risk here in that a previous harmless template can now be used for script injection if you can do some attribute injection. (Also, browsers continue to have security issues around template elements to this day, which isn't reassuring.)

It'd be good to complete the algorithm so it deals with the element already having a shadow root and it details what "moving" means.

[smaug----]

I wonder if converting Firefox UI code from XBL to more web component-y has brought up any ideas related to this issue. @bgrins

[bgrins]

I wonder if converting Firefox UI code from XBL to more web component-y has brought up any ideas related to this issue.

I don't think this would make sense for the Firefox frontend. That said, it seems like we aren't the target audience because of https://github.com/mfreed7/declarative-shadow-dom/blob/master/README.md:

The entire motivation for this feature is no-JS environments

We don't have to support no-js environments or Server Side Rendering at all. We also don't currently use Shadow DOM outside of Custom Elements anywhere (it's possible we may want to do it sometime, but because we always have JS we'd probably just make it a Custom Element in that case). In addition there are some things from my reading of the proposal that would make it inconvenient for our use cases (specifically with shared widgets), and might also be inconvenient for sites that do want to support SSR:

1) Many of our Custom Elements are used many times throughout a single document, so this would require duplication of the template. For instance we have a couple hundred menuitems and toolbarbuttons in the DOM in browser.xhtml at startup. 2) Some Custom Elements are used in a lot of documents across the tree or are used inside of other Custom Elements. In these cases the duplication would be spread across multiple files. This would still require duplication even in the "Instead of inline contents, use an idref to an existing template" alternative. 3) Many Custom Elements get created from JS so AIUI we'd need to programmatically insert the template, or create the same shadow content from JS in another way.

FWIW: what we do now is more-or-less:

• Add a static string getter for markup in an Custom Element class
• Pass that into DOMParser.parseFromString to get a DocumentFragment
• Cache the fragment and do essentially this.shadowRoot.appendChild(document.importNode(fragment, true)) in the constructor or connectedCallback.

I have sort of wished in the past we could have a more declarative way to define the markup in (x)html files alongside scripts and styles, so it's nice to see this being explored though. @mfreed7 I'd be interested to hear more about this point:

Why not wait for, or link this to, declarative custom elements? At first blush, it would seem that these two proposals go together. However, the primary motivating use case for declarative Shadow DOM is SSR and No-JS. Custom element definitions need javascript to function; therefore, this is a different use case/proposal and the two should not be tied together

Specifically if there's a reason that SSR tools couldn't/shouldn't be taught to parse a syntax like that to work even in an environment without JS? So you could declare a custom element with only a template, then have a tool end up creating the same output they would with Declarative Shadow DOM. I'm not familiar with the tooling here, so it's possible I'm missing something.

[calebdwilliams]

I mean, it seems to me what we really need is a declarative way to instantiate at template at which case we could potentially provide some directive to that node that renders the template inside a shadow DOM.

I love this idea, but adding a shadowroot attribute to an HTMLTemplateElement completely changes the semantics of the element. We should have something like <target template="templateID"> that would clone and insert the template’s contents declaratively. Of course that introduces a scoping problem that would need to be figured out since ids are scoped to the shadow root.

[vikerman]

Would there be a way to share the the shadow template and/or style across multiple instances of a host-element?

[FluorescentHallucinogen]

See also https://github.com/ebidel/declarative-web-components by @ebidel.

[UpperCod]

Hi, this should not be standard, since it is rather a hydration technique that can be easily adopted by any library, please analyze the example attached in the documentation, it should generate the polyfill in connectedCallback and not in the constructor.

I think the standards associated with the web should not cover the SSR, this is the work of libraries.

Summary: as a technique it is excellent and I think I'll adopt it atomico , but not this should be a standard

[justinfagnani]

@emilio @calebdwilliams and @vikerman I think standardized templating, along the lines of Template Instantiation is a very valuable feature to explore, but it's a bit different than declarative shadow roots.

I look at this proposal mainly as a way to re-establish the ability to meaningfully serialize a DOM tree (now that include shadow roots). There are a few applications this addresses, including SSR, and some of them overlap with declarative custom elements and templating, but not all of them. I think it's better to keep these proposals separate for now.

[davatron5000]

For me, I think the "slot hoisting" is weird and/or counterintuitive. I also wonder what happens when I need 10x <host-elements> on the page. Do I provide 10x styles and and templates and slots?

Not to bikeshed too much, but I think lots of people would prefer auto-instantiating Custom Elements from a single template like...

<template customelement="host-element" shadowroot="open">
  <style>shadow styles</style>
  <h2>Shadow Content</h2>
  <slot></slot>
</template>

<host-element>
  <h2>Light Content</h2>
</host-element>

Then from a SSR authoring standpoint, I could <?php include('host-element.php') ?> once at the top of my document and author freely.

[justinfagnani]

@davatron5000 what do you mean by "slot hoisting"?

Instantiating declarative custom elements is definitely good future work, but it requires a lot more than what you've sketched to be practically useful.

The key thing to understand here is that this lets us serialize instances of shadow roots. Most shadow roots of the same class of component are not identical - they have been produced by some kind of templating layer or DOM manipulation that makes each instance unique. So you usually can't simply refer to a template and stamp that out, you would need to provide it data and enable the template to specify the transform from data to actual DOM. Again, great future work and where the template instantiation and declarative custom elements ideas/proposals are pointing, but quite a bit different from this.

Even once we have declarative custom elements, it's quite likely that this proposal will be needed as is, since for serialization purposes we'll still need to describe the actual shadow root state of the particular instances in cases where we don't have the data that produced the DOM yet, or in the numerous cases where a shadow root wasn't produced by a declarative custom element.

[tabatkins]

I'm still somewhat on the fence about whether or not we should be able to refer to a pre-existing template so we can avoid repetition.

One of the base use-cases - getting "scoped styles" for a section of your page using the shadow DOM composition boundary without needing JS - is satisfied without that. If you're hand-authoring a no-JS page, you have to repeat all your structure for each element; this doesn't change anything about that, it just adds a little bit more text to each repetition to establish the boundary.

Another satisfied base use-case is shipping server-rendered HTML using shadows that'll be hydrated into full JS-driven custom elements later. You can write custom elements (without having to repeat the contents each time) on your server, then serialize them out into this form; compression should take care of most of the cost of repetition, and post-parsing DOM sizes are comparable.

The use-case not satisfied is wanting to get the less-structural-repetition benefit of a custom element without requiring JS if all you're doing is filling in some DOM and nothing else. That's a reasonable case, I think! But also a less important case than the two I mentioned above. I think if we go without that for now, we're not blocking ourselves from having such a solution later, such as having a <template use=#id></template> that lets it refer to templates already in the DOM? And avoiding that for now lets us skip some more complex scenarios, making the MVP here easier to define.

[Rich-Harris]

I was asked to offer feedback on this proposal in my capacity as a framework author, to help ensure that these additions are relevant to those of us not currently using web components. Let me first say that I'm glad the no-JS use case is being taken seriously — the lack of SSR support (various WC framework hacks notwithstanding) has made web components a non-starter for many of us.

I have a few questions and observations. Most importantly, I agree with @annevk that it's essential to clarify what happens when declarative and programmatic shadow roots collide. Is this.attachShadow(...) an error if there's already a declarative shadow root? Because that would likely cause all sorts of problems.

Is the expectation that custom element authors would do this sort of thing?

class Clock extends HTMLElement {
  constructor() {
    super();

    if (this.shadowRoot) {
      // declarative shadow root exists
      this.hours = this.shadowRoot.querySelector('.hours');
      this.minutes = this.shadowRoot.querySelector('.minutes');
      this.seconds = this.shadowRoot.querySelector('.seconds');
    } else {
      // declarative shadow root doesn't exist
      this.attachShadow({ mode: 'open', serializable: true });
      this.hours = document.createElement('span');
      this.hours.className = 'hours';
      this.minutes = document.createElement('span');
      this.minutes.className = 'minutes';
      this.seconds = document.createElement('span');
      this.seconds.className = 'seconds';

      this.shadowRoot.append(
        this.hours,
        document.createTextNode(' : '),
        this.minutes,
        document.createTextNode(' : '),
        this.seconds
      );
    }
  }

  connectedCallback() {
    this.update();
    this.interval = setInterval(() => {
      this.update();
    }, 1000);
  }

  disconnectedCallback() {
    clearInterval(this.interval);
  }

  update() {
    const d = new Date();
    this.hours.textContent = pad(d.getHours());
    this.minutes.textContent = pad(d.getMinutes());
    this.seconds.textContent = pad(d.getSeconds());
  }
}

Importantly, this doesn't handle the case where the declarative shadow DOM is malformed for whatever reason (a different version of the custom element, for example), so in reality the code would likely be more complex.

Furthermore, in the (probably fairly common) case that the shadow root is populated via innerHTML, we would find ourselves nuking the existing shadow DOM rather than gracefully hydrating it, which seems like it could have negative consequences (performance, but also blowing away state in <input> elements and so on).

In other words, it's hard to see how we can introduce declarative shadow DOM without introducing significant new complexities for custom element authors.

Duplication of content and styles

As @davatron5000 and others have noted, it looks as though this proposal results in duplication of styles and content. But I don't think it's practical to share a <template> between separate instances because the shadow DOM will often differ. Imagine the clock example above also accounted for timezones, and came with styles — the serialized result of using it might look like this:

<p>The time in London is
  <world-clock timezone="GMT">
    <template shadowroot="open">
      <style>
        span {
          font-variant: tabular-nums;
        }

        .seconds {
          font-size: 0.8em;
        }
      </style>

      <span class="hours">12</span> :
      <span class="minutes">34</span> :
      <span class="seconds">56</span>
    </template>
  </world-clock>
</p>

<p>The time in New York is
  <world-clock timezone="EDT">
    <template shadowroot="open">
      <style>
        span {
          font-variant: tabular-nums;
        }

        .seconds {
          font-size: 0.8em;
        }
      </style>

      <span class="hours">07</span> :
      <span class="minutes">34</span> :
      <span class="seconds">56</span>
    </template>
  </world-clock>
</p>

<p>The time in Hong Kong is
  <world-clock timezone="HKT">
    <template shadowroot="open">
      <style>
        span {
          font-variant: tabular-nums;
        }

        .seconds {
          font-size: 0.8em;
        }
      </style>

      <span class="hours">20</span> :
      <span class="minutes">34</span> :
      <span class="seconds">56</span>
    </template>
  </world-clock>
</p>

By contrast, here's what you might get with a non-web-component framework:

<style>
  span.svelte-xyz123 {
    font-variant: tabular-nums;
  }

  .seconds.svelte-xyz123{
    font-size: 0.8em;
  }
</style>

<p>The time in London is
  <span class="svelte-xyz123">18</span> :
  <span class="svelte-xyz123">59</span> :
  <span class="seconds svelte-xyz123">36</span>
</p>

<p>The time in New York is
  <span class="svelte-xyz123">13</span> :
  <span class="svelte-xyz123">59</span> :
  <span class="seconds svelte-xyz123">36</span>
</p>

<p>The time in Hong Kong is
  <span class="svelte-xyz123">02</span> :
  <span class="svelte-xyz123">59</span> :
  <span class="seconds svelte-xyz123">36</span>
</p>

Clearly, the non-custom-element version results in many fewer bytes, and a less complex (i.e. more memory-efficient) DOM.

Serialization

I don't think it makes sense for components to declare their shadow roots to be serializable. For one thing, it's unfortunate if serializable: true, which is presumably the intended default, is something you have to opt in to, though the web compat argument is obviously persuasive.

But more to the point, it's not the component's job to determine that. Whether or not shadow DOM should be serialized is a decision that should be taken at the point of serialization, i.e. by the component consumer. In other words, something like this (after a round of bikeshedding) would make a lot more sense to me:

const html = element.innerHTMLWithShadowDOM;

Intended use case

I expect most people are in agreement about this, but I haven't seen it explicitly addressed, so I'll note it here: we're probably not expecting people to write declarative shadow DOM by hand. That would defeat much of the point of web components, which is to encapsulate the component's behaviour in such a way that HTML authors don't need to worry about it, and would vastly increase the likelihood of errors.

Which is to say that this is a capability directed at frameworks. But this means that those frameworks will, in order to take advantage of this for server-side rendering, need to implement a declarative-shadow-DOM-aware DOM implementation that runs in Node.js (or wherever). Such things add non-trivial complexity, and even performance overhead, to something that is today accomplished using straightforward string concatenation.

---

In summary, while I welcome this discussion, I fear that declarative shadow DOM only gets us part way to what we can already do without web components, but at the cost of additional complexity.

[davatron5000]

@justinfagnani I'm probably not describing it well, but the Light DOM getting consumed by a sibling element (getting "hoisted up" into the slot) was somewhat confusing. I know the sibling <template> is being converted into Shadow DOM and then the Light DOM is being revealed, but it wasn't very intuitive.

If this is a stepping stone towards something great, then I can support that but Rich's summary is pretty spot on for me (except that I want to be able to hand-author stuff).

[mfreed7]

Thanks to everyone for the great comments here. There seem to be a few themes - I'll try to summarize and respond:

1. What should the custom element definition look like?

   The explainer does have a section for this, but I really like the example provided by @Rich-Harris in this comment. That is exactly what I was envisioning - a small if (this.shadowRoot) block that just hooks things up if the shadowRoot already exists, and the actual construction code if not. That code works on both client and server (it is isomorphic), and the added code is minimal. @Rich-Harris asked what happens if the declarative content is malformed - if that is a possibility (due to versioning, etc.), then no matter what the declarative solution, you'll need to do extra work. And in the case where you can assume that an existing #shadowroot means your content is "good to go", you'll get a performance win from not having to blow away the existing content and re-create it.

2. Wouldn't it be better to re-use a single <template> rather than duplicating it for each shadow root?

   Several people already responded to this, but I wanted to point out this section of the explainer that discusses this point at length. The important three points in my mind are: a. As @Rich-Harris and @justinfagnani point out, it is important to remember that we're serializing instances of elements, which likely differ from one another slightly in terms of their DOM content. b. In terms of data/overhead, gzip nicely fixes most of the ills of almost-perfectly-repeated content. Aside from the potentially-shared styles (see the point below), you'll get another copy of the DOM no matter what you do here. So the "overhead" benefits of sharing a single <template> for this seem rather limited. c. Re-using a <template> like this requires a solution to the previously-unsolved "idref" issue. See here for the discussion around ARIA labelled-by. The problem is: how do you deal with nested shadow roots? The idref would then need to cross shadow bounds, potentially in both directions. We don't have a way to allow that, yet.

3. How to handle styles?

   This is definitely an open question. I'm hoping we can come up with a declarative Shadow DOM solution that isn't tied to a particular solution to the styling problem. To do that, I have proposed just using inline <style>s within each shadow root. As mentioned, this would result in a) more bytes on the wire, and b) more DOM memory used. Of those, I'm least concerned with a). For the example HTML provided in this comment, when gzipped, the inline <style> example takes 290 bytes, while the "shared stylesheet" example takes 223 bytes. Yes, that's 25% more, but not the factor of two that it would appear from the raw HTML. I agree that problem b) is a problem that needs a solution. Perhaps the parser could detect exactly-duplicate <style> elements and condense them into a single CSSStyleSheet that gets added to adoptedStylesheets? That might be crazy. I do agree that styles need a solution. I don't think it needs to be solved for this declarative Shadow DOM solution to be useful as-is.

4. Serialization and the "serializable:true" option.

   I love the @Rich-Harris suggestion to add another API (element.innerHTMLWithShadowDOM) that serializes all shadow roots by default. That avoids the need to retrofit existing components with serializable:true, and as you pointed out, this isn't the component's decision to make anyway. I'd be in favor of changing the explainer to match this suggestion.

5. What about existing html parser behaviors, e.g. table fixup: <template shadowroot=open><td>Foo</td></template>

   The advantage of this proposal is that nearly all of the existing "standard" <template> behavior still applies. For example, for this specific example, the "in template" insertion mode rules apply. I don't think there is any additional ambiguity created by this proposal.

6. Is script injection a problem? Can't a previously-harmless <template> be made active by attribute injection?

   No, at least according to the existing proposal. This would be a "parser-only" feature, and adding the shadowroot attribute to an existing <template> would have no effect. As pointed out by @hayatoito, you could still imperatively build a <template>, add a shadowroot attribute, and then do element.innerHTML = element.innerHTML. The innerHTML assignment would see the full <template shadowroot> and would attach a shadow as it is parsed. But that doesn't seem like a security risk, since you're blowing away the entire innerHTML in that case anyway. Please correct me if I'm wrong.

7. It is weird to have a <template shadowroot> turn into a #shadowroot and then have previously-sibling elements get slotted into the #shadowroot.

   Yes, this is definitely different and will take getting used to, no question. But this statement seems like it would apply to any declarative Shadow DOM solution. No matter the semantics, some element will become, or create, a #shadowroot which will then start "pulling in" sibling content into <slot>s.

8. What happens if there is already a shadow root?

   For compatibility and alignment, this needs to be an error. I mentioned several such scenarios in the explainer, here. Basically, you can (still) only attach a shadow root once, and any subsequent attempts (either declarative or imperative) will result in an error.

9. Adding shadowroot to <template> changes the semantics of the element, which is weird.

   Yes, it is, I agree. This is discussed here in the explainer. The one point that seems to kill the idea of creating a new element (e.g. <shadowroot>) is the backwards-compat problem. Until all browsers understand the new element, enclosed <style> and <script> elements will be exposed to the parent page, with potentially bad consequences.

Thanks again for the great points raised here!

[Jamesernator]

That is exactly what I was envisioning - a small if (this.shadowRoot) block that just hooks things up if the shadowRoot already exists, and the actual construction code if not. That code works on both client and server (it is isomorphic), and the added code is minimal.

Personally I think it would be preferable that closed shadow roots can still be SSR-ed, I understand that .attachShadow({ mode: 'closed' }) is kinda weird when a shadow root is already attached so perhaps a way to close a shadow root after the fact would make more sense:

class MyComponent extends HTMLElement {
  #shadowRoot;
  constructor() {
    if (this.shadowRoot) {
      this.#shadowRoot = this.shadowRoot;
      this.#shadowRoot.close(); // Changes the shadow root from open to closed
    } else {
      this.#shadowRoot = this.attachShadow({ mode: 'closed' });
      // Initialize shadow root ...
    }
    // ....
  }
}

Clearly, the non-custom-element version results in many fewer bytes, and a less complex (i.e. more memory-efficient) DOM.

One suggestion I had on the original discourse thread was to use template instantiation so that data can be injected into a single template with even less duplication than current SSR approaches as they don't even need to duplicate the rendered DOM.

This suggestion would address @Rich-Harris concerns about duplication but depends on a very early proposal for template instantiation. Although as a plus the approach could still be used even with duplication because if template instantiation were added later it could be added on without changing the elements significantly e.g.:

<template id="world-clock-template" shadowroot="open">
    <style>
      span {
        font-variant: tabular-nums;
      }

      .seconds {
        font-size: 0.8em;
      }
    </style>

    <span class="hours">{{hours}}</span> :
    <span class="minutes">{{minutes}}</span> :
    <span class="seconds">{{seconds}}</span>
</template>

<template id="prerendered-2" shadowroot="open">
    <style>
      span {
        font-variant: tabular-nums;
      }

      .seconds {
        font-size: 0.8em;
      }
    </style>

    <span class="hours">07</span> :
    <span class="minutes">34</span> :
    <span class="seconds">56</span>
</template>

<template id="prerendered-3" shadowroot="open">
    <style>
      span {
        font-variant: tabular-nums;
      }

      .seconds {
        font-size: 0.8em;
      }
    </style>

    <span class="hours">20</span> :
    <span class="minutes">34</span> :
    <span class="seconds">56</span>
</template>

<p>The time in London is
  <world-clock shadowroot="#prerendered-1" timezone="GMT"></world-clock>
</p>

<p>The time in New York is
  <world-clock shadowroot="#prerendered-2" timezone="EDT"></world-clock>
</p>

<p>The time in Hong Kong is
  <world-clock shadowroot="#prerendered-3" timezone="HKT"></world-clock>
</p>

However with template instantiation this could just become:

<!-- With template instantiation -->

<template id="world-clock-template" shadowroot="open">
    <style>
      span {
        font-variant: tabular-nums;
      }

      .seconds {
        font-size: 0.8em;
      }
    </style>

    <span class="hours">{{hours}}</span> :
    <span class="minutes">{{minutes}}</span> :
    <span class="seconds">{{seconds}}</span>
</template>

<p>The time in London is
  <world-clock
    shadowroot="#world-clock-template"
    shadowrootdata='{ "hours": 12, "minutes": 34, "seconds": 56 }'
    timezone="GMT"
  ></world-clock>
</p>

<p>The time in New York is
  <world-clock
    shadowroot="#world-clock-template"
    shadowrootdata='{ "hours": 7, "minutes": 34, "seconds": 56 }'
    timezone="EDT"
  ></world-clock>
</p>

<p>The time in Hong Kong is
  <world-clock
    shadowroot="#world-clock-template"
    shadowrootdata='{ "hours": 20, "minutes": 34, "seconds": 56 }'
    timezone="HKT"
  ></world-clock>
</p>

[emilio]

I agree that problem b) is a problem that needs a solution. Perhaps the parser could detect exactly-duplicate