[defer-hydration] Requesting hydration of a disconnected component

While hydration usually applies to elements prerendered to the document, there are occasionally use cases where a prerendered element may be hydrated while disconnected from the document. I get that probably sounds like an oxymoron, however I think of five such use cases:

An element is pre-rendered with defer-hydration and does not hydrate. The user then removes this element from the DOM, removes the defer-hydration attribute while disconnected, and then reconnects the component.
An element hydrates not with its own content, but instead from cloning a template stored separately in the document. This allows prerendering a component template once, and then reusing that template for many different instances of the component.
An element is constructed in memory with attributes, properties, and child nodes added to it which can be read during hydration. This effectively allows "client-side props" to be supplied before hydration and used in that process.
- In particular, I think this is useful for the process of "reflecting JS state back onto the component DOM". Sometimes "hydration" isn't just reading DOM state into JS, but also taking client-side state not known at render time (logged in user for example) and rendering it. This requires a component to receive this state from the rest of the page before it hydrates and if it's done in-memory, then hydrating a disconnected element becomes a problem to solve.
HTML fragments performs an XHR fetch of HTML content and parses it to disconnected DOM elements. These elements should be hydrated so they will be in a valid state to manipulate even before being appended to the DOM.
An element is pre-rendered inside a <template /> element and then cloned, hydrated with some input props, and then appended to the DOM.

While the use cases are definitely nuanced, I think there's value in a community protocol for web components to expose some kind of functionality to trigger hydration even when they are disconnected from the DOM. Hydration is often critical to initialize a component and make it functional. A counter component which exposes an increment() method can't really be implemented prior to the initial count being hydrated from prerendered HTML. It should be possible to construct a component, set some initial properties, hydrate it, interact with the initialized component (increment the count one extra time for example), and then append it to the DOM. Essentially, I want to be able to write something like:

const fragment = new DOMParser().parseFromString(`
  <my-counter>
    <div>The count for user named <span>-</span> is: <span>5</span>.</div>
  </my-counter>
`.trim(), 'text/html');
const counter = document.adoptNode(fragment.body.firstElementChild);
customElements.upgrade(counter);

// Some client-side props to use during hydration.
counter.user = { name: 'Doug' };

// Component not yet in a valid state, hasn't been hydrated.
// counter.increment();

// Hydrate the initial count.
hydrateTheComponentSomehow(counter); // How should this work???
// DOM now holds `<div>The count for user named <span>Doug</span> is: <span>5</span>.</div>`.

// Component should be in a valid state.
counter.increment();
// DOM now holds `<div>The count for user named <span>Doug</span> is: <span>6</span>.</div>`.

document.body.appendChild(counter); // Display to the user.

(I might be misusing document.adoptNode() and customElements.upgrade(), I find their nuances very confusing, but I don't think it's actually that related to this use case.)

A community protocol around triggering hydration for disconnected components would be valuable for libraries and tools which process prerendered HTML in various ways and convert them to hydrated components. From what I've seen, hydration tends to happen when the component is first connected to the document, but this means the component is in an invalid, unhydrated state until it is appended to the document and there is no way around that restriction. It is reasonable for the component to be non-functional when in this invalid state, but that means it can never become valid until it is appended to the document and displayed to the user.

Here are some potential ideas for how this protocol could work:

1. Use the `defer-hydration` attribute

We already have a defer-hydration attribute proposal which can trigger hydration on removal for components in the document. It seems reasonable that a component could hydrate itself when this attribute is removed, even if the component itself is not connected to a document. This is possible, though a little weird since you have to write:

const counter = document.createElement('my-counter');
counter.setAttribute('defer-hydration', '');
counter.initialValue = 5;
counter.removeAttribute('defer-hydration'); // Trigger hydration here.
counter.increment(); // Increments to `6`.
document.body.appendChild(counter);

It's pretty strange to set the defer-hydration attribute on the custom element just to remove it to trigger hydration. It's doubly confusing that counter is upgraded and its constructor runs during document.createElement(). Meaning any component which hydrates from its constructor would break this protocol because defer-hydration cannot be set in time to prevent it. It can also be unintuitive to author a component and expect to handle the admittedly very specific case of removing defer-hydration while disconnected from the document.

This also raises the question of "should a component hydrate when connected to the document without defer-hydration, or when defer-hydration is removed"? To support disconnected hydration, we need to take the latter approach, while I imagine most custom elements probably go with the former today.

2. Define a `.hydrate()` method

We can define a .hydrate() method which triggers hydration if it has not run already. This gives an opportunity for users to construct an element, modify its attributes, children, and properties arbitrarily, and then trigger hydration when ready to get the component in a valid, usable state before appending to the DOM.

If the component does not have a hydrate() method, it will be observed as undefined and any component which uses is can interpret this as though the component does not require hydration.

const fragment = new DOMParser().parseFromString(/* ... */);
const counter = document.adoptNode(fragment.body.firstElementChild);
customElements.upgrade(counter);

counter.initialValue = 5;
counter.hydrate?.(); // Trigger hydration if required.

counter.increment(); // Increments to `6`.
document.body.appendChild(counter);

I think this is the most straightforward approach, but I get the concern that this is yet another property to implement for every component.

There is an argument to be made that .hydrate() should optionally return a Promise, so the component can do async work as part of its hydration. I think there's a separate conversation to be had about whether or not hydration is fundamentally synchronous or asynchronous. However, given that the current defer-hydration proposal requires synchronous hydration, I think this should be limited to match.

3. Hydrate on component upgrade.

For prerendered components in the main document, hydration typically happens on connectedCallback() which is usually invoked when customElements.define('my-component', MyComponent) is executed (unless the component or its usage specifically opts out of hydration). This is effectively done at the same time as web component upgrade. The component class is defined, all of its instances of the page are upgraded, and then connectedCallback() is invoked which is the ideal moment for most non-deferred hydration. We could do the same thing for disconnected elements, and use customElements.upgrade() as the trigger for hydration. I see three challenges here:

There is no upgradeCallback() hook, but the constructor function can serve this function, albeit in a confusing way IMHO.
Constructing an element in the same document upgrades it immediately in document.createElement('my-component') before there is any opportunity to provide any properties, children, or attributes.
- You can work around this by constructing in a different document (document.implementation.createHTMLDocument().createElement('my-component') but this is very nuanced and likely expensive given that you need to create an entirely new Document, just to throw it away.
Setting native JS properties on an un-upgraded component, then triggering upgrade is very likely to delete those properties due to nuances in the spec. This means you either can't use native JS fields in the hydrating custom element, or it needs to be upgraded prior to those properties being set. This also implies that the component must hydrate without any of those properties.
- https://blog.dwac.dev/posts/streamable-html-fragments/#custom-element-constructors-and-field-initializers

This approach feels completely impractical to me. document.createElement() upgrades too eagerly and the native JS properties nuance forces any properties to be assigned after hydration, which is very limiting and easy to mess up IMHO.

4. Hydrate lazily.

In the original example, you can argue that counter.increment() should just lazily hydrate, since the component must be in a hydrated state in order to increment the current count.

const fragment = new DOMParser().parseFromString(/* ... */);
const counter = document.adoptNode(fragment.body.firstElementChild);
customElements.upgrade(counter);
counter.user = { name: 'Doug' };
// Component not yet in a valid state, hasn't been hydrated.

// Lazily hydrate and increment the count.
counter.increment();

document.body.appendChild(counter); // Display to the user.

I can see the value here since it means you don't necessarily need to think of the component as in a valid or invalid state.

Personally I'm not a fan of this approach because it means that every operation on the component must have knowledge of whether it is in a valid state and automatically correct that. This makes things particularly complicated for component libraries which may want to abstract away hydration timing, yet any user-exposed function needs to check if the component is hydrated, or the library needs to "magic away" that problem. Simple patterns like extending a class which handles hydration becomes a lot more complicated since there aren't easy hooks to hydrate automatically.

class MyComponent extends LibraryComponentWithHydration {
  doAThing(): void {
    super.hydrateIfNecessary();

    // ...
  }
}

This approach also means that it is very easy to accidentally trigger a potentially expensive hydration step without realizing it. It is also not possible to pay that cost early (in the counter example, you can't hydrate without also incrementing). The component could expose its own implementation-specific hydrate() method, but if this is not part of the agreed-upon community protocol, then it can't be used in a generic fashion without knowledge of the specific component.

Personally I like approaches 1. or 2. the best since they seem the most feasible and ergonomic. Curious to hear what others think or if anyone else has encountered this particular problem and has any interest in coming up with a solution.

I don't see why the attribute doesn't just work here.

If you create an element imperatively, it doesn't need hydration so the API doesn't matter. If you create it from done pre-rendered DOM, then removing the attribute with trigger hydration even for disconnected elements.

I don't ever see a need for imperatively adding then removing the attribute.

I don't see why the attribute doesn't just work here.

If you create an element imperatively, it doesn't need hydration so the API doesn't matter. If you create it from done pre-rendered DOM, then removing the attribute with trigger hydration even for disconnected elements.

It definitely can work. I think there's three potential points of contention worth discussing:

Adding then immediately removing defer-hydration is weird (maybe not that weird, could be fine).
Should a component hydrate when defer-hydration is removed, or when it is removed while connected to the DOM? Basically should we trigger hydration on attributeChangedCallback() always, or can we wait and trigger on the subsequent connectedCallback()? I was previously implementing the later because I thought it didn't matter, but after encountering this problem I realized it does matter and we need to hydrate in attributeChangedCallback(). The current proposal is ambiguous about this point, and we would probably want to call it out as it is easy to forget about this use case.
You can't easily construct a detached element with defer-hydration already set. Calling document.createElement() invokes the constructor before the user has any opportunity to set defer-hydration. That means components can not hydrate in the constructor since they have receive defer-hydration later. I'm not sure how common constructor hydration is (my guess is most do it in connectedCallback()), but again this is another restriction we would need to specify in the protocol.

I don't ever see a need for imperatively adding then removing the attribute.

I listed several use cases in the first comment, are those not compelling enough examples?

The exact motivation for this issue was when I tried use case 2. (admittedly one of the weirder ones) in https://github.com/dgp1130/HydroActive/commit/0e2fa9234a03eabbcd96af911271179d3eef0c5c (functional.ts requestHydration() clones the template). I had to do the add and immediately remove defer-hydration trick (functional.ts factory()). It took me a while to come up with that answer and I'm still not totally sure it's aligned with the protocol which is why I filed this. You can see example usage in templating.html and templated-counter.ts, the component clones a template on hydration and initializes it so users can spawn multiple counters with different initial values, yet the component's internal DOM is all SSR'd.

I don't ever see a need for imperatively adding then removing the attribute.

I listed several use cases in the first comment, are those not compelling enough examples?

I just don't understand why you would ever need to do that. Why not just:

const counter = document.createElement('my-counter');
counter.initialValue = 5;
counter.increment(); // Increments to `6`.
document.body.appendChild(counter);

What is gained by triggering hydration? It would do nothing.

@justinfagnani, you're example is assuming it is client side rendered. For a server-side rendered counter component, it needs to hydrate. A more concrete example might be:

<template id="my-template">
  <my-counter>
    <template shadowrootmode="open">
      <div>The current count is <span>5</span>.</div>
      <button>Increment</button>
    </template>
  </my-counter>
</template>

const template = document.getElementById('my-template');
const counter = template.content.cloneNode(true /* deep */);

// I think you need to adopt and/or upgrade here?
document.adoptNode(counter);
customElements.upgrade(counter);

// Add and remove the `defer-hydration` attribute to force hydration.
counter.setAttribute('defer-hydration', '');
counter.removeAttribute('defer-hydration');

// Component is usable.
counter.increment(); // Increment to 6.

document.body.append(counter);

In this example, we need to hydrate before we can increment, since the initial value came from the prerendered HTML. But since it's in a template, we can actually do this while the component is still disconnected from the DOM.

If it's server-side rendered and the component supports the defer-hydration protocol, then the server response should have the defer-hydration attribute on it. In this case the serialized form of the element should be:

<template id="my-template">
  <my-counter defer-hydration>
    <template shadowrootmode="open">
      <div>The current count is <span>5</span>.</div>
      <button>Increment</button>
    </template>
  </my-counter>
</template>

I'm not sure why there would be a template with a declarative shadow root in it. It seems like you'd want to dynamically create instances without loading the element definition? Where are you loading the definition in your example? Because without one adding and removing the attribute won't do anything, and with the definition loaded I'm not sure why it'd have a declarative shadow root rather than just letting it make one.

It might be useful to state the need for a protocol here. defer-hydration is needed because we want an interoperable and serializable signal that an element should wait. It needs to be something a SSR library could do to all custom elements generically.

If you're not serializing that signal, and you're calling API directly on the element, then you don't really have the two important needs for defer-hydration. You could just put API right on the element:

const template = document.getElementById('my-template');
const counter = document.importNode(template.content, true);
counter.hydrate();
counter.increment(); // Increment to 6.

edit: to be clearer here ^ you can just put hydrate() on your component without a community protocol. Protocols are need for interoperability only.

@justinfagnani I included declarative shadow DOM mostly out of habit. The JS definition could be anywhere else on the page. I don't think it meaningfully changes the example to write:

<template id="my-template">
  <my-counter>
    <div>The current count is <span>5</span>.</div>
    <button>Increment</button>
  </my-counter>
</template>
<script src="/my-counter.js"></script>

The point is that my-counter is not usable until it has hydrated the initial count value of 5 from the prerendered HTML, which leaves open the question of how we trigger that hydration?

To give a little more context: my original motivation was to make it possible to SSR a template and then clone that template automatically when the associated component was constructed on the client (use case 2 in my first comment). It would allow SSR for elements which are dynamically created, such as a "spawn counter" button which creates a new counter on each click, but reuses the same SSR'd template. https://github.com/dgp1130/HydroActive/commit/0e2fa9234a03eabbcd96af911271179d3eef0c5c#diff-13398743febf4827a1490a23c7607b305f34ea5571c47a56195f468bc5c22cc7

I eventually realized I had that backwards, and instead of trying to put a reusable template inside a component, I should put a component inside a reusable template and restructured things like the above snippet (use case 5 in my original comment). https://github.com/dgp1130/HydroActive/commit/519e458176e3c6a8b882e2b352ce65da03f85123#diff-13398743febf4827a1490a23c7607b305f34ea5571c47a56195f468bc5c22cc7

Restructuring things like that made me want to write a generic function to hydrate an unknown component and I realized this was also a great opportunity to set initial properties on the component before hydration so the component can leverage that information in combination with its prerendered DOM (use case 3). The component probably comes from a template, but in theory it could come from anywhere. Since hydration is a generic operation, I wanted to write something like:

const template = document.getElementById('my-template');
const counter = template.content.cloneNode(true /* deep */);

// Given a dehydrated component disconnected from the DOM:
// 1. Upgrade it if not already upgraded.
// 2. Assign properties (`counter.user = props.user;`)
// 3. Hydrate the component.
hydrate(counter, { user: { getName: () => 'Devel' } });

counter.increment();

document.appendChild(counter);

The hydrate() function should be generic, it doesn't need to know that it is given my-counter specifically. So how would one implement hydrate() in a generic fashion? My attempt is here, but it relies on setting/unsetting the defer-hydration attribute, and I'm not sure that's the right approach here, which is why I filed this issue. There's value in a community protocol here given that "hydrating a component" is a generic operation, and so is "hydrating a component disconnected from the DOM". It should be possible to hydrate any component with any (defer-hydration-compatible) implementation, not just components I've authored myself. That means it needs to be a specified part of the protocol, not just a thing this one component implements.

By doing this a component is able to be hydrated not just from its internal DOM, but also from any input properties provided by a parent. Take for example a counter which reads its initial count from SSR'd DOM, but also has an associated user object provided by the client (non-serializable) and can leverage both at hydration time. We can spawn as many counters as we want for as many users as we want without having to CSR the component or ship any such logic to the client. I threw together a quick demo to show this. It's able to prerender a single my-counter element using defer-hydration, but also dynamically spawn more counters from a template while depending on client-side non-serializable state at hydration time.

https://stackblitz.com/edit/typescript-knym5u?file=index.ts

Hopefully that gives a little more insight into what kind of use cases might benefit from hydrating a disconnected component as well as why a generic hydrate() function would be useful. This uses just one possible implementation of this generic hydrate() function, I think my goal for this issue is identifying the right way of implementing that function which would work for any component.

To your point about: "Shouldn't the templated element include defer-hydration?", I think that is a valid argument. Because my approach sets/unsets the defer-hydration attribute and cloning a node from a different document doesn't immediately upgrade it, I'm able to make something which works without requiring defer-hydration. Maybe that's an anti-pattern and it should require defer-hydration in that context. However this generic hydrate() function doesn't know or care how the input element was created. Maybe it was prerendered by the server, maybe it came from a template (use case 5), maybe it was dynamically constructed from DOM APIs on the client (use case 2), or parsed from an HTTP response with DOMParser (use case 4).

To invert the question a little bit, should a defer-hydration compatible component hydrate when it is constructed or when it is connected to the DOM? Should the counter hydrate when not deferred by doing:

// 1. Hydrate in constructor.
class MyCounter extends HTMLElement {
  constructor() {
    super();
    if (!this.hasAttribute('defer-hydration')) this.hydrate();
  }
}

// OR

// 2. Hydrate when connected and not deferred.
class MyCounter extends HTMLElement {
  connectedCallback() {
    if (!this.hasAttribute('defer-hydration')) this.hydrate();
  }
}

Similarly, should the component hydrate when the defer-hydration attribute is removed, or when the attribute is removed and it is connected the DOM?

// 3. Hydrate when `defer-hydration` is removed, even if disconnected.
class MyCounter extends HTMLElement {
  static observedAttributes = [ 'defer-hydration' ];
  attributeChangedCallback(name: string, oldValue: string | null, newValue: string | null) {
    // Hydrate immediately, even if disconnected.
    if (name === 'defer-hydration' && newValue === null) this.hydrate();
  }
}

// OR

// 4. Do nothing when `defer-hydration` is removed and disconnected. Wait until connected to the DOM afterwards.
class MyCounter extends HTMLElement {
  static observedAttributes = [ 'defer-hydration' ];
  attributeChangedCallback(name: string, oldValue: string | null, newValue: string | null) {
    // Only hydrate when already connected. If we're disconnected, then a future `connectedCallback()` invocation will hydrate.
    if (name === 'defer-hydration' && newValue === null && this.isConnected()) this.hydrate();
  }
}

The choice between 1. and 2. affects how a hydratable component can be constructed. 1. performs hydration at the same time as upgrades, which does have a certain intuitiveness to it. Cloning from a template could work with either approach. However document.createElement('my-counter') would immediately hydrate in the case of 1. before there is any opportunity to initialize any state or even set defer-hydration. Using approach 1. also means there is no way to decouple component upgrade from hydration, which is particularly annoying when using native JS class fields.

The choice between 3. and 4. affects whether a component even can hydrate when disconnected or if hydration is always deferred until it is connected to the document.

I think any of these approaches can be valid in isolation for any given component, I know I've been inconsistent about which approach to use for any given component I've written. But they have significant implications for how the hydrate() function should be authored. 1. & 4. basically invalidate hydrate() as a concept and it requires 2. and 3. in combination to be viable. I'm not sure how strongly the defer-hydration protocol should be about exact hydration timing, but doing so could unlock more unique use cases like those described and shown in the demo. I think some of the important questions here are:

Exactly when should a component hydrate when not deferred? 1. vs 2.
Exactly when should a component hydrate when it is deferred? 3. vs 4.
Does the protocol require or allow components to hydrate when disconnected?
What would be the correct implementation of a generic hydrate() function?
Should components cloned from a template require defer-hydration?
Should hydration support other input properties from the client before hydration time?

There's a lot of ideas here and some of them are pretty out there as I've been experimenting in this space, so apologies if this is hard to follow or make sense of. Some of these questions could probably be their own issues, but they all relate to each other in weird ways which is why they all came together here. Hopefully there's some interesting or productive ideas here at least. 😅

One other question which just came to me: How do we detect whether or not a component is already hydrated? A naive solution might look like:

function isHydrated(el: Element): boolean {
  return !el.hasAttribute('defer-hydration');
}

However there are a couple edge cases where this is incorrect:

First, users can re-add defer-hydration.

// HTML:
// <some-component defer-hydration></some-component>

const el = document.querySelector('some-component');
isHydrated(el); // false

el.removeAttribute('defer-hydration'); // Hydrates.
isHydrated(el); // true

el.setAttribute('defer-hydration', ''); // ???
isHydrated(el); // false, incorrect.

I'm not sure why you would do this, but I could see some code which sets arbitrary attributes and may have a name collision with some other interpretation of defer-hydration. Also can just happen due to programmer error. It's probably a rare problem in practice, but certainly possible.

Second (and why I'm bringing it up in this bug), disconnected components. This actually depends on exactly when hydration happens for disconnected components. To modify one of my previous examples:

<template id="my-template">
  <my-counter>
    <template shadowrootmode="open">
      <div>The current count is <span>5</span>.</div>
      <button>Increment</button>
    </template>
  </my-counter>
</template>

const template = document.getElementById('my-template');
const counter = template.content.cloneNode(true /* deep */);
isHydrated(counter); // true, incorrect but can be fixed if we check whether the component is upgraded.

// I think you need to adopt and/or upgrade here?
document.adoptNode(counter);
customElements.upgrade(counter);
// ^--- Does hydration happen here?

isHydrated(counter); // true, only correct if hydration happens in the constructor.

document.body.append(counter); // Or does hydration happen here?
isHydrated(counter); // true, correct

counter.remove();
isHydrated(counter); // true, correct

Based on the above, a slightly more correct implementation would look like:

function isHydrated(el: Element): boolean {
  // Can determine whether the constructor is called by checking if the element has been upgraded.
  return customElements.get(el.tagName.toLowerCase()) && !el.hasAttribute('defer-hydration');
}

That would change the first isHydrated call to false, and correct that answer.

This goes back to an earlier question: Do components hydrate on construction or connection? If they hydrate on construction, then checking whether the custom element is defined and defer-hydration is probably accurate. However, if hydration happens on connection, then we actually can't answer this question because we cannot know if a currently disconnected component was ever connected previously. Consider:

If a disconnected component has ever been connected before (without defer-hydration), then the component has already hydrated.
If a disconnected component has never been connected (or connected with defer-hydration), then it has not hydrated.

We can't distinguish those two cases. The only solution is to require a component to track whether or not it has hydrated (this is also necessary if we assume it is possible for users to erroneously re-add defer-hydration to an already hydrated component).

class MyElement extends HTMLElement {
  public hydrated?: true; // Should never be `false`, only `undefined` prior to hydration.

  private hydrate(): void {
    this.hydrated = true;
  }
}

function isHydrated(el: Element): boolean {
  return el.hydrated ?? false;
}

This is a lot of questions and not a whole lot of answers. I think the process we should go through here is:

Answer the question: "When does hydration happen? Is it when a component is constructed without defer-hydration or when it is connected without defer-hydration?"
Answer the question: Is re-adding defer-hydration is a supported action?
Based on the above questions: Is it possible to write an isHydrated function?
If isHydrated is not possible, should we build a mechanism into the community protocol for tracking this information? We can always decide the protocol shouldn't support this, but I'd like for that to be a deliberate decision at least.

Hopefully there are some interesting thoughts here and a reasonable path forward to identify answers.

webcomponents-cg / community-protocols