feat(runtime): support declarative shadow DOM

christian-bromann commented 4 months ago

What is the current behavior?

This patch introduces support for declarative shadow DOM in Stencil 🎉

GitHub Issue Number: #4010

What is the new behavior?

This patch includes:

New Feature: new new flags to the renderToString method:
- fullDocument: allows to define whether to return a full document or just the component
- serializeShadowRoot: defines whether a component marked with shadow: true is being rendered in a declarative shadow DOM:
```
<my-cmp>
<template shadowrootmode="open">
  <style>...</style>
  <div>Hello World</div>
  <slot></slot>
</template>

Slotted content
</my-cmp>
```
  or as scoped component:
```
<my-cmp>
<div>Hello World</div>
Slotted content
</my-cmp>
```
New Feature: the hydrate module now exports a streamToString function that returns a Readable object that can be passed into a server response.
Potentially Breaking: users doing DOM snapshot tests of shadow components will need to update their snapshots as all shadow components are now serialized in a Declarative Shadow DOM
Runtime changes as suggested in #5787
ESM version of the hydrate script
Removal of isPromise helper in the code
Support for objects in component properties, e.g. renderToString(<car-detail car=${JSON.stringify({ year: 1234 })}>)
Removal of duplicated logic within renderToString and hyrdrateDocument
Rename of serializeToHtml to streamToHtml which uses generators
some changes to how we call setTimeout or setInterval in mock doc in order to enable support for using the hydrate script in the browser

Documentation

ToDo missing

Does this introduce a breaking change?

[x] Yes
[ ] No

Testing

Added e2e tests for using the hydration script in the browser as well as part of the e2e test suite.

Other information

n/a

github-actions[bot] commented 4 months ago

@stencil/core@4.18.3 ts tsc --noEmit --project scripts/tsconfig.json && tsx scripts/tech-debt-burndown-report.ts

`--strictNullChecks` error report

Typechecking with --strictNullChecks resulted in 1067 errors on this branch.

That's 13 fewer than on main! 🎉🎉🎉

reports and statistics

Our most error-prone files

| Path | Error Count | | --- | --- | | src/dev-server/index.ts | 37 | | src/dev-server/server-process.ts | 32 | | src/compiler/prerender/prerender-main.ts | 22 | | src/runtime/vdom/vdom-render.ts | 22 | | src/runtime/client-hydrate.ts | 20 | | src/runtime/vdom/test/patch.spec.ts | 19 | | src/runtime/vdom/test/util.spec.ts | 19 | | src/screenshot/connector-base.ts | 19 | | src/testing/puppeteer/puppeteer-element.ts | 19 | | src/dev-server/request-handler.ts | 15 | | src/compiler/prerender/prerender-optimize.ts | 14 | | src/compiler/sys/stencil-sys.ts | 14 | | src/runtime/connected-callback.ts | 14 | | src/sys/node/node-sys.ts | 14 | | src/compiler/prerender/prerender-queue.ts | 13 | | src/compiler/sys/in-memory-fs.ts | 13 | | src/runtime/set-value.ts | 13 | | src/compiler/output-targets/output-www.ts | 12 | | src/compiler/transformers/test/parse-vdom.spec.ts | 12 | | src/compiler/transformers/transform-utils.ts | 12 |

Our most common errors

| [Typescript Error Code](https://github.com/microsoft/TypeScript/blob/main/src/compiler/diagnosticMessages.json) | Count | | --- | --- | | TS2322 | 336 | | TS2345 | 322 | | TS18048 | 185 | | TS18047 | 99 | | TS2722 | 27 | | TS2532 | 23 | | TS2531 | 19 | | TS2790 | 11 | | TS2454 | 10 | | TS2352 | 9 | | TS2769 | 8 | | TS2416 | 7 | | TS2538 | 4 | | TS2493 | 3 | | TS18046 | 2 | | TS2684 | 1 | | TS2430 | 1 |

Unused exports report

There are 15 unused exports on this PR. That's the same number of errors on main, so at least we're not creating new ones!

Unused exports

| File | Line | Identifier | | --- | --- | --- | | src/runtime/bootstrap-lazy.ts | 21 | setNonce | | src/screenshot/screenshot-fs.ts | 18 | readScreenshotData | | src/testing/testing-utils.ts | 198 | withSilentWarn | | src/utils/index.ts | 145 | CUSTOM | | src/utils/index.ts | 245 | NODE_TYPES | | src/utils/index.ts | 269 | normalize | | src/utils/index.ts | 7 | escapeRegExpSpecialCharacters | | src/compiler/app-core/app-data.ts | 25 | BUILD | | src/compiler/app-core/app-data.ts | 116 | Env | | src/compiler/app-core/app-data.ts | 118 | NAMESPACE | | src/compiler/fs-watch/fs-watch-rebuild.ts | 123 | updateCacheFromRebuild | | src/compiler/types/validate-primary-package-output-target.ts | 82 | satisfies | | src/compiler/types/validate-primary-package-output-target.ts | 82 | Record | | src/testing/puppeteer/puppeteer-declarations.ts | 485 | WaitForEventOptions | | src/compiler/sys/fetch/write-fetch-success.ts | 7 | writeFetchSuccessSync |

github-actions[bot] commented 4 months ago

PR built and packed!

Download the tarball here: https://github.com/ionic-team/stencil/actions/runs/9569595832/artifacts/1613920781

If your browser saves files to ~/Downloads you can install it like so:

unzip -d ~/Downloads ~/Downloads/stencil-core-4.18.3-dev.1718732065.7afd94d.tgz.zip && npm install ~/Downloads/stencil-core-4.18.3-dev.1718732065.7afd94d.tgz

christian-bromann commented 3 months ago

@tanner-reits thanks for reviewing!

I am struggling reproducing the issue you are describing. I've set-up a new Stencil project with the following component:

import { Component, Prop, h } from '@stencil/core';
import { format } from '../../utils/utils';

@Component({
  tag: 'my-component',
  styleUrl: 'my-component.css',
  shadow: true,
})
export class MyComponent {
  /**
   * The first name
   */
  @Prop() first: string;

  /**
   * The middle name
   */
  @Prop() middle: string;

  /**
   * The last name
   */
  @Prop() last: string;

  private getText(): string {
    return format(this.first, this.middle, this.last);
  }

  render() {
    return <div>
      Hello, World! I'm {this.getText()}
      <slot></slot>
    </div>;
  }
}

When creating an hydration script and calling this script:

const { renderToString } = require('./hydrate');

(async () => {
  console.log(await renderToString('<my-component>Jimmy</my-component><my-component first="Bob"></my-component>', {
    prettyHtml: true,,
    serializeShadowRoot: true
  }));
})()

I correctly get the following output:

<!doctype html>
<html class="hydrated" data-stencil-build="u56d8u2z">
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <my-component class="hydrated sc-my-component-h" s-id="1">
      <template shadowrootmode="open">
        <style sty-id="sc-my-component">
          /*!@:host*/.sc-my-component-h{display:block}
        </style>
        <div c-id="1.0.0.0" class="sc-my-component">
          <!--t.1.1.1.0-->
          Hello, World! I'm
          <slot c-id="1.2.1.1" class="sc-my-component"></slot>
        </div>
      </template>
      <!--r.1-->
      Jimmy
    </my-component>
    <my-component class="hydrated sc-my-component-h" first="Bob" s-id="2">
      <template shadowrootmode="open">
        <style sty-id="sc-my-component">
          /*!@:host*/.sc-my-component-h{display:block}
        </style>
        <div c-id="2.0.0.0" class="sc-my-component">
          <!--t.2.1.1.0-->
          Hello, World! I'm Bob
          <slot c-id="2.2.1.1" class="sc-my-component"></slot>
        </div>
      </template>
      <!--r.2-->
    </my-component>
  </body>
</html>

Which just shows two elements. Mind sharing your example?

tanner-reits commented 3 months ago

@christian-bromann The renderToString method is generating the correct output, but it doesn't render correctly when handed over to the browser

mayerraphael commented 3 months ago

@christian-bromann I tried your PR build in a more advanced use case (nested components, multiple named slots) and with serializeShadowRoot in renderToString we get duplicated renderings (only for some components), which do not occur without.

With serializeShadowRoot: true

With serializeShadowRoot: false

The components that have duplicated rendering, some elements inside are missing c-id or html comments:

Tested with 4.18.3-dev.1718220868.532557e

christian-bromann commented 3 months ago

@mayerraphael thanks so much for providing feedback 🙏 is there any chance you can create a minimal reproducible example? I will try to reproduce this myself but haven't come across this behavior.

mayerraphael commented 3 months ago

@christian-bromann I invited you to my repository.

Its the same repo as before. Disable javascript in the browser to see the correct rendering (but with missing c-ids in the my-other-component), as soon as Stencil hydrates in the browser, those elements get rendered double.

With disabled JS:

After hydration:

I hope the example helps :)

christian-bromann commented 3 months ago

@mayerraphael thanks for reporting, I proposed a fix and made a dev release (4.18.3-dev.1719344120.e32bcd8) which resolves the issue.

mayerraphael commented 3 months ago

@christian-bromann Thanks, that was fast.

Fixes some duplicate rendering, but not all. Also got some missing s-ids on top level with multiple components. Will try to replicate in my repository again.

Edit:

I updated my repository again. I couldn't replicate the missing s-ids. But i got another bug with s-ids beeing set on child-components.

From our demo page we still get some duplicated rendering and missing c-id/s-ids:

thure commented 3 months ago

@christian-bromann I was working from babe807b5b72a46bfd9bd8e9c9b4cda962e25607, which correctly hydrated my components, however none of the releases on NPM will hydrate the components, they instead seem not to recognize them.

In my use-case:

  const { html } = await renderToString(
  '<ch-oklch-picker lightness="0.43" chroma="0.4" hue="256"></ch-oklch-picker>'
  {
    serializeShadowRoot: true,
    fullDocument: false,
  });

should return what babe807 returned:

<ch-oklch-picker lightness="0.43" chroma="0.4" hue="256" role="group" class="hydrated" s-id="1"><!--r.1--><label id="hue-0329" c-id="1.0.0.0"><!--t.1.1.1.0-->Hue (0–360)</label><input property="hue" aria-labelledby="hue-0329" type="number" min="0" max="360" step="1" value="256" c-id="1.2.0.1"><input property="hue" aria-labelledby="hue-0329" type="range" min="0" max="360" step="1" value="256" c-id="1.3.0.2"><label id="chroma-66e5" c-id="1.4.0.3"><!--t.1.5.1.0-->Chroma (0–0.4)</label><input property="chroma" aria-labelledby="chroma-66e5" type="number" min="0.000" max="0.400" step="0.004" value="0.4" c-id="1.6.0.4"><input property="chroma" aria-labelledby="chroma-66e5" type="range" min="0.000" max="0.400" step="0.004" value="0.4" c-id="1.7.0.5"><label id="lightness-4fd3" c-id="1.8.0.6"><!--t.1.9.1.0-->Lightness (0–1)</label><input property="lightness" aria-labelledby="lightness-4fd3" type="number" min="0.00" max="1.00" step="0.01" value="0.43" c-id="1.10.0.7"><input property="lightness" aria-labelledby="lightness-4fd3" type="range" min="0.00" max="1.00" step="0.01" value="0.43" c-id="1.11.0.8"></ch-oklch-picker>

However 4.19.0 and any of the prereleases just return the input string.

Did the API change, or is there something I should be doing to configure the hydrate app properly?

My prototype Astro integration which uses the hydrate app is here: https://github.com/ch-ui-dev/ch-ui/blob/thure/feat-astro/packages/astro-stencil/src/server.ts

thure commented 3 months ago

I’ve posted a comparison PR for the hydrate outputs of babe807 and 4.19.0 here: https://github.com/thure/hydrate-diff/pull/1/files?diff=split&w=1

Could one of these differences account for the component not hydrating?

christian-bromann commented 3 months ago

Did the API change, or is there something I should be doing to configure the hydrate app properly?

No, we've build it to be backward compatible. Let me take a look.

christian-bromann commented 3 months ago

@thure it seems like setting up your project and the branch and running this script:

import { renderToString } from './packages/elements-hydrate-temp/index.mjs'

const { html } = await renderToString(
  '<ch-oklch-picker lightness="0.43" chroma="0.4" hue="256"></ch-oklch-picker>',
  {
    serializeShadowRoot: true,
    fullDocument: false,
  }
);

console.log(html);

gives me above mentioned hydrated string. Can you provide some concrete steps I can walk through to properly reproduce what you see?

christian-bromann commented 3 months ago

I updated my repository again. I couldn't replicate the missing s-ids. But i got another bug with s-ids beeing set on child-components.

@mayerraphael thanks again for your feedback, I check out your repository, updated Stencil to v4.19.0 which we released yesterday and ran server.js which returned the following HTML code:

<my-component last-page="5" class="sc-my-component-h hydrated" s-id="1">
    <template shadowrootmode="open">
      <style sty-id="sc-my-component">
        /*!@:host*/
        .sc-my-component-h {
          display: block
        }
      </style>
      <div class="sc-my-component" c-id="1.0.0.0">
        <div class="pagination sc-my-component" c-id="1.1.1.0">
          <div class="pagination-pages pagination-notation sc-my-component" c-id="1.2.2.0">
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.3.3.0" s-id="2">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="2.0.0.0">
                  <!--t.2.1.1.0-->0
                </div>
              </template>
              <!--r.2-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.4.3.1" s-id="3">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="3.0.0.0">
                  <!--t.3.1.1.0-->1
                </div>
              </template>
              <!--r.3-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.5.3.2" s-id="4">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="4.0.0.0">
                  <!--t.4.1.1.0-->2
                </div>
              </template>
              <!--r.4-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.6.3.3" s-id="5">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="5.0.0.0">
                  <!--t.5.1.1.0-->3
                </div>
              </template>
              <!--r.5-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.7.3.4" s-id="6">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="6.0.0.0">
                  <!--t.6.1.1.0-->4
                </div>
              </template>
              <!--r.6-->
            </my-other-component>
          </div>
        </div>
      </div>
    </template>
    <!--r.1-->
  </my-component>
  <div>
    <my-other-component label="2" class="sc-my-other-component-h hydrated" s-id="7">
      <template shadowrootmode="open">
        <style sty-id="sc-my-other-component">
          /*!@:host*/
          .sc-my-other-component-h {
            display: block
          }
        </style>
        <div class="pagination-item sc-my-other-component" c-id="7.0.0.0">
          <!--t.7.1.1.0-->2
        </div>
      </template>
      <!--r.7-->
    </my-other-component>
  </div>
  <my-component last-page="2" class="sc-my-component-h hydrated" s-id="8">
    <template shadowrootmode="open">
      <style sty-id="sc-my-component">
        /*!@:host*/
        .sc-my-component-h {
          display: block
        }
      </style>
      <div class="sc-my-component" c-id="8.0.0.0">
        <div class="pagination sc-my-component" c-id="8.1.1.0">
          <div class="pagination-pages pagination-notation sc-my-component" c-id="8.2.2.0">
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="8.3.3.0" s-id="9">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="9.0.0.0">
                  <!--t.9.1.1.0-->0
                </div>
              </template>
              <!--r.9-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="8.4.3.1" s-id="10">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="10.0.0.0">
                  <!--t.10.1.1.0-->1
                </div>
              </template>
              <!--r.10-->
            </my-other-component>
          </div>
        </div>
      </div>
    </template>
    <!--r.8-->
  </my-component>
  <script type="module">
    import {
      defineCustomElements
    } from "./static/loader/index.js";
    defineCustomElements().catch(console.error);
  </script>

I can see all c-ids properly assigned. I can't find any of the patternlib components though so you might look at a different example. Can you get something reproducible for me? Thank you!

thure commented 3 months ago

@christian-bromann Yes, for sure:

git clone git@github.com:ch-ui-dev/ch-ui.git
cd ch-ui
git checkout thure/debug-stencil – this branch is one commit behind thure/feat-astro where I’d committed the hydrate app produced by babe807 into its own package, which might be why you couldn’t repro the issue
pnpm install
pnpm nx build docs
Observe that any HTML files in the build e.g. apps/docs/dist/index.html have a non-hydrated <ch-oklch-picker

If you then git checkout thure/feat-astro, which has the pinned hydrate app, then repeat steps 5 and 6, the build will have the hydrated component.

thure commented 3 months ago

I should note that ch-oklch-picker is a shadow: false component — did 4.19.0 drop support for hydrating light DOM components?

christian-bromann commented 3 months ago

did 4.19.0 drop support for hydrating light DOM components?

No!

then repeat steps 5 and 6, the build will have the hydrated component.

Unfortunately I am getting this error:

> nx run icons:build-esm

✘ [ERROR] No loader is configured for ".node" files: node_modules/.pnpm/@resvg+resvg-js-darwin-arm64@2.6.2/node_modules/@resvg/resvg-js-darwin-arm64/resvgjs.darwin-arm64.node

    node_modules/.pnpm/@resvg+resvg-js@2.6.2/node_modules/@resvg/resvg-js/js-binding.js:1:2588:
      1 │ ...eBinding=require("@resvg/resvg-js-darwin-arm64")}catch(e){loadEr...
        ╵                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

christian-bromann commented 3 months ago

@thure I think I was able to reproduce this using internal test infrastructure.

christian-bromann commented 3 months ago

@thure false alarm 🙈 it does seem with our current tests that scoped components hydrate just fine. Mind providing more minimalistic example of what you experience?

thure commented 3 months ago

@christian-bromann I’ve set up a Codesandbox with the hydrate apps produced by both babe807 and 4.19.0 — these are the same files as the diff I provided earlier.

If you have the ch-ui repo locally and want to build the hydrate app yourself, you could cd packages/elements and run pnpm link $pathToYourLocalStencil, then switch to different versions in your local stencil project (rebuild stencil when doing so) and run pnpm nx build elements in the root of ch-ui to see the different outputs. The elements package doesn’t depend on the icons package so you shouldn’t encounter the issue you encountered trying to build the docs.

I can isolate @ch-ui/elements if that would help, will just need some time to do so, let me know.

mayerraphael commented 3 months ago

I updated my repository again. I couldn't replicate the missing s-ids. But i got another bug with s-ids beeing set on child-components.

@mayerraphael thanks again for your feedback, I check out your repository, updated Stencil to v4.19.0 which we released yesterday and ran server.js which returned the following HTML code:
...
  </script>
I can see all c-ids properly assigned. I can't find any of the patternlib components though so you might look at a different example. Can you get something reproducible for me? Thank you!

Sorry those patternlib components are some internal ones.

In the debugger it looks like addHostEventListeners crashes, which is inside the hydrateComponent function call.

Exception has occurred: TypeError: Cannot read properties of undefined (reading 'addEventListener')
  at Object.ael
    at hydrate\index.js:1712:11
    at Array.map (<anonymous>)
    at addHostEventListeners (\hydrate\index.js:1708:15)
    at hydrateComponent (\hydrate\index.js:2071:7)
    at connectElement2 (\hydrate\index.js:2030:18)

I am not sure why this is used inside hydrateComponent, but it is always undefined. Components that have a @Listen() crash.

@christian-bromann I updated my example with an @Listen(), this replicated the error:

The my-whatever-component crashes and does not render/hydrate:

That was a tough one :)

Repo: https://github.com/mayerraphael/stencil-dsd-ssr-playground

Edit:

This also affects the "old" serializeShadowRoot: false.

mayerraphael commented 3 months ago

@christian-bromann

I issued individual tickets as i found more problems with the new Version:

https://github.com/ionic-team/stencil/issues/5869 https://github.com/ionic-team/stencil/issues/5870

christian-bromann commented 3 months ago

@mayerraphael thanks a lot! I will take a look.

ionic-team / stencil