enhancement/Issue-1118: Single File Bundles for SSR and API routes

Related Issue

Resolves #1118

Summary of Changes

Refactor the API Rollup config generation to create a single entry point for each API route, rather than a single Rollup process for all API routes.
Refactor the SSR Rollup config generation to mimic the API generation, in that, the SSR array is now separate input config files.
Refactor SSR pages entry point "facade" implementation in bundle lifecycle to use URL so we can bundle instead of the hacky string replace solution
Documented breaking changes in adapter plugin docs

Demo implementation in - https://github.com/ProjectEvergreen/greenwood-demo-adapter-vercel/pull/22

TODO

[x] Documentation
[x] automatic SSR chunks / bundle mapping (nice to have)
- actually, this is just the same SSR pages issue with us now using URL, otherwise it would just be inlined, fwiw
- just did this for API routes for now, will circle back on SSR pages as part of https://github.com/ProjectEvergreen/greenwood/issues/1163
[x] fix windows test cases
[x] WCC Upstreams and delete patches/ + postinstall
- Some sort of bug with clobbered custom element definitions - https://github.com/ProjectEvergreen/wcc/issues/144
- Seems like I may have introduced an infinite loop bug? 😮‍💨 - https://github.com/ProjectEvergreen/wcc/issues/147
[x] Should create a discussion to see if there's a better alternative to the URL mechanism, since its good for bundling, but forces a chunk since it really can't be inlined... which isn't the worst thing in the world I suppose 🤷‍♂️
- should capture this a comment in adapter plugins that have to find / filter for route chunk files to copy over to adapter output directories
- https://github.com/ProjectEvergreen/greenwood/discussions/1204

@thescientist13

I'm not totally sure if this PR is ready. At the moment, there are ~50+~ 80+ test failures, all seemingly around the fetch call to the Artists API. One example is here: https://github.com/ProjectEvergreen/greenwood/blob/8fad6f78d337c1d836abc1571e5ba34590c41487/packages/cli/test/cases/build.default.ssr-static-export/src/pages/artists.js#L34

If that call does not return an array, the suite fails. I attempted to add some defensive code as a test, which resulted in a single failure of a spec due to a mismatch of values. The code below allows for the process to continue, but it doesn't produce the desired outcome...obviously.

const artistsListItems = Array.isArray(artists) ? artists : []
    .filter(artist => artist.isActive === '1')
    .map((artist) => {

When hitting the API directly in the browser, I get the same error I'm seeing from the fetch https://www.analogstudios.net/api/artists

My bad @DevLab2425 🤦

Please see https://github.com/ProjectEvergreen/greenwood/pull/1188 for the fix 😇

@thescientist13 I'm struggling to make sense of what I'm seeing at the moment and starting to feel like I'm chasing my tail.

If I isolate a suite of tests, that suite will typically pass. However, anytime I attempt to run the full set of tests, I get errors from various suites outside of the bundling logic I've changed.

Here's an example of the failures I'm seeing outside of the build.config.plugins-adapter.spec.js.

As I work through the errors I'm seeing things like in-use ports, missing file imports, etc. I'm just not sure whether what I'm seeing is environmental or code-based due to my changes. Could we setup some time later this week to help me see where my blast zone might be, and what might be a red herring?

Also, why type of tests are running with test:exp? They seem to be E2E since the CLI is multi-layered, but, are the test cases independent/standalone, or do they rely on suites running in sequence? Is using .only a recommended approach for this? (I thought that's what you were doing during the walkthrough).

The latest suite I'm chasing is the serve.default.api.spec.js suite. The api/fragment.js is failing to be created, but the other API files do get created for those tests. That sounds like it could be bundling related for sure, but since other files are being generated, maybe not?

Serve command with API specific behaviors for an HTML ("fragment") API
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/Users/paul/Projects/ProjectEvergreen/greenwood/packages/cli/test/cases/serve.default.api/public/api/fragment.js' imported from /Users/paul/Projects/ProjectEvergreen/greenwood/packages/cli/src/lifecycles/serve.js
    at new NodeError (node:internal/errors:393:5)
    at finalizeResolution (node:internal/modules/esm/resolve:323:11)
    at moduleResolve (node:internal/modules/esm/resolve:916:10)
    at defaultResolve (node:internal/modules/esm/resolve:1124:11)
    at nextResolve (node:internal/modules/esm/loader:163:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:841:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
    at ESMLoader.import (node:internal/modules/esm/loader:525:22)
    at importModuleDynamically (node:internal/modules/esm/translators:110:35)
    at importModuleDynamicallyCallback (node:internal/process/esm_loader:35:14) {
  code: 'ERR_MODULE_NOT_FOUND'
}

 0 passing (10s)
 4 failing

🤷

As I work through the errors I'm seeing things like in-use ports, missing file imports, etc. I'm just not sure whether what I'm seeing is environmental or code-based due to my changes.

Could you be running other Greenwood project's at the same time by chance, occupying ports 1984 or 8080 / 8181 (I definitely do this from time to time)? Also, depending on the process exits when running tests, it might not teardown the server running as part of a test case, thus causing a zombie process.

One way to confirm if a process is running is run yarn develop or yarn serve in this project and see if either of them crash. You can also do a ps aux | grep node to find running Node instances and see if develop or serve commands are running. You can then kill the process using kill -9 <PID>

Another tip is to pass the debug flag to new Runner to get all console output from Greenwood

runner = new Runner(true);

Fwiw, I ran all the tests locally from this branch and only got 8 tests failing, exclusively related to API behaviors, which is also what CI is reporting, so seems like your blast radius is small 💣 😃

    3070 passing (5m)
  30 pending
  8 failing

  1) Serve Greenwood With:
       API Routes
         Serve command with API specific behaviors for an HTML ("fragment") API
           should return a 200 status:

      AssertionError: expected 500 to equal 200
      + expected - actual

      -500
      +200

      at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/serve.default.api/serve.default.api.spec.js:109:36)
      at process.processImmediate (node:internal/timers:471:21)

  2) Serve Greenwood With:
       API Routes
         Serve command with API specific behaviors for an HTML ("fragment") API
           should return a custom status message:

      AssertionError: expected 'Internal Server Error' to equal 'SUCCESS!!!'
      + expected - actual

      -Internal Server Error
      +SUCCESS!!!

      at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/serve.default.api/serve.default.api.spec.js:114:40)
      at process.processImmediate (node:internal/timers:471:21)

  3) Serve Greenwood With:
       API Routes
         Serve command with API specific behaviors for an HTML ("fragment") API
           should return the correct content type:

      AssertionError: expected 'text/plain; charset=utf-8' to equal 'text/html'
      + expected - actual

      -text/plain; charset=utf-8
      +text/html

      at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/serve.default.api/serve.default.api.spec.js:119:57)
      at process.processImmediate (node:internal/timers:471:21)

  4) Serve Greenwood With:
       API Routes
         Serve command with API specific behaviors for an HTML ("fragment") API
           should return the correct response body:
     AssertionError: expected 'Internal Server Error' to include 'Hello Greenwood!!!'
      at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/serve.default.api/serve.default.api.spec.js:124:25)
      at process.processImmediate (node:internal/timers:471:21)

  5) Serve Greenwood With:
       A Server Rendered Application (SSR) with API Routes importing TypeScript
         Serve command with API specific behaviors for an HTML ("fragment") API
           should return a 200 status:

      AssertionError: expected 500 to equal 200
      + expected - actual

      -500
      +200

      at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/plugin-typescript/test/cases/exp-serve.ssr/exp-serve.ssr.spec.js:75:36)
      at process.processImmediate (node:internal/timers:471:21)

  6) Serve Greenwood With:
       A Server Rendered Application (SSR) with API Routes importing TypeScript
         Serve command with API specific behaviors for an HTML ("fragment") API
           should return a custom status message:

      AssertionError: expected 'Internal Server Error' to equal 'OK'
      + expected - actual

      -Internal Server Error
      +OK

      at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/plugin-typescript/test/cases/exp-serve.ssr/exp-serve.ssr.spec.js:79:40)
      at process.processImmediate (node:internal/timers:471:21)

  7) Serve Greenwood With:
       A Server Rendered Application (SSR) with API Routes importing TypeScript
         Serve command with API specific behaviors for an HTML ("fragment") API
           should return the correct content type:

      AssertionError: expected 'text/plain; charset=utf-8' to equal 'text/html'
      + expected - actual

      -text/plain; charset=utf-8
      +text/html

      at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/plugin-typescript/test/cases/exp-serve.ssr/exp-serve.ssr.spec.js:83:57)
      at process.processImmediate (node:internal/timers:471:21)

  8) Serve Greenwood With:
       A Server Rendered Application (SSR) with API Routes importing TypeScript
         Serve command with API specific behaviors for an HTML ("fragment") API
           should make sure to have the expected CSS inlined into the page for each :

      AssertionError: expected 0 to equal 2
      + expected - actual

      -0
      +2

      at Context. (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/plugin-typescript/test/cases/exp-serve.ssr/exp-serve.ssr.spec.js:89:42)
      at process.processImmediate (node:internal/timers:471:21)

error Command failed with exit code 8.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Also, why type of tests are running with test:exp? They seem to be E2E since the CLI is multi-layered, but, are the test cases independent/standalone, or do they rely on suites running in sequence?

All test cases are pretty much the same, they run the CLI and validate some output, so in the context of a CLI app I suppose they would be E2E tests? (technically though for something like the static router feature, I would love to run it in a browser). But no, the answer is more boring than that.

The test:exp is just a way to distinguish tests that need to run with the --experimental-loader flag for NodeJS set, that enables Greenwood to support Custom Imports. You will see these test cases prefixed with exp- in their folder / file name.

Is using .only a recommended approach for this? (I thought that's what you were doing during the walkthrough).

Not related to test:exp, it's a just a convenient way to "solo" / isolate instead of running all the specs. Often I'm just iterating TDD style with one test case until I get it right, then I run against the rest and hope for the best. 😅

But then if others fail, I will .only my way towards solving them all.

The latest suite I'm chasing is the serve.default.api.spec.js suite. The api/fragment.js is failing to be created, but the other API files do get created for those tests. That sounds like it could be bundling related for sure, but since other files are being generated, maybe not?

I wonder if it could be related to https://github.com/ProjectEvergreen/greenwood/pull/1186#discussion_r1418257440? I confirmed that rollup config had an entry when logging input but yeah, no output file. I would see if refactoring those for loops helps and I can take a look into it as well, since the code seems pretty on the 💰 from what I can tell.

I have passing tests locally, but the CI is still failing many times. I'll use the CI reports to try to track things down, but having difficulty identifying things without failures locally.

@DevLab2425 OK, so I think I found the culprit. Looks like Greenwood fails on Node 18.19.0, which is the version I see running in GitHub Actions. If I nvm use 18.19.0 on my machine, I can reproduce the error in CI and with some additional logging can get some helpful information. I basically see a lot of errors like this

TypeError: Failed to parse URL from file://file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/node_modules/lit-html/lit-html.js.map
    at new Request (node:internal/deps/undici/undici:5272:19)
    at NodeModulesResource.resolve (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/plugins/resource/plugin-node-modules.js:41:12)
    ... 5 lines matching cause stack trace ...
    at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
    at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
    at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37 {
  [cause]: TypeError [ERR_INVALID_URL]: Invalid URL
      at new NodeError (node:internal/errors:405:5)
      at new URL (node:internal/url:676:13)
      at new Request (node:internal/deps/undici/undici:5270:25)
      at NodeModulesResource.resolve (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/plugins/resource/plugin-node-modules.js:41:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:44:29
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37
      at async file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/src/lifecycles/serve.js:42:37 {
    input: 'file://file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/node_modules/lit-html/lit-html.js.map',
    code: 'ERR_INVALID_URL'
  }
}

Taking a look, maybe it's something was doing wrong, or perhaps a regression in Node? What version are you using? Perhaps something related to package exports changed or went unflagged or something?

Worst case we can pin CI to 18.15.0 and i can open issue to tracking 18.19.0 compat. Will report back! 🫡

edit 2: OOOOOOH, I get it now. import.meta.resolve is now supported in NodeJS and so this code is running successfully for the first time ever lol 🤦

It's obviously breaking in some way.

So this weekend let me make an issue for tracking this bug with 18.19.0 and if it's a quick fix then i will submit a PR, otherwise I can submit a PR to just pin this project's CI to 18.15.0 in the meantime.

edit 1:

By changing this line to omit the file:// I got it down to one error for the test case i was working against

  1) Develop Greenwood With:
       Default Greenwood Configuration and Workspace
         Develop command specific HTML behaviors
           should return an import map shim <script> in the <head> of the document:
     AssertionError: expected undefined to equal '/node_modules/regenerator-runtime/runtime.js'
      at file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/develop.default/develop.default.spec.js:488:37
      at Array.forEach (<anonymous>)
      at Context.<anonymous> (file:///Users/owenbuckley/Workspace/project-evergreen/greenwood/packages/cli/test/cases/develop.default/develop.default.spec.js:487:40)
      at process.processImmediate (node:internal/timers:476:21)

That could hint to a package exports / node module resolution change on NodeJS side, so might require a bit more investigation, so let me tinker with this some more and we'll get some way forward on this ASAP. (perhaps time to escalate this issue - https://github.com/ProjectEvergreen/greenwood/issues/684 . 🤔 )

OK, #1190 has been merged 👍

ProjectEvergreen / greenwood