Incorrect monorepo package resolution? @glimmer/syntax -> @glimmer/util, but in ember-source, except not found

NullVoxPopuli commented 3 years ago

ref: https://github.com/glimmerjs/glimmer.js/issues/345

Reason why I'm thinking this might be a dependency resolution issue, In this PR where renovate upgrades my app from 3.26.x to 3.27.x I get the error

Build Error (PackagerRunner) in ../../../../home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js

Module not found: Error: Can't resolve 'ember-source/@glimmer/util.js' in '/home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js'

Log

``` ================================================================================= ENV Summary: TIME: Thu May 20 2021 07:39:40 GMT-0400 (Eastern Daylight Time) TITLE: ember ARGV: - /home/me/.volta/tools/image/node/14.17.0/bin/node - /home/me/Development/NullVoxPopuli/limber/frontend/node_modules/.bin/ember - serve EXEC_PATH: /home/me/.volta/tools/image/node/14.17.0/bin/node TMPDIR: /tmp SHELL: /bin/bash PATH: - /tmp/yarn--1621510740646-0.05927598237875187 - /home/me/Development/NullVoxPopuli/limber/frontend/node_modules/.bin - /home/me/.config/yarn/link/node_modules/.bin - /home/me/Development/NullVoxPopuli/limber/node_modules/.bin - /home/me/.volta/tools/image/node/14.17.0/libexec/lib/node_modules/npm/bin/node-gyp-bin - /home/me/.volta/tools/image/node/14.17.0/lib/node_modules/npm/bin/node-gyp-bin - /home/me/.volta/tools/image/node/14.17.0/bin/node_modules/npm/bin/node-gyp-bin - /tmp/yarn--1621510740233-0.7965985169595542 - /home/me/Development/NullVoxPopuli/limber/node_modules/.bin - /home/me/.config/yarn/link/node_modules/.bin - /home/me/Development/NullVoxPopuli/limber/node_modules/.bin - /home/me/.yarn/bin - /home/me/.volta/tools/image/node/14.17.0/libexec/lib/node_modules/npm/bin/node-gyp-bin - /home/me/.volta/tools/image/node/14.17.0/lib/node_modules/npm/bin/node-gyp-bin - /home/me/.volta/tools/image/node/14.17.0/bin/node_modules/npm/bin/node-gyp-bin - /home/me/.volta/tools/image/npm/7.11.1/bin - /home/me/.volta/tools/image/yarn/1.22.10/bin - /home/me/.volta/tools/image/node/14.17.0/bin - /home/me/.volta/bin - /home/me/.pythons/Python-3.6.3/bin - /home/me/.cargo/bin - /home/me/Applications - /home/me/apps/phantomjs/bin - /home/me/scripts/system-utils - /home/me/scripts/git - /home/me/scripts/rails - /home/me/scripts - /home/me/.volta/bin - /usr/local/sbin - /usr/local/bin - /usr/sbin - /usr/bin - /sbin - /bin - /usr/games - /usr/local/games - /snap/bin - /home/me/.dotnet/tools - /home/me/.dotnet/tools - /home/me/.fzf/bin PLATFORM: linux x64 FREEMEM: 277475328 TOTALMEM: 17388204032 UPTIME: 421813.43 LOADAVG: 2.24,0.84,0.35 CPUS: - Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz - 3192 - Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz - 3192 - Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz - 3192 - Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz - 3192 - Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz - 3192 - Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz - 3192 ENDIANNESS: LE VERSIONS: - ares: 1.17.1 - brotli: 1.0.9 - cldr: 38.1 - icu: 68.2 - llhttp: 2.1.3 - modules: 83 - napi: 8 - nghttp2: 1.42.0 - node: 14.17.0 - openssl: 1.1.1k - tz: 2020d - unicode: 13.0 - uv: 1.41.0 - v8: 8.4.371.23-node.63 - zlib: 1.2.11 ERROR Summary: - broccoliBuilderErrorStack: ModuleNotFoundError: Module not found: Error: Can't resolve 'ember-source/@glimmer/util.js' in '/home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib' at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/Compilation.js:1762:28 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:730:13 at eval (eval at create (/home/me/Development/NullVoxPopuli/limber/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), :10:1) at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:273:22 at eval (eval at create (/home/me/Development/NullVoxPopuli/limber/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), :9:1) at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:402:22 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:117:11 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:646:24 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:800:8 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:920:5 - code: [undefined] - codeFrame: Module not found: Error: Can't resolve 'ember-source/@glimmer/util.js' in '/home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js' - errorMessage: ../../../../home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js: Module not found: Error: Can't resolve 'ember-source/@glimmer/util.js' in '/home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js' at PackagerRunner (@embroider/webpack) - errorType: Build Error - location: - column: [undefined] - file: ../../../../home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js - line: [null] - treeDir: [undefined] - message: ../../../../home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js: Module not found: Error: Can't resolve 'ember-source/@glimmer/util.js' in '/home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js' at PackagerRunner (@embroider/webpack) - name: Error - nodeAnnotation: @embroider/webpack - nodeName: PackagerRunner - originalErrorMessage: Module not found: Error: Can't resolve 'ember-source/@glimmer/util.js' in '/home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js' - stack: ModuleNotFoundError: Module not found: Error: Can't resolve 'ember-source/@glimmer/util.js' in '/home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib' at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/Compilation.js:1762:28 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:730:13 at eval (eval at create (/home/me/Development/NullVoxPopuli/limber/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), :10:1) at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:273:22 at eval (eval at create (/home/me/Development/NullVoxPopuli/limber/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), :9:1) at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:402:22 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:117:11 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:646:24 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:800:8 at /home/me/Development/NullVoxPopuli/limber/node_modules/webpack/lib/NormalModuleFactory.js:920:5 ================================================================================= ```

And here is where I think it might be a resolution problem, ember-source is in my monorepo-root node_modules (with @glimmer/util.js):

❯ ls node_modules/ember-source/dist/dependencies/@glimmer/
destroyable.js  encoder.js  env.js  global-context.js  low-level.js  manager.js  node.js  opcode-compiler.js  owner.js  program.js  reference.js  runtime.js  util.js  validator.js  vm.js  wire-format.js

the same path does not exist in my app's local node_modules

ls frontend/node_modules/ember-source/
ls: cannot access 'frontend/node_modules/ember-source/': No such file or directory

in my app's package.json:

"ember-source": "3.27.1", (and higher, 3.28 beta, 4 alpha)
"ember-cli-babel": "^7.26.6",
"@glimmer/syntax": "^0.79.1",

ef4 commented 3 years ago

The problem here is probably that we are looking at this file at all:

../../../../home/me/Development/NullVoxPopuli/limber/frontend/node_modules/@glimmer/syntax/dist/modules/es2017/lib/parser.js

In a normal ember app, you don't have a real @glimmer/syntax package because that is packaged inside ember-source instead.

Did you add @glimmer/syntax to use it manually for some purpose? That would mean you have two of them, and the compatibility adjustments we need to make on the copy inside ember-source would break the unbundled copy.

NullVoxPopuli commented 3 years ago

Yup, I added it for:

import { getTemplateLocals } from '@glimmer/syntax';

used for in-browser compliing strict mode stuff on https://limber.glimdown.com

how does one adjust builtin compat rules?

ef4 commented 3 years ago

It's hard to fix your case without breaking the built-in case.

One workaround you can try, since you have a monorepo anyway, is to import from @glimmer/syntax in a new package that is not an ember addon. By doing it in code that's "not ember code", you avoid some of these compatibility features. That new package can then reexport the things you need and the app can import them from the new package.

Another option is to find what you need from inside ember-source/dist/template-compiler.js. That is what embroider itself does to get access to @glimmer/syntax, and that is how mho gets access to the template compiler from within a service worker.

NullVoxPopuli commented 3 years ago

awesome, thanks! I think I'll try the new package route, because I've been wanting to make the gjs compiler in here its own library anyway and publish that. I'll report back soon™!

NullVoxPopuli commented 3 years ago

This is related, but still interesting, testing out just this small change

-import { getTemplateLocals } from '@glimmer/syntax';
-import { precompileTemplate } from '@ember/template-compilation';
+import { getTemplateLocals, precompileTemplate } from '@ember/template-compilation';

(and removing @glimmer/syntax from package.json)

results in the build succeeding, but, with a runtime error:

``` Uncaught Error: Could not find module `@ember/-internals/utils` imported from `(require)` missingModule loader.js:247 findModule loader.js:258 requireModule loader.js:24 object.js:16 js chunk.bd06e5b59763daf483e5.js:5928 __webpack_require__ chunk.909881dca69b23cf60bd.js:31 decorator.js:9 js chunk.bd06e5b59763daf483e5.js:5917 __webpack_require__ chunk.909881dca69b23cf60bd.js:31 index.js:11 js chunk.bd06e5b59763daf483e5.js:5939 __webpack_require__ chunk.909881dca69b23cf60bd.js:31 Ember js chunk.bd06e5b59763daf483e5.js:5862 __webpack_require__ chunk.909881dca69b23cf60bd.js:31 Ember js chunk.bd06e5b59763daf483e5.js:5873 __webpack_require__ chunk.909881dca69b23cf60bd.js:31 Ember js chunk.bd06e5b59763daf483e5.js:5895 __webpack_require__ chunk.909881dca69b23cf60bd.js:31 Ember js chunk.79c93d843abd4d8f4997.js:619 __webpack_require__ chunk.909881dca69b23cf60bd.js:31 limber.js:25 ```

which comes from tracked-built-ins, so I'll follow up there

(probably more Ember global issues with 3.27 stuff)

NullVoxPopuli commented 3 years ago

🤔 I'm still having issues with module resolution it seems, even ember stuff:

for example, in tracked-built-ins, I'm getting an error that this utils module doesn't exist:

but, it's right here:

when debugging require,

I would expect that the @ember/* modules would appear in there -- yet they don't. 🤔 Gonna try an isolated repro

NullVoxPopuli commented 3 years ago

Just connecting the issues I'm finding together: https://github.com/pzuraq/tracked-built-ins/issues/96 (unexpected new issue 🤷 )

NullVoxPopuli commented 3 years ago

alright, with some git references in my package.json, and removing @glimmer/syntax and importing getTemplateLocals from the template compiler, I get this error:

Error: Cannot call `compile` with only the template compiler loaded. Please load `ember.debug.js` or `ember.prod.js` prior to calling `compile`.

which... I assume means that the template compiler just isn't compatible with ember 3.27 + embroider?

ef4 commented 3 years ago

I would expect that the @ember/* modules would appear in there -- yet they don't.

You really don't want them in there if you want true modules and tree shaking. We only put things in there that absolutely must be for compatibility. Under embroider ES module imports don't go through the runtime loader.

NullVoxPopuli commented 3 years ago

bug for the template compiler issue: https://github.com/emberjs/ember.js/issues/19568

NullVoxPopuli commented 3 years ago

idk if I've created too many x/y problems by doing this, but, I think we can close this issue, as the recommendation is to either:

use the builtin export
or pre-build the thing using @glimmer/syntax

embroider-build / embroider

Incorrect monorepo package resolution? @glimmer/syntax -> @glimmer/util, but in ember-source, except not found #817