codegen with swc on cross platform dev + docker not working

[mrtom]

I'm getting a load of errors of the form:

The 'this' context of type 'typeof Foo' is not assignable to method's 'this' of type 'new (viewer: Viewer<Ent<any> | null, ID | null>, data: Data) => FooBase'.
...
...
Property '__setRawDBData' is missing in type 'FooBase' but required in type 'Ent<Viewer<Ent<any> | null, ID | null>>'

I was using 0.1.0-alpha30, attempting to update to 0.1.0-alpha147 but that doesn't appear to have made a difference.

Unsure how best to diagnose. I can see the property exists in the latest version of the ent code (here).

Should I just remove all the generated files and re-gen? Is that safe?

Thanks :)

[mrtom]

(I think I can just update the base classes myself easily enough to add in this one change, but that feels sketchy - I'd expect the framework to re-gen generated files for me)

[lolopinto]

I believe it's because the image needs to change too. can you try with the latest image 0.1.0-alpha.48-nodejs-18-dev

update your develop.Dockerfile:

FROM ghcr.io/lolopinto/ent:0.1.0-alpha.48-nodejs-18-dev

[lolopinto]

I just updated ent-starter images too

[mrtom]

Thanks for the quick response.

Just so I'm clear on a couple of things:

1. Has the __setRawDBData property been added or removed? Should I end up with it included if things go to plan?
2. Should I expect the implementation of generated files to be overwritten as the ent framework evolves?

To your actual suggestion. I have tried upgrading as you suggested, but am running into problems. Here's what I've done:

1. Upgraded by ent dependency: yarn add @snowtop/ent@0.1.0-alpha147
2. Pulled the latest docker image: docker pull ghcr.io/lolopinto/ent:0.1.0-alpha.48-nodejs-18-dev
3. Updated develop.Dockerfile: Replaced image to FROM ghcr.io/lolopinto/ent:0.1.0-alpha.48-nodejs-18-dev
4. Updated the Dockerfile (although I'm not using that atm: FROM ghcr.io/lolopinto/ent:0.1.0-alpha.48-nodejs-18-slim
5. Rebuilt the image: yarn run rebuild-image

Now, if I run eg yarn codegen I'm was getting an error about not being able to find swc/core. So I added that (yarn add -D @swc/core), and now I'm getting an error about missing bindings:

~/Code/backend (upgrade2*) » yarn codegen                                                                                                                                                                                                                        tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
Error: Bindings not found
    at Compiler.transformSync (/app/node_modules/@swc/core/index.js:250:15)
    at transformSync (/app/node_modules/@swc/core/index.js:348:21)
    at transformSync (/app/node_modules/@swc-node/core/index.ts:72:26)
    at compile (/app/node_modules/@swc-node/register/register.ts:102:40)
    at exts (/app/node_modules/@swc-node/register/register.ts:113:38)
    at Module._compile (/app/node_modules/pirates/lib/index.js:113:29)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Object.newLoader [as .ts] (/app/node_modules/pirates/lib/index.js:121:7)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Function.Module._load (node:internal/modules/cjs/loader:958:12)
Error:
  exit status 1
Usage:
  tsent [command]

Available Commands:

[mrtom]

Oh, but that might be because I've not updated all the @gqlField bindings everywhere, which I think may have changed during the upgrade. BRB.

[mrtom]

Are the docs here up to date?

i.e. should it be something like:

@gqlQuery({
    name: "viewer",
    type: ViewerType,
  })
export default class ViewerResolver {
  @gqlQuery({ name: "viewer", type: ViewerType })
  viewer(@gqlContextType() context: RequestContext): ViewerType {
    return new ViewerType(context.getViewer());
  }
}

(as per the docs), or

@gqlQuery({
    class: "ViewerType",
    name: "viewer",
    type: ViewerType,
    args: [
      gqlContextType(),
    ],
  })
export default class ViewerResolver {
  @gqlQuery({ name: "viewer", type: ViewerType })
  viewer(context: RequestContext): ViewerType {
    return new ViewerType(context.getViewer());
  }
}

Am assuming the latter, as per the simple example.

Will update my code assuming that's the case.

[lolopinto]

Are the docs here up to date?

no :( there's https://github.com/lolopinto/ent/issues/1406 to update them which I haven't done. will get on that soon.

it's the latter because I changed it to use new decorators in TS 5.0 since I kept running into issues with that and swc for faster codegen

https://github.com/lolopinto/ent/pull/1405

[lolopinto]

Has the __setRawDBData property been added or removed? Should I end up with it included if things go to plan?

yes, you should end up with them

[lolopinto]

Should I expect the implementation of generated files to be overwritten as the ent framework evolves?

yes why files are broken up into generated/* and user editable files.

how can I make that clearer?

[lolopinto]

not sure what this missing bindings issue is? if you're still seeing them, can you add DISABLE_SWC=true in your docker-compose.dev.yml?

e.g. around https://github.com/lolopinto/ent/blob/main/examples/simple/docker-compose.dev.yml#L16

[lolopinto]

thanks for going through again. I'm about to release v0.1 and planned to do a documentation refresh around all the issues tagged with documentation

[mrtom]

how can I make that clearer?

I'm not sure you need to. My assumption was that this would be the case, but having run into issues I was doubting it.

What I don't quite understand is that I started this project a couple of months ago using ghcr.io/lolopinto/ent:0.1.0-alpha.30-nodejs-17-dev, and this problem happened immediately.

So I don't think this is a bodged upgrade issue. Although it's possible I just happened to pick a version of the framework where this transition was in progress?

[mrtom]

OK, adding the DISABLE_SWC flag has thrown up a new error when running yarn codegen. Looking into it now, but pasting here too for visibility:

~/Code/LimeAndSoda/roomtool-server (upgrade2*) » yarn codegen                                                                                                                                                                                                                    1 ↵ tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
checking for custom graphql definitions...
Error: Error: Cannot read properties of undefined (reading 'toString') loading /app/src/graphql/resolvers/organization.ts
    at /app/node_modules/@snowtop/ent/scripts/custom_graphql.js:286:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async requireFiles (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:273:5)
    at async captureCustom (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:238:9)
    at async Promise.all (index 1)
    at async main (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:371:42)
Error:
  error generating custom graphql: exit status 1
Usage:
  tsent [command]

FWIW I tried twice and had a different error the first time:

~/Code/LimeAndSoda/roomtool-server (upgrade2*) » yarn codegen                                                                                                                                                                                                                  130 ↵ tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
checking for custom graphql definitions...
Error: Error: Cannot read properties of undefined (reading 'kind') loading /app/src/graphql/resolvers/viewer.ts
    at /app/node_modules/@snowtop/ent/scripts/custom_graphql.js:286:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async requireFiles (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:273:5)
    at async captureCustom (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:238:9)
    at async Promise.all (index 1)
    at async main (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:371:42)
Error:
  error generating custom graphql: exit status 1
Usage:
  tsent [command]

[mrtom]

Should also say that fixing up the gql field changes has fixed running the server with yarn start

[mrtom]

I'm afraid I've got to bounce for the night - am on holiday and should probably spend some time with my wife :)

If useful, I've added you to the project on GH to have a look around - https://github.com/LimeAndSoda/roomtool-server/tree/upgrade2 is the branch that I'm running the upgrade in.

[lolopinto]

I'm afraid I've got to bounce for the night - am on holiday and should probably spend some time with my wife :)

If useful, I've added you to the project on GH to have a look around - https://github.com/LimeAndSoda/roomtool-server/tree/upgrade2 is the branch that I'm running the upgrade in.

cool thanks. will take a look later!

[lolopinto]

changes I made:

• changed tsconfig.json to comment out these two:

  // "experimentalDecorators": true,
  // "emitDecoratorMetadata": true,

^ related to documentation changes with 5.0

in organization.ts:

changed from type: 'string' to

        type: "String",

I think those were the 2 big ones. everything else was small stuff.

had to create local db, had to change docker-compose.dev.yml to work for ola instead of tom and usual yarn codegen, yarn rebuild-image etc

[mrtom]

Oh amazing, thank you!

Confirmed that's got both codegen and running the app working using the latest image.

Although I'm still having issues if I don't disable SWC. I'll keep that disabled for now, but let me know if you'd like any help/testing/whatever later with that. Thanks again!

[lolopinto]

can you try with this image

FROM ghcr.io/lolopinto/ent:0.1.0-alpha.49-test1-nodejs-18-dev

i made a small tweak and wanna see if it works? it works for me but i had done some debugging and not 100% sure if i somehow changed something independently

[mrtom]

Steps:

• Updated developer.Dockerfile, as above
• Removed - DISABLE_SWC=true from docker-compose.dev.yaml
• Ran docker pull ghcr.io/lolopinto/ent:0.1.0-alpha.49-test1-nodejs-18-dev from terminal
• Ran yarn rebuild-image
• Ran yarn codegen

Same issue as before I'm afraid:

~/Code/LimeAndSoda/roomtool-server (main*) » yarn codegen                                                                                                                                                                                                                            tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
Error: Bindings not found
    at Compiler.transformSync (/app/node_modules/@swc/core/index.js:250:15)
    at transformSync (/app/node_modules/@swc/core/index.js:348:21)
    at transformSync (/app/node_modules/@swc-node/core/index.ts:72:26)
    at compile (/app/node_modules/@swc-node/register/register.ts:102:40)
    at exts (/app/node_modules/@swc-node/register/register.ts:113:38)
    at Module._compile (/app/node_modules/pirates/lib/index.js:113:29)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Object.newLoader [as .ts] (/app/node_modules/pirates/lib/index.js:121:7)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Function.Module._load (node:internal/modules/cjs/loader:958:12)
Error:
  exit status 1
Usage:
  tsent [command]

Available Commands:
  alembic             alembic command
  codegen             runs the codegen (and db schema) migration
  completion          Generate the autocompletion script for the specified shell
  delete_schema       deletes the given schema
  detect_dangling     detects any dangling schema files
  downgrade           downgrade db
  fix-edges           fix edges db
  generate            generate a schema or other object
  help                Help about any command
  migrate_v0.1        migrate v0.1
  print_custom_schema prints the parsed custom graphql schema. only exists for debugging purposes and not guaranteed to exist forever
  print_schema        prints the parsed schema. only exists for debugging purposes and not guaranteed to exist forever
  run_script          run script
  squash              squash last N revs of the db into one
  upgrade             upgrade db

Flags:
      --debug         debug mode. add debug information to codegen e.g. how long commands take
      --debug_files   debug files mode. log files written and not written

Use "tsent [command] --help" for more information about a command.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Code/LimeAndSoda/roomtool-server (main*) »

[mrtom]

And just for completeness, if I re-add the flag to disable SWC in docker-compose.dev.yaml:

~/Code/LimeAndSoda/roomtool-server (main*) » yarn codegen                                                                                                                                                                                                                        1 ↵ tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
checking for custom graphql definitions...
generating ent code...
generating graphql code...
✨  Done in 13.58s.
~/Code/LimeAndSoda/roomtool-server (main*) »

[lolopinto]

ok, thanks! :(

[lolopinto]

I've been playing with it for a while and I can't 100% reliably solve it.

I think it's something to do with yarn

[mrtom]

Which package manager are you using yourself?

If not yarn, would it help for me to repro the absence of an error with the other package manager?

[lolopinto]

I'm using npm. I'll figure it out with yarn. Was just giving an update so it wouldn't look like I forgot about it.

[mrtom]

Thanks!

[lolopinto]

ok, i played with it even more and i believe i narrowed it down

when i had been using this for development, i had been building on an arm64 mac and building things locally without using docker

in production, doing it on an amd64 linux box and the docker image mirrors that

the issue that you were running into was the platform mismatch. there's ways around it but doesn't seem like it's worth doing that by default so i'm changing the behavior to not use swc and now to enable SWC need to use ENABLE_SWC=true env variable

[lolopinto]

i assume you're on a mac?

are you on arm64 or amd64?

if arm64, can you try this image https://github.com/lolopinto/ent/pkgs/container/ent/107819773?tag=0.1.0-alpha.50-test7-nodejs-18-dev

amd64 https://github.com/lolopinto/ent/pkgs/container/ent/107822985?tag=0.1.0-alpha.50-test8-nodejs-18-dev

different images because i was testing and faster to create single platform images

with @snowtop/ent@0.1.0-alpha160-test4

things work for me with these

[mrtom]

OK, so this time:

• Updated develop.Dockerfile to contain: FROM ghcr.io/lolopinto/ent:0.1.0-alpha.50-test7-nodejs-18-dev
• Removed - DISABLE_SWC=true from docker-compose.dev.yaml
• Added ENABLE_SWC=true to docker-compose.dev.yaml.
• Ran docker pull ghcr.io/lolopinto/ent:0.1.0-alpha.50-test7-nodejs-18-dev in the terminal
• Updated package.json to include "@snowtop/ent": "0.1.0-alpha160-test4", as a dependency
• Ran yarn install
• Ran yarn rebuild-image
• Ran yarn codegen

➜  roomtool-server git:(main) ✗ yarn codegen
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
node:internal/modules/cjs/loader:1080
  throw err;
  ^

Error: Cannot find module '@swc-node/register'
Require stack:
- internal/preload
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at internalRequire (node:internal/modules/cjs/loader:174:19)
    at Module._preloadModules (node:internal/modules/cjs/loader:1433:5)
    at loadPreloadModules (node:internal/process/pre_execution:606:5)
    at setupUserModules (node:internal/process/pre_execution:118:3)
    at prepareExecution (node:internal/process/pre_execution:109:5)
    at prepareMainThreadExecution (node:internal/process/pre_execution:37:3)
    at node:internal/main/run_main_module:10:1 {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ 'internal/preload' ]
}

Node.js v18.16.1
Error:
  exit status 1
Usage:
  tsent [command]

Available Commands:
  alembic             alembic command
  codegen             runs the codegen (and db schema) migration
  completion          Generate the autocompletion script for the specified shell
  delete_schema       deletes the given schema
  detect_dangling     detects any dangling schema files
  downgrade           downgrade db
  fix-edges           fix edges db
  generate            generate a schema or other object
  help                Help about any command
  migrate_v0.1        migrate v0.1
  print_custom_schema prints the parsed custom graphql schema. only exists for debugging purposes and not guaranteed to exist forever
  print_schema        prints the parsed schema. only exists for debugging purposes and not guaranteed to exist forever
  run_script          run script
  squash              squash last N revs of the db into one
  upgrade             upgrade db

Flags:
      --debug         debug mode. add debug information to codegen e.g. how long commands take
      --debug_files   debug files mode. log files written and not written

Use "tsent [command] --help" for more information about a command.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[mrtom]

Tried adding @swc-node/register as a dependency:

• yarn add @swc-node/register
• yarn codegen

➜  roomtool-server git:(main) ✗ yarn add @swc-node/register
yarn add v1.22.19
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
warning Pattern ["strip-ansi@^6.0.1"] is trying to unpack in the same destination "/Users/tom/Library/Caches/Yarn/v6/npm-strip-ansi-cjs-6.0.1-9e26c63d30f53443e9489495b2105d37b67a85d9-integrity/node_modules/strip-ansi-cjs" as pattern ["strip-ansi-cjs@npm:strip-ansi@^6.0.1"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["string-width@^4.1.0"] is trying to unpack in the same destination "/Users/tom/Library/Caches/Yarn/v6/npm-string-width-cjs-4.2.3-269c7117d27b05ad2e536830a8ec895ef9c6d010-integrity/node_modules/string-width-cjs" as pattern ["string-width-cjs@npm:string-width@^4.2.0"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["strip-ansi@^6.0.0"] is trying to unpack in the same destination "/Users/tom/Library/Caches/Yarn/v6/npm-strip-ansi-cjs-6.0.1-9e26c63d30f53443e9489495b2105d37b67a85d9-integrity/node_modules/strip-ansi-cjs" as pattern ["strip-ansi-cjs@npm:strip-ansi@^6.0.1"]. This could result in non-deterministic behavior, skipping.
[3/4] 🔗  Linking dependencies...
warning " > @snowtop/ent-phonenumber@0.1.0-alpha2" has incorrect peer dependency "@snowtop/ent@>= 0.1.0-alpha95".
warning " > next-auth@4.22.0" has unmet peer dependency "next@^12.2.5 || ^13".
warning " > next-auth@4.22.0" has unmet peer dependency "react@^17.0.2 || ^18".
warning " > next-auth@4.22.0" has unmet peer dependency "react-dom@^17.0.2 || ^18".
warning " > @swc-node/register@1.6.6" has unmet peer dependency "typescript@>= 4.3".
[4/4] 🔨  Building fresh packages...
success Saved lockfile.
success Saved 8 new dependencies.
info Direct dependencies
└─ @swc-node/register@1.6.6
info All dependencies
├─ @swc-node/core@1.10.4
├─ @swc-node/register@1.6.6
├─ @swc-node/sourcemap-support@0.3.0
├─ buffer-from@1.1.2
├─ colorette@2.0.20
├─ pirates@4.0.6
├─ source-map-support@0.5.21
└─ source-map@0.6.1
✨  Done in 2.81s.
➜  roomtool-server git:(main) ✗ yarn codegen
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
Error: Bindings not found
    at Compiler.transformSync (/app/node_modules/@swc/core/index.js:250:15)
    at transformSync (/app/node_modules/@swc/core/index.js:348:21)
    at transformSync (/app/node_modules/@swc-node/core/index.ts:73:26)
    at compile (/app/node_modules/@swc-node/register/register.ts:102:40)
    at exts (/app/node_modules/@swc-node/register/register.ts:113:38)
    at Module._compile (/app/node_modules/pirates/lib/index.js:113:29)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Object.newLoader [as .ts] (/app/node_modules/pirates/lib/index.js:121:7)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Function.Module._load (node:internal/modules/cjs/loader:960:12)
Error:
  exit status 1
Usage:
  tsent [command]

Available Commands:
  alembic             alembic command
  codegen             runs the codegen (and db schema) migration
  completion          Generate the autocompletion script for the specified shell
  delete_schema       deletes the given schema
  detect_dangling     detects any dangling schema files
  downgrade           downgrade db
  fix-edges           fix edges db
  generate            generate a schema or other object
  help                Help about any command
  migrate_v0.1        migrate v0.1
  print_custom_schema prints the parsed custom graphql schema. only exists for debugging purposes and not guaranteed to exist forever
  print_schema        prints the parsed schema. only exists for debugging purposes and not guaranteed to exist forever
  run_script          run script
  squash              squash last N revs of the db into one
  upgrade             upgrade db

Flags:
      --debug         debug mode. add debug information to codegen e.g. how long commands take
      --debug_files   debug files mode. log files written and not written

Use "tsent [command] --help" for more information about a command.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[mrtom]

And yes, this is an M2 so arm64.

[lolopinto]

the first one should have failed so that's not surprising since no dependency and ENABLE_SWC was added

the default is no extra envs and it should work because it's not doing SWC

the latter not working is maddening lol. i tested it multiple times the other day but yup, not working for me anymore.

I think I'm going to punt on this for now and come back to it later with the node_modules hidden from docker approach. my guess is we need to run yarn install in develop.Dockerfile separately on docker.

[lolopinto]

thanks for debugging with me

[mrtom]

You're welcome!

[mrtom]

the default is no extra envs and it should work because it's not doing SWC

Yeah, I didn't test this because AIUI this is a problem with SWC. Let me know if you do want me to test this, but assume not for now.