Migration not possible after build without tsconfig.json

geisterfurz007 commented 1 month ago

Link to reproduction

https://github.com/geisterfurz007/payload-3-docker-migrate

Payload Version

3.0.0-beta.58

Node Version

20.14.0

Next.js Version

15.0.0-canary.56

Describe the Bug

Migrating the database after building the project currently requires a tsconfig.json to be present, otherwise the loader fails. This seems to be a regression from Payload 2 where having a compiled payload.config.js file was enough to run migrations.

The usecase is a standalone self-hosted deployment using a Docker image where migration of the database happens at an unknown point in time after building the image. The flow is:

Compile the image in the CI
When releasing a new version, pull the image
The entrypoint of the image runs the migrations against the database, then runs next start

Naturally, the image should contain as few files as possible to function (i.e. no TS source files, tsconfig.json, or other build-specific files)

The two approaches to achieve this I tried so far are:

pnpm run payload migrate as a command directly (hoping that the CLI would automagically pick up the config from the compiled .next folder)
Compile the payload.config.ts to payload.config.js and use a standalone script to import the config from there (in theory removing the need for a compilation step because the compiled file is already present)

Both fail with the same error pointing at

https://github.com/payloadcms/payload/blob/187813ef6342076eb4b6b1026ecc0ca0cfdaaf32/packages/payload/src/bin/loader/index.ts#L27-L28

TypeError [Error]: Cannot read properties of null (reading 'config')
    at file:///app/node_modules/.pnpm/payload@3.0.0-beta.58_@swc+core@1.6.5_@swc+types@0.1.9_graphql@16.9.0_typescript@5.5.2/node_modules/payload/dist/bin/loader/index.js:9:32
    at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:316:24)
    at async Hooks.register (node:internal/modules/esm/hooks:165:26)
    at async MessagePort.handleMessage (node:internal/modules/esm/worker:196:18)

with the loader seemingly requiring a tsconfig.json to run without error.

I would have expected at least approach 2 above to work (1 is probably very optimistic given the complex nature of the Next compilation output).

Reproduction Steps

Clone the repro repository, build the docker image and run it using the commands below:

git clone git@github.com:geisterfurz007/payload-3-docker payload-3-docker-gf007
cd payload-3-docker-gf007
docker build -t payload-3-docker-gf007
docker run --rm payload-3-docker-gf007

Adapters and Plugins

No response

yobottehg commented 1 month ago

I'm running in the same problem.

We have a working Next.js application in a docker container with Payload CMS but no way of interacting with the payload CLI. Our difference is that we use the standalone output instead of the normal one like in the reproduction link.

The node_modules inside the .next/standalone folder do only contain the Next.js runtime dependencies. For example the .bin directory is missing and therefore we can't call payload migrate.

Copying additional node_module files like this:

COPY --from=builder --chown=nextjs:nodejs /app/node_modules/.bin/payload ./node_modules/.bin/payload
COPY --from=builder --chown=nextjs:nodejs /app/node_modules/payload ./node_modules/payload

Results in :

(node:46) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)
/app/node_modules/.bin/payload:3
import { register } from 'node:module'
^^^^^^

So I'm currently stuck here without a path forward.

Are there any workarounds to get a production docker container running and usable with Payload v3?

jmikrut commented 3 weeks ago

Hey @geisterfurz007 — I left some comments in the above linked issue (we want to enable what you're trying to do in the near-ish future) but right now our CLI is set up to run using your TypeScript config.

It's possible that if you defined your config path as an environment variable PAYLOAD_CONFIG_PATH, we could skip the tsconfig location, and thus not crash the CLI. This might tide you over in the interim because you are indeed not using Next.js standalone mode like the other issue. We would be happy to make some adjustments in that file to accommodate that.

But, I think longer term it would be good to allow for JS configs to be loaded via the CLI (and pre-built JS migrations) therefore not needing the heavy TS transpilation dependencies in the production image like @swc/core, etc.

Check out that issue above for more information here, but because this one is duplicative, I'd like to close this one and bring the discussions to the link above. I should have responded to this one first because your issue is older but I didn't see it until just now.

Thanks for bringing this up - I'll keep working with the team to determine the best way forward here!

payloadcms / payload