Cannot use local API with payload v3 on next.js custom server

[ikenox]

Link to reproduction

https://github.com/ikenox/payload-3.0-demo/pull/1/commits/aa8f8634267a808928d1ad2a1272b3a81b563012

Payload Version

beta.55

Node Version

20.13.1

Next.js Version

15.0.0-rc.0

Describe the Bug

I’m considering migrating from an existing payload v2 project to v3. Then I need to keep running the server based on express.js. So for now, based on the payload-3.0-demo repository, I tried to create a minimal example server with express.js + payload v3 (+ next.js custom server) . https://github.com/ikenox/payload-3.0-demo/pull/1/commits/aa8f8634267a808928d1ad2a1272b3a81b563012

But an error occurs when I access to admin panel http://localhost:3000/admin or custom endpoint http://localhost:3000/users:

Error: Invariant: AsyncLocalStorage accessed in runtime where it is not available

I confirmed that using only local API with ts-node works well. The error seems to cause by a combination of next.js custom server + payload local API + ts-node. And the error occurs when load payload config by importConfig function.

This might be a blocker for expressjs-based monolithic v2 project that tries to migrate to v3.

Reproduction Steps

I created minimal example server. https://github.com/ikenox/payload-3.0-demo/pull/1/commits/aa8f8634267a808928d1ad2a1272b3a81b563012

On this branch,

1. Start the server by pnpm dev:ts-node
2. access to http://localhost:3000/admin
3. The following error occurs

Error: Invariant: AsyncLocalStorage accessed in runtime where it is not available

Adapters and Plugins

No response

[SimYunSup]

This seems to be a bug caused by inserting the nextHandler too quickly. nextjs injects AsyncLocalStorage into globalThis when it's ready, and I think it's caused by using the nextHandler before nextjs is ready.

Similar to custom-server example in nextjs, replacing main.ts like below will fix it.

import next from 'next'
import express from 'express'
import { importConfig } from 'payload/node'

import { getPayload } from 'payload'

const expressApp = express()

async function main() {
  const nextApp = next({
    dev: process.env.NODE_ENV !== 'production',
  })
  const nextHandler = nextApp.getRequestHandler()

  const config = await importConfig('../payload.config.ts')
  const payload = await getPayload({ config })
  expressApp.get('/users', (req, res) => {
    payload
      .find({ collection: 'users' })
      .then(({ docs }) => res.json(docs))
      .catch(next)
  })

  nextApp.prepare().then(() => {
    console.log('Next.js started')
    expressApp.use((req, res) => nextHandler(req, res))
    expressApp.listen(3000, () => {
      console.log(globalThis.AsyncLocalStorage)
      console.log('listen on port 3000')
    })
  })
}

void main()

[ikenox]

@SimYunSup Thank you for the advice!

Just moving nextHandler after nextApp.prepare() gave the same error. https://github.com/ikenox/payload-3.0-demo/pull/1/commits/a482e0abb9906a50a8596f8322d74865f0447108

By loading payload config after nextApp.prepare(), I managed to show the admin panel. https://github.com/ikenox/payload-3.0-demo/pull/1/commits/bd47e01159bb7a8a2e1290ddeee690ee3f963d27 But I got different errors when visiting admin panel.

Invalid hook call. Hooks can only be called inside of the body of a function component. This could happen for one of the following reasons:
1. You might have mismatching versions of React and the renderer (such as React DOM)
2. You might be breaking the Rules of Hooks
3. You might have more than one copy of React in the same app

 ⨯ TypeError: Cannot read properties of null (reading 'useContext')
    at AsyncLocalStorage.run (node:async_hooks:346:14)
    at stringify (<anonymous>)
    at stringify (<anonymous>)

 ⨯ TypeError: Cannot read properties of null (reading 'useState')
    at AsyncLocalStorage.run (node:async_hooks:346:14)
    at stringify (<anonymous>)

And these errors are disappeared if I remove const payload = await getPayload({ config }) and its related code. It means that payload v3 with next.js custom server works well, but still cannot use payload local API with it.

[SimYunSup]

Oh, sorry. I uploaded previous version code. importConfig and getPayload code must be in then block in nextApp.prepare. Because importConfig call loader and loader calls next/navigation. So This will work on your machine too.

import next from 'next'
import express from 'express'
import { importConfig } from 'payload/node'

import { getPayload } from 'payload'

const expressApp = express()

async function main() {
  const nextApp = next({
    dev: process.env.NODE_ENV !== 'production',
  })
  const nextHandler = nextApp.getRequestHandler()

  nextApp.prepare().then(async () => {
    const config = await importConfig('../payload.config.ts')
    const payload = await getPayload({ config })
    expressApp.get('/users', async (req, res) => {
      await payload
        .find({ collection: 'users' })
        .then(({ docs }) => res.json(docs))
        .catch(next)
    })
    console.log('Next.js started')
    expressApp.use((req, res) => nextHandler(req, res))
    expressApp.listen(3000, () => {
      console.log('listen on port 3000')
    })
  })
}

void main()

[ikenox]

@SimYunSup Thanks!

importConfig and getPayload code must be in then block in nextApp.prepare

Yes, actually I can start the server by your code. https://github.com/ikenox/payload-3.0-demo/pull/1/commits/ea3bad8ab873af67bd34ba2d13e0e31998c6740c

But something still seems not to work well. As I mentioned in this my comment, some errors are occurred instead. For example, I got the following error when I visit http://localhost:3000/admin/collections/pages :

If I stop to use local API, this error is resolved. (https://github.com/ikenox/payload-3.0-demo/pull/1/commits/8e1da7406c7ea2d0acda0d2885ea482366174151)

[jmikrut]

Hey @ikenox — is this still happening if you update to the most recent version of Payload beta packages and Next.js canary?

One thing, we're working on compartmentalizing client-side JS from the Payload config which will probably resolve your new useState issue. I bet this is all related.

But we will keep this issue open regardless until we can come up with a custom server example that works with the Local API.

[ikenox]

Hi @jmikrut, thanks for looking into this issue!

is this still happening if you update to the most recent version of Payload beta packages and Next.js canary?

Yes, I upgraded payload and next.js but the problem that is mentioned in my this comment still remains. https://github.com/ikenox/payload-3.0-demo/pull/1/commits/b02acc3c6d94af5bb2fc8343162eb77cef82eba9

One thing, we're working on compartmentalizing client-side JS from the Payload config

It sounds great improvement. I too believe it resolves this issue.