Data requests with `X-Nextjs-Data: 1` header when middleware is present return 200 on 404

[huumn]

Link to the code that reproduces this issue

https://github.com/huumn/repro-nextjs-data-404-resp-200

To Reproduce

1. start application (it's the repro template with the example middleware - any middleware will trigger the bug)
2. open browser console
3. run the following

   const headers = new Headers()
   headers.append('X-Nextjs-data', 1)
   fetch('http://localhost:3000/_next/data/this_doesnt_exist.json', { headers }).then(console.log)

4. note the response code and empty object body

This is recreating fetching serverside props on client navigation.

Any /_next/data/ path exhibits this behavior but this is a practical problem as these paths change on deployment, ie /_next/data/:deployment_id/:path*. If the middleware.js file is removed, the application 404s as expected.

Current vs. Expected behavior

Current: returns 200 and an empty json object in body so client thinks route is good and the data is just empty Expected: returns 404 so client can refresh routes

Verify canary release

• [X] I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000
Binaries:
  Node: 18.17.0
  npm: 9.6.7
  Yarn: 1.22.10
  pnpm: N/A
Relevant Packages:
  next: 13.5.5-canary.15
  eslint-config-next: N/A
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.1.3
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

Data fetching (gS(S)P, getInitialProps)

Additional context

Anytime we deploy in prod, our customers using pwas or those actively online get served stale data when navigating until they hard close or refresh respectively.

I believe the 404 is getting rewritten to 200 here: https://github.com/vercel/next.js/blob/c6fe20a31ff4f62ee75c08cd0eae9ceaba7ec50d/packages/next/src/server/lib/router-server.ts#L205-L215

[huumn]

If anyone runs into this problem and has a service worker, we are planning to workaround it with a workbox plugin:

    fetchDidSucceed: async ({ request, response, event, state }) => {
      if (
        response.ok &&
        request.headers.get('x-nextjs-data') &&
        response.headers.get('x-nextjs-matched-path') &&
        response.headers.get('content-type') === 'application/json' &&
        response.headers.get('content-length') === '2' &&
        response.status === 200) {
        console.log('service worker detected a successful yet empty nextjs SSR data response')
        console.log('nextjs has a bug where it returns a 200 with empty data when it should return a 404')
        console.log('see https://github.com/vercel/next.js/issues/56852')
        console.log('HACK ... intercepting response and returning 404')

        const headers = new Headers(response.headers)
        headers.delete('x-nextjs-matched-path')
        headers.delete('content-type')
        headers.delete('content-length')
        return new Response(null, {
          status: 404,
          statusText: 'Not Found',
          headers,
          ok: false
        })
      }
      return response
    }
  }

[ydubinskyi]

Hi, I have the same problem: get 200 status with empty response after every deployment - even when I specifically ignore _next/data requests in middleware matcher.

[cteer-everquote]

Also seeing this issue

[richg0ld]

In version 13.5.6, the same issue was discovered, so we downgraded to version 13.4.10 to resolve it. However, we are unsure how many recent versions this problem persists in.. 😢