trappar
commented
7 months ago I've been digging at this trying to replicate the issue without using a next build and have come up with nothing helpful, but in the process I found something very interesting...
Simply manually unarchiving the files from the turbo cache .tar.zst file yields an identical match to the original output files. This works in my reproduction repo and my big complicated monorepo project.
The command I'm using to unarchive the files is: tar --use-compress-program=unzstd -xvf xxx.tar.zst
This manually unarchived version also works perfectly. No server errors like when I try to run a server based on the files restored from turbo's cache.
So this tells me that the issue is probably one or both of:
- Turbo is doing something post-unarchive to the files.
- This bug is within whatever Turbo is using to unarchive the files.
Just quickly looking at the code I feel like topologically_restore_symlinks stands out as a potential source for this bug, so I tried augmenting the next build like so:
"build": "next build && cp -RL .next .next-deref && rm -rf .next && mv .next-deref .next"The idea being that by copying the files and following symlinks, the resulting build would be devoid of both hard links (due to copying the files) and symlinks (due to the -L option).
The results here at least partially confirm my hunch. Before making this change in my big monorepo project the diff showed both:
- Files present in the original version which aren't in the cache restored version
- Files present in the cached restored version which aren't in the original version
After making this change to copy the files and dereference links, all the files which were present in the original but not in the cached version were fixed and the diff now only shows loads of files which are only present in the cached version.
I was hoping that this would at least get things working again (and could maybe be a temporary workaround), however, doing the file copying seems to screw up the next build. Trying to run the server using the copied/dereferenced version using either the original files or the turbo cached version errors out with:
node:internal/modules/cjs/loader:1147
throw err;
^
Error: Cannot find module 'styled-jsx/package.json'
Require stack:
- /app/apps/[redacted]/node_modules/next/dist/server/require-hook.js
- /app/apps/[redacted]/node_modules/next/dist/server/next.js
- /app/apps/[redacted]/server.js
at Module._resolveFilename (node:internal/modules/cjs/loader:1144:15)
at resolve (node:internal/modules/helpers:187:19)
at Object.<anonymous> (/app/apps/[redacted]/node_modules/next/dist/server/require-hook.js:38:32)
at Module._compile (node:internal/modules/cjs/loader:1376:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
at Module.load (node:internal/modules/cjs/loader:1207:32)
at Module._load (node:internal/modules/cjs/loader:1023:12)
at Module.require (node:internal/modules/cjs/loader:1235:19)
at require (node:internal/modules/helpers:176:18)
at Object.<anonymous> (/app/apps/[redacted]/node_modules/next/dist/server/next.js:25:1) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'/app/apps/[redacted]/node_modules/next/dist/server/require-hook.js',
'/app/apps/[redacted]/node_modules/next/dist/server/next.js',
'/app/apps/[redacted]/server.js'
]
}So to summarize, this issue is almost certainly due to how Turbo unarchives its caches and handles the output files. I'm still trying to find the precise file/directory conditions which cause these issues.
arlyon
bmelonis
chris-olszewski
Verify canary release
Link to code that reproduces this issue
https://github.com/trappar/turbo-cache-missing-output-files
What package manager are you using / does the bug impact?
pnpm
What operating system are you using?
Mac
Which canary version will you have in your reproduction?
1.12.4
Describe the Bug
When building my app with turbo, I have it configured to cache the
.nextdirectory. I noticed that if I build, then delete the.nextdirectory, then recover it from my turbo cache - the resulting files differ from the original files I had after building.Expected Behavior
The files cached by turbo cache should precisely match the original output files.
To Reproduce
The reproduction repo I linked has a
reproduce-issue.shscript that you can run to see the issue. Since this is a minimal reproduction repo, the number of divergent files between the fresh and cached copies are minimal. In a large enterprise monorepo I work on, the number of files not being copied is massive and is causing our servers to error/crash when we try to run a build which was recalled from the turbo cache.Just for context, here's all that I'm doing in that script:
And this is producing the following list of file differences:
Additional context
The only context where I've seen this happen is a Next.js app using the "standalone" output mode. I suspect this is because the directory I'm trying to cache has long file paths and complex/internal links. It's possible that this could be distilled down even further into a reproduction that simply creates a directory structure like what Next.js creates, but this is probably the best repo since it's a fairly common application configuration.
I know this issue can be reproduced on Ubuntu and on Mac since I'm seeing these kinds of failures in CI, and I've reproduced it locally in that reproduction repo. I'm curious if this bug will happen on all OSes (windows?)
TURBO-2414