search

vercel / turborepo

Build system optimized for JavaScript and TypeScript, written in Rust
https://turbo.build/repo/docs
MIT License
26.21k stars 1.81k forks source link

[turborepo] some crashes when running turbo prune --docker #5202

Closed Klaitos closed 1 year ago

Klaitos commented 1 year ago

What version of Turborepo are you using?

1.10.1

What package manager are you using / does the bug impact?

npm

What operating system are you using?

Linux

Describe the Bug

Hello,

For almost ~2 weeks now, we are experiencing some random crashes when using turbo prune command on our mono repo in a Docker alpine 3.17 container on a ubuntu 22.04 machine with 4 cores.

It is not happening every time, but few times a week. Retrying the exactly same CI step does not reproduce the issue, the command finally succeed. We always use last version of turbo repo. (crash is present in 1.9 and 1.10 of turbo repo) It seems happening more or less right after the 14 or 15 or 16 package iteration but its a guess. We are using turbo repo remote cache (custom endpoint) but it is not used in this command as far as I know.

(package names have been anonymized)

#22 3.126  - Added @choose/package
#22 3.129  - Added @choose/package
#22 3.131  - Added @choose/package
#22 3.133  - Added @choose/package
#22 3.134  - Added @choose/package
#22 3.135  - Added @choose/package
#22 3.137  - Added @choose/package
#22 3.142  - Added @choose/package
#22 3.143  - Added @choose/package
#22 3.144  - Added @choose/package
#22 3.145  - Added @choose/package
#22 3.146  - Added @choose/package
#22 3.147  - Added @choose/package
#22 3.148  - Added @choose/package
#22 3.149  - Added @choose/package
#22 3.150  - Added @choose/package
#22 3.151  - Added @choose/package
#22 3.184 runtime: bad pointer in frame github.com/vercel/turbo/cli/internal/ffi.Unmarshal[...] at 0xc000017678: 0x1
#22 3.184 fatal error: invalid pointer found on stack

I have also observed this slightly different error

 runtime: bad pointer in frame github.com/vercel/turbo/cli/internal/ffi.RecursiveCopy at 0xc0002776f0: 0x1

FULL STACK TRACE

runtime stack:
#21 4.137 runtime.throw({0x368c4e?, 0x11ba380?})
#21 4.137   runtime/panic.go:992 +0x71 fp=0x7f210d5ac1d8 sp=0x7f210d5ac1a8 pc=0x918491
#21 4.137 runtime.adjustpointers(0x7f210d5ac5c8?, 0x12daa0?, 0x6ec462?, {0x83e710?, 0x11ba380?})
#21 4.138   runtime/stack.go:628 +0x1d0 fp=0x7f210d5ac240 sp=0x7f210d5ac1d8 pc=0x931150
#21 4.138 runtime.adjustframe(0x7f210d5ac5c8, 0x7f210d5ac6b8)
#21 4.138   runtime/stack.go:670 +0xc9 fp=0x7f210d5ac2d0 sp=0x7f210d5ac240 pc=0x931269
#21 4.138 runtime.gentraceback(0x64799789?, 0x7f56db4?, 0x64799772?, 0x1e5f2bd7?, 0x0, 0x0, 0x7fffffff, 0x3a05a8, 0x0?, 0x0)
#21 4.138   runtime/traceback.go:345 +0xc47 fp=0x7f210d5ac630 sp=0x7f210d5ac2d0 pc=0x93db87
#21 4.138 runtime.copystack(0xc000003ba0, 0x0?)
#21 4.138   runtime/stack.go:930 +0x2f5 fp=0x7f210d5ac7e8 sp=0x7f210d5ac630 pc=0x931a75
#21 4.138 runtime.shrinkstack(0xc000003ba0)
#21 4.138   runtime/stack.go:1212 +0x126 fp=0x7f210d5ac808 sp=0x7f210d5ac7e8 pc=0x932946
#21 4.138 runtime.newstack()
#21 4.138   runtime/stack.go:1060 +0x3aa fp=0x7f210d5ac9c0 sp=0x7f210d5ac808 pc=0x931f0a
#21 4.138 runtime.morestack()
#21 4.138   runtime/asm_amd64.s:547 +0x8b fp=0x7f210d5ac9c8 sp=0x7f210d5ac9c0 pc=0x948fab
#21 4.138 
#21 4.138 goroutine 6 [running]:
#21 4.138 github.com/vercel/turbo/cli/internal/ffi/proto.(*RecursiveCopyResponse).Reset(0xc0009aaa80?)
#21 4.138   github.com/vercel/turbo/cli/internal/ffi/proto/messages.pb.go:1626 +0x8a fp=0xc000a594f8 sp=0xc000a594f0 pc=0xddddca
#21 4.138 google.golang.org/protobuf/proto.Reset({0x45fea0, 0xc0009aaa80})
#21 4.138   google.golang.org/protobuf@v1.28.0/proto/reset.go:18 +0x3e fp=0xc000a59530 sp=0xc000a594f8 pc=0xd6525e
#21 4.138 google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x0, 0x0, 0x0, {0x463618, 0xc0001f4450}, 0x2710}, {0x1, 0x0, 0x0}, ...)
#21 4.138   google.golang.org/protobuf@v1.28.0/proto/decode.go:87 +0xc5 fp=0xc000a595e8 sp=0xc000a59530 pc=0xd5aa45
#21 4.138 google.golang.org/protobuf/proto.Unmarshal({0x1, 0x0, 0x0}, {0x45fea0?, 0xc0009aaa80?})
#21 4.138   google.golang.org/protobuf@v1.28.0/proto/decode.go:54 +0x65 fp=0xc000a59640 sp=0xc000a595e8 pc=0xd5a6e5
#21 4.138 github.com/vercel/turbo/cli/internal/ffi.Unmarshal[...]({0xae8d300?, 0x1?}, {0x45fea0?, 0xc0009aaa80?})
#21 4.138   github.com/vercel/turbo/cli/internal/ffi/ffi.go:32 +0x94 fp=0xc000a596b0 sp=0xc000a59640 pc=0xde14d4
#21 4.138 github.com/vercel/turbo/cli/internal/ffi.RecursiveCopy({0xc00046fe18, 0x13}, {0xc00046b960, 0x1c})
#21 4.138   github.com/vercel/turbo/cli/internal/ffi/ffi.go:285 +0x165 fp=0xc000a59720 sp=0xc000a596b0 pc=0xde1065
#21 4.138 github.com/vercel/turbo/cli/internal/fs.RecursiveCopy(...)
#21 4.138   github.com/vercel/turbo/cli/internal/fs/recursive_copy.go:10
#21 4.138 github.com/vercel/turbo/cli/internal/prune.(*prune).prune(0xc000165f60, 0xc000254c30, {0xc000298032, 0x5})
#21 4.138   github.com/vercel/turbo/cli/internal/prune/prune.go:160 +0x250a fp=0xc000a59f08 sp=0xc000a59720 pc=0xe5ae4a
#21 4.138 github.com/vercel/turbo/cli/internal/prune.ExecutePrune(0x0?, 0xc0001fbd40)
#21 4.138   github.com/vercel/turbo/cli/internal/prune/prune.go:45 +0xf9 fp=0xc000a59f78 sp=0xc000a59f08 pc=0xe585f9
#21 4.138 github.com/vercel/turbo/cli/internal/cmd.RunWithExecutionState.func1()
#21 4.138   github.com/vercel/turbo/cli/internal/cmd/root.go:67 +0x107 fp=0xc000a59fe0 sp=0xc000a59f78 pc=0xf38467
#21 4.138 runtime.goexit()
#21 4.138   runtime/asm_amd64.s:1571 +0x1 fp=0xc000a59fe8 sp=0xc000a59fe0 pc=0x94b081
#21 4.138 created by github.com/vercel/turbo/cli/internal/cmd.RunWithExecutionState
#21 4.138   github.com/vercel/turbo/cli/internal/cmd/root.go:64 +0x31a
#21 4.138 
#21 4.138 goroutine 1 [select]:
#21 4.138 runtime.gopark(0xc000267ec0?, 0x2?, 0x1?, 0x0?, 0xc000267e44?)
#21 4.138   runtime/proc.go:361 +0xd6 fp=0xc000068cb0 sp=0xc000068c90 pc=0x91af76
#21 4.138 runtime.selectgo(0xc000068ec0, 0xc000267e40, 0xc000267e40?, 0x0, 0x1e5?, 0x1)
#21 4.138   runtime/select.go:[328](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:332) +0x772 fp=0xc000068df0 sp=0xc000068cb0 pc=0x92a872
#21 4.138 github.com/vercel/turbo/cli/internal/cmd.RunWithExecutionState(0xc0001fbd40, {0x3528f8, 0x5})
#21 4.138   github.com/vercel/turbo/cli/internal/cmd/root.go:79 +0x36a fp=0xc000068ef8 sp=0xc000068df0 pc=0xf3814a
#21 4.138 main.main()
#21 4.138   github.com/vercel/turbo/cli/cmd/turbo/main.go:30 +0x1bb fp=0xc000068f80 sp=0xc000068ef8 pc=0xf38d9b
#21 4.138 runtime.main()
#21 4.138   runtime/proc.go:250 +0x212 fp=0xc000068fe0 sp=0xc000068f80 pc=0x91abb2
#21 4.138 runtime.goexit()
#21 4.138   runtime/asm_amd64.s:1571 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x94b081
#21 4.138 
#21 4.138 goroutine 2 [force gc (idle)]:
#21 4.138 runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
#21 4.138   runtime/proc.go:[361](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:365) +0xd6 fp=0xc00005afb0 sp=0xc00005af90 pc=0x91af76
#21 4.138 runtime.goparkunlock(...)
#21 4.138   runtime/proc.go:367
#21 4.138 runtime.forcegchelper()
#21 4.138   runtime/proc.go:301 +0xad fp=0xc00005afe0 sp=0xc00005afb0 pc=0x91ae0d
#21 4.138 runtime.goexit()
#21 4.138   runtime/asm_amd64.s:1571 +0x1 fp=0xc00005afe8 sp=0xc00005afe0 pc=0x94b081
#21 4.138 created by runtime.init.6
#21 4.138   runtime/proc.go:289 +0x25
#21 4.138 
#21 4.138 goroutine 3 [GC sweep wait]:
#21 4.138 runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
#21 4.138   runtime/proc.go:361 +0xd6 fp=0xc00005b790 sp=0xc00005b770 pc=0x91af76
#21 4.138 runtime.goparkunlock(...)
#21 4.138   runtime/proc.go:[367](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:371)
#21 4.138 runtime.bgsweep(0x0?)
#21 4.138   runtime/mgcsweep.go:297 +0xd7 fp=0xc00005b7c8 sp=0xc00005b790 pc=0x906f77
#21 4.138 runtime.gcenable.func1()
#21 4.138   runtime/mgc.go:177 +0x26 fp=0xc00005b7e0 sp=0xc00005b7c8 pc=0x8fcb26
#21 4.139 runtime.goexit()
#21 4.139   runtime/asm_amd64.s:1571 +0x1 fp=0xc00005b7e8 sp=0xc00005b7e0 pc=0x94b081
#21 4.139 created by runtime.gcenable
#21 4.139   runtime/mgc.go:177 +0x6b
#21 4.139 
#21 4.139 goroutine 4 [sleep]:
#21 4.139 runtime.gopark(0xc00008e000?, 0x2cf62fb0cc?, 0x0?, 0x0?, 0x3a05c8?)
#21 4.139   runtime/proc.go:361 +0xd6 fp=0xc00005bed0 sp=0xc00005beb0 pc=0x91af76
#21 4.139 runtime.goparkunlock(...)
#21 4.139   runtime/proc.go:367
#21 4.139 runtime.scavengeSleep(0x3b9aca00)
#21 4.139   runtime/mgcscavenge.go:248 +0x9d fp=0xc00005bf20 sp=0xc00005bed0 pc=0x904a5d
#21 4.139 runtime.bgscavenge(0x0?)
#21 4.139   runtime/mgcscavenge.go:[394](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:398) +0x1af fp=0xc00005bfc8 sp=0xc00005bf20 pc=0x904c8f
#21 4.139 runtime.gcenable.func2()
#21 4.139   runtime/mgc.go:178 +0x26 fp=0xc00005bfe0 sp=0xc00005bfc8 pc=0x8fcac6
#21 4.139 runtime.goexit()
#21 4.139   runtime/asm_amd64.s:1571 +0x1 fp=0xc00005bfe8 sp=0xc00005bfe0 pc=0x94b081
#21 4.139 created by runtime.gcenable
#21 4.139   runtime/mgc.go:178 +0xaa
#21 4.139 
#21 4.139 goroutine 18 [finalizer wait]:
#21 4.139 runtime.gopark(0xc0001041a0?, 0x0?, 0x70?, 0xa7?, 0x927b31?)
#21 4.139   runtime/proc.go:361 +0xd6 fp=0xc00005a630 sp=0xc00005a610 pc=0x91af76
#21 4.139 runtime.goparkunlock(...)
#21 4.139   runtime/proc.go:367
#21 4.139 runtime.runfinq()
#21 4.139   runtime/mfinal.go:177 +0xb3 fp=0xc00005a7e0 sp=0xc00005a630 pc=0x8fbbd3
#21 4.139 runtime.goexit()
#21 4.139   runtime/asm_amd64.s:1571 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x94b081
#21 4.139 created by runtime.createfing
#21 4.139   runtime/mfinal.go:157 +0x45
#21 4.139 
#21 4.139 goroutine 19 [select, locked to thread]:
#21 4.139 runtime.gopark(0xc0000567a8?, 0x2?, 0xf2?, 0xb2?, 0xc0000567a4?)
#21 4.139   runtime/proc.go:361 +0xd6 fp=0xc000056618 sp=0xc0000565f8 pc=0x91af76
#21 4.139 runtime.selectgo(0xc0000567a8, 0xc0000567a0, 0x0?, 0x0, 0x0?, 0x1)
#21 4.139   runtime/select.go:328 +0x772 fp=0xc000056758 sp=0xc000056618 pc=0x92a872
#21 4.139 runtime.ensureSigM.func1()
#21 4.139   runtime/signal_unix.go:973 +0x1bb fp=0xc0000567e0 sp=0xc000056758 pc=0x92ebfb
#21 4.139 runtime.goexit()
#21 4.139   runtime/asm_amd64.s:1571 +0x1 fp=0xc0000567e8 sp=0xc0000567e0 pc=0x94b081
#21 4.139 created by runtime.ensureSigM
#21 4.139   runtime/signal_unix.go:956 +0xbd
#21 4.139 
#21 4.139 goroutine 20 [syscall]:
#21 4.139 runtime.notetsleepg(0x0?, 0x0?)
#21 4.139   runtime/lock_futex.go:235 +0x34 fp=0xc000056fa0 sp=0xc000056f68 pc=0x8ef4f4
#21 4.139 os/signal.signal_recv()
#21 4.139   runtime/sigqueue.go:151 +0x2f fp=0xc000056fc0 sp=0xc000056fa0 pc=0x9475cf
#21 4.139 os/signal.loop()
#21 4.139   os/signal/signal_unix.go:23 +0x19 fp=0xc000056fe0 sp=0xc000056fc0 pc=0xba8e19
#21 4.139 runtime.goexit()
#21 4.139   runtime/asm_amd64.s:1571 +0x1 fp=0xc000056fe8 sp=0xc000056fe0 pc=0x94b081
#21 4.139 created by os/signal.Notify.func1.1
#21 4.139   os/signal/signal.go:151 +0x2a
#21 4.139 
#21 4.139 goroutine 5 [chan receive]:
#21 4.139 runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
#21 4.139   runtime/proc.go:361 +0xd6 fp=0xc00005c700 sp=0xc00005c6e0 pc=0x91af76
#21 4.139 runtime.chanrecv(0xc00012ab40, 0x0, 0x1)
#21 4.139   runtime/chan.go:577 +0x56c fp=0xc00005c790 sp=0xc00005c700 pc=0x8e9f4c
#21 4.139 runtime.chanrecv1(0x0?, 0x0?)
#21 4.139   runtime/chan.go:[440](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:444) +0x18 fp=0xc00005c7b8 sp=0xc00005c790 pc=0x8e9978
#21 4.139 github.com/vercel/turbo/cli/internal/signals.NewWatcher.func1()
#21 4.139   github.com/vercel/turbo/cli/internal/signals/signals.go:56 +0x28 fp=0xc00005c7e0 sp=0xc00005c7b8 pc=0xf2d7e8
#21 4.139 runtime.goexit()
#21 4.139   runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x94b081
#21 4.139 created by github.com/vercel/turbo/cli/internal/signals.NewWatcher
#21 4.139   github.com/vercel/turbo/cli/internal/signals/signals.go:55 +0x12f
#21 4.139 
#21 4.139 goroutine 96 [GC worker (idle)]:
#21 4.139 runtime.gopark(0x2cba803211?, 0xc0000468f0?, 0xd0?, 0x7?, 0xdea727?)
#21 4.139   runtime/proc.go:361 +0xd6 fp=0xc0000e0758 sp=0xc0000e0738 pc=0x91af76
#21 4.139 runtime.gcBgMarkWorker()
#21 4.139   runtime/mgc.go:1207 +0xe5 fp=0xc0000e07e0 sp=0xc0000e0758 pc=0x8fec05
#21 4.139 runtime.goexit()
#21 4.139   runtime/asm_amd64.s:1571 +0x1 fp=0xc0000e07e8 sp=0xc0000e07e0 pc=0x94b081
#21 4.139 created by runtime.gcBgMarkStartWorkers
#21 4.139   runtime/mgc.go:1131 +0x25
#21 4.139 
#21 4.139 goroutine 95 [GC worker (idle)]:
#21 4.139 runtime.gopark(0x2cba7908e1?, 0xc0000[468](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:472)f0?, 0xd0?, 0xf?, 0xdea727?)
#21 4.139   runtime/proc.go:361 +0xd6 fp=0xc0000e0f58 sp=0xc0000e0f38 pc=0x91af76
#21 4.139 runtime.gcBgMarkWorker()
#21 4.139   runtime/mgc.go:1207 +0xe5 fp=0xc0000e0fe0 sp=0xc0000e0f58 pc=0x8fec05
#21 4.139 runtime.goexit()
#21 4.139   runtime/asm_amd64.s:1571 +0x1 fp=0xc0000e0fe8 sp=0xc0000e0fe0 pc=0x94b081
#21 4.139 created by runtime.gcBgMarkStartWorkers
#21 4.139   runtime/mgc.go:1131 +0x25
#21 4.143 npm notice 
#21 4.144 npm notice New minor version of npm available! 9.5.1 -> 9.6.7
#21 4.144 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.6.7>
#21 4.144 npm notice Run `npm install -g npm@9.6.7` to update!
#21 4.144 npm notice 
#21 ERROR: process "/bin/sh -c npx --yes turbo@1.10.1 prune --scope="legacy" --docker" did not complete successfully: exit code: 2

Thanks

Expected Behavior

The turpo repo prune command succeeded with a lighter package.json.

To Reproduce

Not able to reproduce every time, it seems to be very hazardous.

Reproduction Repo

No response

chris-olszewski commented 1 year ago

Thanks for this report! This helped us track down a very interesting mistake we made with our FFI calls from Go to Rust. The description in #5230 has more information.

Klaitos commented 1 year ago

That was fast ! thanks guys