search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

roachtest: schemachange/mixed-versions-compat failed [unexpected EOF during cockroach debug declarative-corpus-validate] #135807

Closed cockroach-teamcity closed 4 days ago

cockroach-teamcity commented 5 days ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions-compat failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
    pkg/util/log/log_flush.go:79 +0x26
goroutine 22 gp=0xc0017c01c0 m=8 mp=0xc001746008 [syscall]:
runtime.notetsleepg(0xd5873e0, 0xffffffffffffffff)
    GOROOT/src/runtime/lock_futex.go:246 +0x29 fp=0xc0009bcfa0 sp=0xc0009bcf78 pc=0x46f1a9
os/signal.signal_recv()
    GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc0009bcfc0 sp=0xc0009bcfa0 pc=0x4d51a9
os/signal.loop()
    GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0009bcfe0 sp=0xc0009bcfc0 pc=0xd98b53
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009bcfe8 sp=0xc0009bcfe0 pc=0x4d90a1
created by os/signal.Notify.func1.1 in goroutine 21
    GOROOT/src/os/signal/signal.go:151 +0x1f
goroutine 37 gp=0xc0011d0540 m=nil [chan receive]:
runtime.gopark(0x0?, 0x4?, 0xd0?, 0x1e?, 0x26a63bf?)
    GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009c1e40 sp=0xc0009c1e20 pc=0x4a050e
runtime.chanrecv(0xc0016fe8a0, 0xc0009c1fb8, 0x1)
    GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0009c1eb8 sp=0xc0009c1e40 pc=0x468ddf
runtime.chanrecv1(0x878b440?, 0xc0016e7090?)
    GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0009c1ee0 sp=0xc0009c1eb8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
    pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0009c1fe0 sp=0xc0009c1ee0 pc=0x26a6665
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009c1fe8 sp=0xc0009c1fe0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
    pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 38 gp=0xc001cd2380 m=nil [runnable]:
runtime.gopark(0xc0000defb0?, 0x2?, 0x0?, 0x0?, 0xc0000def8c?)
    GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000dee38 sp=0xc0000dee18 pc=0x4a050e
runtime.selectgo(0xc0000defb0, 0xc0000def88, 0xc00241a4f0?, 0x0, 0xc0023f7a70?, 0x1)
    GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0000def58 sp=0xc0000dee38 pc=0x4b2465
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
    cli.go:333 +0x91 fp=0xc0000defe0 sp=0xc0000def58 pc=0x5944311
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000defe8 sp=0xc0000defe0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
    cli.go:331 +0xe5
goroutine 39 gp=0xc001cd2540 m=nil [runnable]:
runtime.gopark(0xc0000e17a8?, 0x2?, 0x0?, 0x0?, 0xc0000e171c?)
    GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000e15a8 sp=0xc0000e1588 pc=0x4a050e
runtime.selectgo(0xc0000e17a8, 0xc0000e1718, 0x0?, 0x0, 0x0?, 0x1)
    GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0000e16c8 sp=0xc0000e15a8 pc=0x4b2465
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
    cli.go:355 +0x9f fp=0xc0000e17e0 sp=0xc0000e16c8 pc=0x5943d9f
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000e17e8 sp=0xc0000e17e0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
    cli.go:353 +0x1a5
bash: line 8: 82770 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus680433151/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #134934 roachtest: schemachange/mixed-versions-compat failed [missing corpus file] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.3.0-rc] - #133003 roachtest: schemachange/mixed-versions-compat failed [missing corpus file] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.3]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-44726

cockroach-teamcity commented 4 days ago

roachtest.schemachange/mixed-versions-compat failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
    pkg/util/log/log_flush.go:79 +0x26
goroutine 8 gp=0xc001382380 m=nil [select, locked to thread]:
runtime.gopark(0xc0009bf7a8?, 0x2?, 0x60?, 0xf6?, 0xc0009bf794?)
    GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009bf638 sp=0xc0009bf618 pc=0x4a050e
runtime.selectgo(0xc0009bf7a8, 0xc0009bf790, 0x0?, 0x0, 0x0?, 0x1)
    GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009bf758 sp=0xc0009bf638 pc=0x4b2465
runtime.ensureSigM.func1()
    GOROOT/src/runtime/signal_unix.go:1034 +0x19f fp=0xc0009bf7e0 sp=0xc0009bf758 pc=0x4cf3ff
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009bf7e8 sp=0xc0009bf7e0 pc=0x4d90a1
created by runtime.ensureSigM in goroutine 52
    GOROOT/src/runtime/signal_unix.go:1017 +0xc8
goroutine 53 gp=0xc000174540 m=nil [chan receive]:
runtime.gopark(0xc001568540?, 0x0?, 0x4?, 0x0?, 0xc0000df6d0?)
    GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0000df640 sp=0xc0000df620 pc=0x4a050e
runtime.chanrecv(0xc000136240, 0xc0000df7b8, 0x1)
    GOROOT/src/runtime/chan.go:583 +0x3bf fp=0xc0000df6b8 sp=0xc0000df640 pc=0x468ddf
runtime.chanrecv1(0x8760ec0?, 0xc0013100a0?)
    GOROOT/src/runtime/chan.go:442 +0x12 fp=0xc0000df6e0 sp=0xc0000df6b8 pc=0x4689f2
github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
    pkg/util/goschedstats/runnable.go:190 +0xe5 fp=0xc0000df7e0 sp=0xc0000df6e0 pc=0x269c005
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000df7e8 sp=0xc0000df7e0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
    pkg/util/goschedstats/runnable.go:182 +0x1a
goroutine 36 gp=0xc004592a80 m=nil [runnable]:
runtime.gopark(0xc0009c37b0?, 0x2?, 0x0?, 0x0?, 0xc0009c378c?)
    GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009c3638 sp=0xc0009c3618 pc=0x4a050e
runtime.selectgo(0xc0009c37b0, 0xc0009c3788, 0x4?, 0x0, 0x0?, 0x1)
    GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009c3758 sp=0xc0009c3638 pc=0x4b2465
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()
    cli.go:333 +0x91 fp=0xc0009c37e0 sp=0xc0009c3758 pc=0x591c871
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009c37e8 sp=0xc0009c37e0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
    cli.go:331 +0xe5
goroutine 37 gp=0xc004592c40 m=nil [runnable]:
runtime.gopark(0xc0009c3fa8?, 0x2?, 0x0?, 0x0?, 0xc0009c3f1c?)
    GOROOT/src/runtime/proc.go:402 +0xce fp=0xc0009c3da8 sp=0xc0009c3d88 pc=0x4a050e
runtime.selectgo(0xc0009c3fa8, 0xc0009c3f18, 0x0?, 0x0, 0x0?, 0x1)
    GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0009c3ec8 sp=0xc0009c3da8 pc=0x4b2465
github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func2()
    cli.go:355 +0x9f fp=0xc0009c3fe0 sp=0xc0009c3ec8 pc=0x591c2ff
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009c3fe8 sp=0xc0009c3fe0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1
    cli.go:353 +0x1a5
bash: line 8: 57593 Aborted                 (core dumped) bash -c "./cockroach debug declarative-corpus-validate /tmp/corpus4225922576/corpus"
test artifacts and logs in: /artifacts/schemachange/mixed-versions-compat/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #134934 roachtest: schemachange/mixed-versions-compat failed [missing corpus file] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.3.0-rc] - #133003 roachtest: schemachange/mixed-versions-compat failed [missing corpus file] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.3]

This test on roachdash | Improve this report!

rafiss commented 4 days ago 
run_074604.592078636_n1_cockroach-debug-decl: 2024/11/21 07:46:04 cluster.go:2550: > ./cockroach debug declarative-corpus-validate /tmp/corpus4225922576/corpus
teamcity-17868467-1732171355-04-n1cpu4:[1]: ./cockroach debug declarati...
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1  a panic has occurred!
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +unexpected EOF
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +(1) attached stack trace
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  -- stack trace:
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.init.MaybeDecorateError.func69
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    pkg/cli/clierrorplus/decorate_error.go:67
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).execute
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    external/com_github_spf13_cobra/command.go:916
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).ExecuteC
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    external/com_github_spf13_cobra/command.go:1044
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/spf13/cobra.(*Command).Execute
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    external/com_github_spf13_cobra/command.go:968
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.Run
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    cli.go:301
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.doMain
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    cli.go:139
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.Main
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    cli.go:64
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | main.main
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    pkg/cmd/cockroach/main.go:21
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | runtime.main
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    GOROOT/src/runtime/proc.go:271
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  | runtime.goexit
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +  |    src/runtime/asm_amd64.s:1695
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +Wraps: (2) unexpected EOF
E241121 07:46:05.280450 1 1@util/log/logcrash/crash_reporting.go:192  [-] 1 +Error types: (1) *withstack.withStack (2) *errors.errorString
panic: unexpected EOF [recovered]
    panic: unexpected EOF

goroutine 1 gp=0xc0000061c0 m=10 mp=0xc001daa008 [running]:
panic({0x62eaa00?, 0xc8b3950?})
    GOROOT/src/runtime/panic.go:779 +0x158 fp=0xc003195548 sp=0xc003195498 pc=0x49ca98
github.com/cockroachdb/cockroach/pkg/util/log/logcrash.RecoverAndReportPanic({0x87aca60, 0xd54cf00}, 0xc0032c2000)
    pkg/util/log/logcrash/crash_reporting.go:135 +0x6e fp=0xc003195598 sp=0xc003195548 pc=0xf10a2e
github.com/cockroachdb/cockroach/pkg/cli.doMain.deferwrap1()
    cli.go:137 +0x28 fp=0xc0031955c0 sp=0xc003195598 pc=0x591b788
panic({0x62eaa00?, 0xc8b3950?})
    GOROOT/src/runtime/panic.go:770 +0x132 fp=0xc003195670 sp=0xc0031955c0 pc=0x49ca72
github.com/cockroachdb/cockroach/pkg/cli.init.func17(0xc9d20e0, {0xc00180e900?, 0x4be49e0?, 0xc003305800?})
    declarative_corpus.go:38 +0x54c fp=0xc003195b18 sp=0xc003195670 pc=0x590c88c
github.com/cockroachdb/cockroach/pkg/cli.init.MaybeDecorateError.func69(0xc9d20e0?, {0xc00180e900?, 0x1?, 0x1?})
    pkg/cli/clierrorplus/decorate_error.go:67 +0x34 fp=0xc003195c58 sp=0xc003195b18 pc=0x58f9114
github.com/spf13/cobra.(*Command).execute(0xc9d20e0, {0xc00180e8d0, 0x1, 0x1})
    external/com_github_spf13_cobra/command.go:916 +0x882 fp=0xc003195d78 sp=0xc003195c58 pc=0x4bdaf02
github.com/spf13/cobra.(*Command).ExecuteC(0xc9cc200)
    external/com_github_spf13_cobra/command.go:1044 +0x3a5 fp=0xc003195e30 sp=0xc003195d78 pc=0x4bdb745
github.com/spf13/cobra.(*Command).Execute(...)
    external/com_github_spf13_cobra/command.go:968
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
    cli.go:301
github.com/cockroachdb/cockroach/pkg/cli.doMain(0xc9d20e0, {0xc00215120a, 0x21})
    cli.go:139 +0x1e6 fp=0xc003195eb0 sp=0xc003195e30 pc=0x591b6a6
github.com/cockroachdb/cockroach/pkg/cli.Main()
    cli.go:64 +0x11f fp=0xc003195f40 sp=0xc003195eb0 pc=0x591b35f
main.main()
    pkg/cmd/cockroach/main.go:21 +0xf fp=0xc003195f50 sp=0xc003195f40 pc=0x5b0542f
runtime.main()
    GOROOT/src/runtime/proc.go:271 +0x29d fp=0xc003195fe0 sp=0xc003195f50 pc=0x4a00bd
runtime.goexit({})
    src/runtime/asm_amd64.s:1695 +0x1 fp=0xc003195fe8 sp=0xc003195fe0 pc=0x4d90a1
rafiss commented 4 days ago

Assigning to @Dedej-Bergin as this is the same as https://github.com/cockroachdb/cockroach/issues/133003

blathers-crl[bot] commented 4 days ago

Based on the specified backports for linked PR #135933, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.