roachtest: sqlsmith/setup=tpcc/setting=ddl-nodrop failed

[cockroach-teamcity]

roachtest.sqlsmith/setup=tpcc/setting=ddl-nodrop failed with artifacts on master @ 0c0af9540ed3f9d63eba523bc870eeb6c7eebe90:

(test_runner.go:1308).runTest: test timed out (20m0s)
test artifacts and logs in: /artifacts/sqlsmith/setup=tpcc/setting=ddl-nodrop/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

Jira issue: CRDB-42712

[mgartner]

I241003 11:56:55.920938 22 util/log/clog.go:441 â‹® [-] 1748  SIGQUIT received. stack traces:
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹goroutine 22 [running]:›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x0?, 0xc002388100?})›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:27 +0x74›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:14›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/util/log.DumpStacks({0x857e6d8, 0xd20de00}, {0x6d0d2cf, 0x10})›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/util/log/clog.go:439 +0x47›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup.func1()›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/cli.go:337 +0x53›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹created by github.com/cockroachdb/cockroach/pkg/cli.debugSignalSetup in goroutine 1›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/cli.go:331 +0xe5›
I241003 11:56:55.920938 22 util/log/clog.go:441 â‹® [-] 1748 +
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹goroutine 1 [select, 20 minutes]:›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.waitForShutdown(0xc00468e2d0, 0xc004688cc0, 0xc0045b6ba0, 0xc00162d3e0)›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/start.go:980 +0x17f›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.runStartInternal(0xc6976e0, {0x6cc4ec4, 0x4}, 0xc0030d3a90, 0x715d690, 0x0)›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/start.go:666 +0x855›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.runStart(0x0?, {0x0?, 0x0?, 0x0?}, 0x0?)›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/start.go:373 +0x4c›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.runStartJoin(0x0?, {0xc0045c3380?, 0x0?, 0x4a7b435900000000?})›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/start.go:341 +0x1a›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.init.MaybeDecorateError.func114(0xc0030d3b48?, {0xc0045c3380?, 0x0?, 0x0?})›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/decorate_error.go:67 +0x34›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.init.MaybeShoutError.func115(0xc6976e0?, {0xc0045c3380?, 0x0?, 0xd?})›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/clierrorplus/shout.go:19 +0x1c›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/spf13/cobra.(*Command).execute(0xc6976e0, {0xc0045c32b0, 0xd, 0xd})›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:916 +0x882›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/spf13/cobra.(*Command).ExecuteC(0xc689f40)›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:1044 +0x3a5›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/spf13/cobra.(*Command).Execute(...)›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:968›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.Run(...)›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/cli.go:301›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.doMain(0xc6976e0, {0xc001a5ee6a, 0x5})›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/cli.go:139 +0x1e6›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹github.com/cockroachdb/cockroach/pkg/cli.Main()›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   github.com/cockroachdb/cockroach/pkg/cli/cli.go:64 +0x11f›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹main.main()›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   main/pkg/cmd/cockroach/main.go:21 +0xf›
I241003 11:56:55.920938 22 util/log/clog.go:441 â‹® [-] 1748 +
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹goroutine 8 [select]:›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹go.opencensus.io/stats/view.(*worker).start(0xc000e63a80)›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:292 +0x9f›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹created by go.opencensus.io/stats/view.init.0 in goroutine 1›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:34 +0x8d›
I241003 11:56:55.920938 22 util/log/clog.go:441 â‹® [-] 1748 +
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹goroutine 11 [syscall]:›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹os/signal.signal_recv()›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   GOROOT/src/runtime/sigqueue.go:152 +0x29›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹os/signal.loop()›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   GOROOT/src/os/signal/signal_unix.go:23 +0x13›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹created by os/signal.Notify.func1.1 in goroutine 37›
I241003 11:56:55.920938 22 util/log/clog.go:441 ⋮ [-] 1748 +‹   GOROOT/src/os/signal/signal.go:151 +0x1f›

[mgartner]

This also shows up a few times in the logs:

E241003 11:51:03.288570 66524 jobs/registry.go:1679 ⋮ [T1,Vsystem,n1] 1660  job 1008877739859214337: running execution encountered retriable error: failed to construct index entries during backfill: st_geogfromewkb(): error parsing EWKB: ‹EOF›
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +(1) forced error mark
E241003 11:51:03.288570 66524 jobs/registry.go:1679 ⋮ [T1,Vsystem,n1] 1660 +  | ‹"retriable job error"›
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +Wraps: (2) tags: [n1,received-error]
E241003 11:51:03.288570 66524 jobs/registry.go:1679 ⋮ [T1,Vsystem,n1] 1660 +Wraps: (3) tags: [n‹3›,f‹5003ecb2›,job=‹NEW SCHEMA CHANGE id=1008877739859214337›,distsql.gateway=‹1›,distsql.appname=‹$ internal-plan-index-backfill›,sent-error=]
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +Wraps: (4)
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | (opaque error wrapper)
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | reportable 0:
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:311
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.Group.GoCtx.func3
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:163
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | golang.org/x/sync/errgroup.(*Group).Go.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | runtime.goexit
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   src/runtime/asm_amd64.s:1695
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +Wraps: (5) failed to construct index entries during backfill
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +Wraps: (6) keys: [st_geogfromewkb()]
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +Wraps: (7)
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | (opaque error wrapper)
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | reportable 0:
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FuncExpr).MaybeWrapError
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/tree/eval.go:2148
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalFuncExpr
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:480
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FuncExpr).Eval
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:282
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.Expr
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:21
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func668
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/builtins/builtins.go:6489
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalFuncExpr
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:463
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FuncExpr).Eval
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:282
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.Expr
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:21
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/backfill.(*IndexBackfiller).BuildIndexEntriesChunk.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/backfill/backfill.go:894
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/backfill.(*IndexBackfiller).BuildIndexEntriesChunk
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/backfill/backfill.go:932
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).buildIndexEntryBatch.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:418
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:1917
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:2004
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/kv/txn.go:1052
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/kv/db.go:1098
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/kv/db.go:1061
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/kv/db.go:1036
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:1991
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:1918
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).buildIndexEntryBatch
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:411
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).constructIndexEntries
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:124
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:309
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.Group.GoCtx.func3
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:163
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | golang.org/x/sync/errgroup.(*Group).Go.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | runtime.goexit
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   src/runtime/asm_amd64.s:1695
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +Wraps: (8) st_geogfromewkb()
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +Wraps: (9) candidate pg code: 22023
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +Wraps: (10)
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | (opaque error wrapper)
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | reportable 0:
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/geo.parseEWKB
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/geo/parse.go:86
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/geo.ParseGeographyFromEWKB
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/geo/geo.go:512
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func73
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/builtins/geo_builtins.go:1056
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.bytesOverload1.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/builtins/builtins.go:10124
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalFuncExpr
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:478
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FuncExpr).Eval
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:282
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.Expr
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:21
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func668
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/builtins/builtins.go:6489
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalFuncExpr
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:463
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FuncExpr).Eval
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:282
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.Expr
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/sem/eval/expr.go:21
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/backfill.(*IndexBackfiller).BuildIndexEntriesChunk.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/backfill/backfill.go:894
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/backfill.(*IndexBackfiller).BuildIndexEntriesChunk
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/backfill/backfill.go:932
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).buildIndexEntryBatch.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:418
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:1917
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:2004
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/kv/txn.go:1052
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/kv/db.go:1098
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/kv/db.go:1061
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/kv/db.go:1036
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:1991
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:1918
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).buildIndexEntryBatch
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:411
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).constructIndexEntries
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:124
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexbackfiller.go:309
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.Group.GoCtx.func3
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:163
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | golang.org/x/sync/errgroup.(*Group).Go.func1
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  | runtime.goexit
E241003 11:51:03.288570 66524 jobs/registry.go:1679 â‹® [T1,Vsystem,n1] 1660 +  |   src/runtime/asm_amd64.s:1695