search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.96k stars 3.79k forks source link

pkg/sql/catalog/replication/replication_test: TestReaderCatalogTSAdvance failed #131719

Open github-actions[bot] opened 1 day ago

github-actions[bot] commented 1 day ago

pkg/sql/catalog/replication/replication_test.TestReaderCatalogTSAdvance failed on master @ b38c8dd1faa68970c3310c1e82da71d593a67052:

Fatal error:

panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008524757991981057/0, keybytes=[254 146 243 137 253 13 254 255 236 139 76 128 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008524757991981057/0, keybytes=[254 146 243 137 253 13 254 255 236 139 76 128 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008524757991981057/0, keybytes=[254 146 243 137 253 13 254 255 236 139 76 128 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008524757991981057/0, keybytes=[254 146 243 137 253 13 254 255 236 139 76 128 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008524757991981057/0, keybytes=[254 146 243 137 253 13 254 255 236 139 76 128 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008524757991981057/0, keybytes=[254 146 243 137 253 13 254 255 236 139 76 128 1 136] oldPrefix=[254 147 243]

Stack: 

goroutine 14417 [running]:
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc0078b6008, {0x6ba1158, 0xc00c3e5ec0}, {0x5114540?, 0xc00176ac90?})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1258 +0x19d
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1({0x6ba1158, 0xc00c3e5ec0}, {0xc0021ddbe0?})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:966 +0x45
panic({0x5114540?, 0xc00176ac90?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:127 +0x554
panic({0x5114540?, 0xc00176ac90?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:127 +0x554
panic({0x5114540?, 0xc00176ac90?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:127 +0x554
panic({0x5114540?, 0xc00176ac90?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:127 +0x554
panic({0x5114540?, 0xc00176ac90?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeSendFunc.makeExternalSpanSendFunc.func3({0xc006464220, 0xe, 0x10})
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:314 +0x38e
github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeSendFunc.makeExternalSpanSendFunc.func4({0x6ba1158, 0xc00e9ecba0}, 0xc001141040)
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:360 +0x184
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch(0xc01093d2c0, {0x6ba1158, 0xc00e9ecba0})
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:693 +0x522
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch(0xc01093d2c0, {0x6ba1158, 0xc00e9ecba0})
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:976 +0x1186
github.com/cockroachdb/cockroach/pkg/sql/row.(*kvBatchFetcherHelper).NextBatch(0xc01093d2c0, {0x6ba1158?, 0xc00e9ecba0?})
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:1125 +0xa2
github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchDirectScan).Next(0xc004aae240)
    github.com/cockroachdb/cockroach/pkg/sql/colfetcher/colbatch_direct_scan.go:79 +0xbc
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next(0xc01149d680)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93 +0x42
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next(0xc01149d6c0)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/cancel_checker.go:59 +0x2a
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next(0xc01149d700)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93 +0x42
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next(...)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0xc01149d740?)
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next(0xc003557700)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126 +0x4c
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*allSpooler).spool(0xc003d3de00)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:137 +0xda
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*sortOp).Next(0xc00aac2580)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:290 +0x6b
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:212 +0x29
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x2?)
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next(0xc0113626c0)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:210 +0x68
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next(0xc01149d900)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93 +0x42
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next(...)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0xc01149d980?)
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next(0xc003557800)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126 +0x4c
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter(...)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:246
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x6ba1120?)
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next(...)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:250
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run(0xc002d37900, {0x6ba1120?, 0xc00db7c1e0?})
    github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:282 +0x17f
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run(0xc003277840, {0x6ba1120, 0xc00db7c1e0}, 0x0)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320 +0x234
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc00a085b80, {0x6ba1158, 0xc00e9ec4e0}, 0xc009f51d40, 0xc0068a5040, 0xc00c8d2b00, 0xc0141c1808, 0xc0078b6bb8, 0xc0021dba10)
    github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:928 +0xd47
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc00a085b80, {0x6ba1158, 0xc00e9ec4e0}, 0xc0078b6bb8, 0xc009f51d40, 0xc0068a5040, {{0x6ba1ef0, 0xc00e9ec450}, 0x0}, 0xc0141c1808, ...)
    github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1990 +0x215
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3(0x10?, 0x10?, {0x6ba1158?, 0xc00e9ec4e0?}, 0x7fdf93b51d18?, 0xc00176a790?, 0x30?)
    github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1704 +0xbb
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll(0xc00a085b80, {0x6ba1158, 0xc00e9ec4e0}, 0xc0078b6bb8, 0xc009f51d40, 0xc0078b66b0, 0xc0141c1808, 0x0)
    github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1707 +0x229
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc0078b6008, {0x6ba1158, 0xc00e9ec4e0}, 0xc0078b66b0, 0x3?, {0x7fdf92c3d2d8?, 0xc002fa7f48?}, 0x1, 0xc004a94818, {0x0, ...})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2438 +0x485
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc0078b6008, {0x6ba1158, 0xc00e9d8ba0}, 0xc0078b66b0, {0x7fdf92c3d2d8, 0xc002fa7f48})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1985 +0x119e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc0078b6008, {0x6ba1158, 0xc00e9d8a80}, {{0x6bc6ba0, 0xc00dac17c0}, {0x0, 0x0, 0x0}, {0xc006d449c1, 0x1b}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1192 +0x4a85
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x6ba1158?, 0xc00e9d8a80?})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:146 +0xb0
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0x6ba1158?, {0x6ba1158?, 0xc00e9d8a80?}, {0x6bc6ba0?, 0xc00dac17c0?}, 0xc006d449c1?, 0x6?)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3454 +0x2ec
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc0078b6008, {0x6ba1158, 0xc00e9d8a80}, {{0x6bc6ba0, 0xc00dac17c0}, {0x0, 0x0, 0x0}, {0xc006d449c1, 0x1b}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145 +0x725
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1(0xc0021dd820, 0xc0021dd8d0, 0xc0078b6008, 0xc0021dd4f8, 0xc0021dd958, 0xc0021dd948, 0xc0021dd978, 0xc0021dd990)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2345 +0x3f1
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0078b6008)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2350 +0xa39
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0078b6008, {0x6ba1158, 0xc00c3e5ec0}, 0xc013b0fb30?, 0x0?, 0xc006caadf0?)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2267 +0x208
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc0083ddb00?, {0x6ba1158?, 0xc00c3e5ec0?}, {0xc0060b7680?}, 0x2?, 0xc00c3e5d10?)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:968 +0xda
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands(0xc002fa7808, {0x6ba1158, 0xc00c3e5ec0}, {0x0, 0x4, {0x1, {0x5608607, 0x3}, {0xc00301b3f0, 0xf}, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:256 +0x4a6
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func4()
    github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1136 +0xec
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl in goroutine 14415
    github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1133 +0x5db
Log preceding fatal error

``` E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*sortOp).Next E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:290 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next.func1 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:212 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:210 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:246 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:250 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:282 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:928 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1990 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1704 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1707 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2438 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1985 E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +Wraps: (6) external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008524757991981057/0, keybytes=[254 146 243 137 253 13 254 255 236 139 76 128 1 136] oldPrefix=[254 147 243] E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) *assert.withAssertionFailure (5) *withstack.withStack (6) *errutil.leafError E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +HINT: You have encountered an unexpected error. E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +Please check the public issue tracker to check whether this problem is E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +already tracked. If you cannot find it there, please report the error E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +with details by creating a new issue. E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +If you would rather not post publicly, please contact us directly E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +using the support form. E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 + E241002 05:42:19.322418 14417 1@util/log/logcrash/crash_reporting.go:196 [T11,Vdest,n1,client=127.0.0.1:45728,hostssl,user=root] 2260 +We appreciate your feedback. ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-42658

fqazi commented 1 day ago

This is similar to: https://github.com/cockroachdb/cockroach/issues/128070

cockroach-teamcity commented 7 hours ago

pkg/sql/catalog/replication/replication_test.TestReaderCatalogTSAdvance failed on master @ 0c0af9540ed3f9d63eba523bc870eeb6c7eebe90:

Fatal error:

panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008847998088642561/0, keybytes=[254 146 243 137 253 14 0 37 232 190 112 0 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008847998088642561/0, keybytes=[254 146 243 137 253 14 0 37 232 190 112 0 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008847998088642561/0, keybytes=[254 146 243 137 253 14 0 37 232 190 112 0 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008847998088642561/0, keybytes=[254 146 243 137 253 14 0 37 232 190 112 0 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008847998088642561/0, keybytes=[254 146 243 137 253 14 0 37 232 190 112 0 1 136] oldPrefix=[254 147 243] [recovered]
    panic: external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008847998088642561/0, keybytes=[254 146 243 137 253 14 0 37 232 190 112 0 1 136] oldPrefix=[254 147 243]

Stack: 

goroutine 14542 [running]:
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc00b812f88, {0x6bbb858, 0xc00fdd3110}, {0x5129f60?, 0xc0013ff7f0?})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1254 +0x19d
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1({0x6bbb858, 0xc00fdd3110}, {0xc0020ffbc8?})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:961 +0x45
panic({0x5129f60?, 0xc0013ff7f0?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:122 +0x554
panic({0x5129f60?, 0xc0013ff7f0?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:122 +0x554
panic({0x5129f60?, 0xc0013ff7f0?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:122 +0x554
panic({0x5129f60?, 0xc0013ff7f0?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:122 +0x554
panic({0x5129f60?, 0xc0013ff7f0?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeSendFunc.makeExternalSpanSendFunc.func3({0xc0085544a0, 0xe, 0x10})
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:309 +0x38e
github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeSendFunc.makeExternalSpanSendFunc.func4({0x6bbb858, 0xc00d3ccc30}, 0xc001523900)
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:355 +0x184
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch(0xc0141f41e0, {0x6bbb858, 0xc00d3ccc30})
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:688 +0x522
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch(0xc0141f41e0, {0x6bbb858, 0xc00d3ccc30})
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:971 +0x1186
github.com/cockroachdb/cockroach/pkg/sql/row.(*kvBatchFetcherHelper).NextBatch(0xc0141f41e0, {0x6bbb858?, 0xc00d3ccc30?})
    github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:1120 +0xa2
github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchDirectScan).Next(0xc00256c000)
    github.com/cockroachdb/cockroach/pkg/sql/colfetcher/colbatch_direct_scan.go:74 +0xbc
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next(0xc006472e80)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:88 +0x42
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next(0xc006472ec0)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/cancel_checker.go:54 +0x2a
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next(0xc006472f00)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:88 +0x42
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next(...)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:113
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0xc006472f40?)
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:147 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next(0xc014f37400)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:121 +0x4c
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*allSpooler).spool(0xc012155a40)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:132 +0xda
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*sortOp).Next(0xc00d34a420)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:285 +0x6b
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next.func1()
    github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:207 +0x29
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x2?)
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:147 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next(0xc004671950)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:205 +0x68
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next(0xc006473040)
    github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:88 +0x42
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next(...)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:113
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0xc006473080?)
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:147 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next(0xc014f37500)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:121 +0x4c
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter(...)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:241
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x6bbb820?)
    github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:147 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next(...)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:245
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run(0xc00cbddcc0, {0x6bbb820?, 0xc00c2e0b40?})
    github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:277 +0x17f
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run(0xc00a3e0b80, {0x6bbb820, 0xc00c2e0b40}, 0x0)
    github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:315 +0x234
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc012a0c840, {0x6bbb858, 0xc00d3cc5d0}, 0xc00f5cec30, 0xc0078f0a00, 0xc00a394900, 0xc003387808, 0xc00b813b38, 0xc0020fd9f8)
    github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:923 +0xd47
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc012a0c840, {0x6bbb858, 0xc00d3cc5d0}, 0xc00b813b38, 0xc00f5cec30, 0xc0078f0a00, {{0x6bbc630, 0xc00d3cc540}, 0x0}, 0xc003387808, ...)
    github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1985 +0x215
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3(0x10?, 0x10?, {0x6bbb858?, 0xc00d3cc5d0?}, 0x7fd0837f0058?, 0xc0013feea0?, 0x30?)
    github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1699 +0xbb
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll(0xc012a0c840, {0x6bbb858, 0xc00d3cc5d0}, 0xc00b813b38, 0xc00f5cec30, 0xc00b813630, 0xc003387808, 0x0)
    github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1702 +0x229
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc00b812f88, {0x6bbb858, 0xc00d3cc5d0}, 0xc00b813630, 0x3?, {0x7fd0818fc398?, 0xc01123f760?}, 0x1, 0xc0024e2f98, {0x0, ...})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2433 +0x485
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc00b812f88, {0x6bbb858, 0xc00a7dd3b0}, 0xc00b813630, {0x7fd0818fc398, 0xc01123f760})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1980 +0x119e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc00b812f88, {0x6bbb858, 0xc00a7dd2c0}, {{0x6be13e0, 0xc00c26bea0}, {0x0, 0x0, 0x0}, {0xc00205e9c1, 0x1b}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1187 +0x4a85
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x6bbb858?, 0xc00a7dd2c0?})
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141 +0xb0
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0x6bbb858?, {0x6bbb858?, 0xc00a7dd2c0?}, {0x6be13e0?, 0xc00c26bea0?}, 0xc00205e9c1?, 0x6?)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3449 +0x2ec
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc00b812f88, {0x6bbb858, 0xc00a7dd2c0}, {{0x6be13e0, 0xc00c26bea0}, {0x0, 0x0, 0x0}, {0xc00205e9c1, 0x1b}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:140 +0x725
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1(0xc0020ff808, 0xc0020ff8b8, 0xc00b812f88, 0xc0020ff4e0, 0xc0020ff940, 0xc0020ff930, 0xc0020ff960, 0xc0020ff978)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2341 +0x3f1
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc00b812f88)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2346 +0xa39
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc00b812f88, {0x6bbb858, 0xc00fdd3110}, 0xc006e9ec30?, 0x0?, 0xc0139b0970?)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2263 +0x208
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc0150c3b80?, {0x6bbb858?, 0xc00fdd3110?}, {0xc0025997a0?}, 0x2?, 0x0?)
    github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:963 +0xda
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands(0xc01123f008, {0x6bbb858, 0xc00fdd3110}, {0x0, 0x4, {0x1, {0x561f767, 0x3}, {0xc0089980d0, 0xf}, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:251 +0x4c8
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func4()
    github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1131 +0xec
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl in goroutine 14540
    github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1128 +0x5db
Log preceding fatal error

``` E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*sortOp).Next E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:285 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next.func1 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:207 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:147 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:205 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:88 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:113 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:147 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:121 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:241 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:147 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:245 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:277 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:315 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:923 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1985 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1699 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1702 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2433 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1980 E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +Wraps: (6) external row data does not have old prefix, key=/Tenant/10/Table/107/1/1008847998088642561/0, keybytes=[254 146 243 137 253 14 0 37 232 190 112 0 1 136] oldPrefix=[254 147 243] E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) *assert.withAssertionFailure (5) *withstack.withStack (6) *errutil.leafError E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +HINT: You have encountered an unexpected error. E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +Please check the public issue tracker to check whether this problem is E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +already tracked. If you cannot find it there, please report the error E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +with details by creating a new issue. E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +If you would rather not post publicly, please contact us directly E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +using the support form. E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 + E241003 09:06:23.468867 14542 1@util/log/logcrash/crash_reporting.go:231 [T11,Vdest,n1,client=127.0.0.1:53132,hostssl,user=root] 2261 +We appreciate your feedback. ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!