Open cockroach-teamcity opened 2 weeks ago
hrm; here's what i can see:
{
"workerId": 0,
"clientTimestamp": "10:42:22.220515",
"ops": [
"BEGIN",
{
"sql": "INSERT INTO public.table_w0_141 (col114_w0_115, col114_w0_116, col_w0_114_w0_140, col114_w0_118, col114_w0_119,
... [big insert here]
\xc35fe88abebb76':::BYTES,'{\"test\": \"json\"}':::JSONB)",
"potentialExecErr": "23502,23505,23514"
},
"COMMIT"
],
"expectedExecErrors": "",
"expectedCommitErrors": "",
"message": ""
}
I241010 10:42:26.488762 1 (gostd) testmain.go:85 [-] 1 Test //pkg/ccl/testccl/workload/schemachange:schemachange_test exited with error code -1
Execution result: https://tanzanite.cluster.engflow.com/actions/executions/ChB7V6iPtp1H0oPFeEInbgekEgdkZWZhdWx0GiUKIOtFdduN3wEq54bsOZMTOyyaoo_fhlkktKpSOfMxrJ3uEN8C
I believe we commit the txn above right before our failure:
I241010 10:42:25.633289 11426 9@util/log/event_log.go:39 ⋮ [T1,Vsystem,n1,client=127.0.0.1:41086,hostssl,user=‹testuser›] 3550 ={"Timestamp":1728556942130925307,"EventType":"query_execute","Statement":"COMMIT TRANSACTION","Tag":"COMMIT","User":"‹testuser›","ExecMode":"exec","Age":106.80501,"TxnCounter":86,"StmtPosInTxn":466}
Was this insert just too big?
pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 645eb8c99796b3b88f5631aa0fc92a011010ce64:
},
"COMMIT"
],
"expectedExecErrors": "",
"expectedCommitErrors": "",
"message": ""
}
schema_change_external_test.go:131:
Error Trace: pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:131
Error: Received unexpected error:
***UNEXPECTED ERROR; Failed to generate a random operation: failed generating operation: insertRow: ERROR: column "crdb_internal_column_3_name_placeholder" does not exist, referenced in "crdb_internal_column_3_name_placeholder IS NOT NULL" (SQLSTATE 42703)
(1) forced error mark
| "fatal error when running txn"
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (2)
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:486
| [...repeated from below...]
Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
Wraps: (5) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:219
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:473
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:592
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:120
| golang.org/x/sync/errgroup.(*Group).Go.func1
| golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (6) failed generating operation: insertRow
Wraps: (7) ERROR: column "crdb_internal_column_3_name_placeholder" does not exist, referenced in "crdb_internal_column_3_name_placeholder IS NOT NULL" (SQLSTATE 42703)
Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
Test: TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 33
Total Schema Statements Succeeded = 33
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 24
Total Transactions Rolled Back = 18
Total Transactions Executed = 42
schema_change_external_test.go:96: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_2_of_25/test.outputs/logTestWorkload299870959
panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload299870959
--- FAIL: TestWorkload (13.34s)
Parameters:
attempt=1
run=2
shard=1
pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 49ca24cedb042579e9645c206640d59975805d12:
],
"result": [
"abs(col14_w1_16)",
"abs(col14_w1_16)"
]
}
],
"previousStatements": [
"ALTER TABLE public.table_w1_14 DROP COLUMN col14_w1_19"
],
"usesDeclarativeSchemaChanger": true
}
}
schema_change_external_test.go:131:
Error Trace: pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:131
Error: Received unexpected error:
***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: column "col14_w1_19" is referenced by the primary key (SQLSTATE 42P10)
(1) forced error mark
| "fatal error when running txn"
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (2)
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3188
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:495
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:592
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:120
| golang.org/x/sync/errgroup.(*Group).Go.func1
| golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
Wraps: (5) ERROR: column "col14_w1_19" is referenced by the primary key (SQLSTATE 42P10)
Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
Test: TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 22
Total Schema Statements Succeeded = 22
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 11
Total Transactions Rolled Back = 18
Total Transactions Executed = 29
schema_change_external_test.go:96: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_12_of_25/test.outputs/logTestWorkload1711528738
panic.go:629: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload1711528738
--- FAIL: TestWorkload (9.37s)
Parameters:
attempt=1
run=12
shard=1
pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 47b5992396988e25e4b0c4c99d8e39265a3e481f:
"CASE WHEN col25_w1_27 IS NULL THEN e'@\\x18\\\\':::STRING ELSE e'\\x00':::STRING END",
"lower(CAST(col25_w1_29 AS STRING))",
"lower(CAST(col25_w1_29 AS STRING))",
"abs(col25_w1_28)"
]
}
],
"previousStatements": [
"ALTER TABLE public.table_w1_61 DROP COLUMN col25_w1_32"
],
"usesDeclarativeSchemaChanger": true
}
}
schema_change_external_test.go:131:
Error Trace: pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:131
Error: Received unexpected error:
***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: column "col25_w1_32" is referenced by the primary key (SQLSTATE 42P10)
(1) forced error mark
| "fatal error when running txn"
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (2)
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3188
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:495
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:592
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:120
| golang.org/x/sync/errgroup.(*Group).Go.func1
| golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
Wraps: (5) ERROR: column "col25_w1_32" is referenced by the primary key (SQLSTATE 42P10)
Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
Test: TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 25
Total Schema Statements Succeeded = 25
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 17
Total Transactions Rolled Back = 11
Total Transactions Executed = 28
schema_change_external_test.go:96: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload395502182
panic.go:629: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload395502182
--- FAIL: TestWorkload (9.01s)
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
{
"workerId": 0,
"clientTimestamp": "13:01:28.819945",
"ops": [
"BEGIN",
{
"sql": "ALTER TABLE public.table_w1_61 DROP COLUMN col25_w1_32"
}
],
"expectedExecErrors": "",
"expectedCommitErrors": "",
"message": "***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: column \"col25_w1_32\" is referenced by the primary key (SQLSTATE 42P10)",
this seems like a new flake. we should fix the workload so it does not try to drop a column that is in the PK.
pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ ebbe08756db4defbb6388041737e240ac07783a3:
],
"result": [
"abs(col6_w0_14)",
"abs(col6_w0_14)"
]
}
],
"previousStatements": [
"ALTER TABLE public.table_w0_6 DROP COLUMN col6_w0_15"
],
"usesDeclarativeSchemaChanger": true
}
}
schema_change_external_test.go:131:
Error Trace: pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:131
Error: Received unexpected error:
***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: column "col6_w0_15" is referenced by the primary key (SQLSTATE 42P10)
(1) forced error mark
| "fatal error when running txn"
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (2)
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3188
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:495
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:592
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:120
| golang.org/x/sync/errgroup.(*Group).Go.func1
| golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
Wraps: (5) ERROR: column "col6_w0_15" is referenced by the primary key (SQLSTATE 42P10)
Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
Test: TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 22
Total Schema Statements Succeeded = 22
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 10
Total Transactions Rolled Back = 10
Total Transactions Executed = 20
schema_change_external_test.go:96: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_13_of_25/test.outputs/logTestWorkload4025396994
panic.go:629: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload4025396994
--- FAIL: TestWorkload (10.95s)
Parameters:
attempt=1
run=13
shard=1
pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:
"query": "WITH tab_json AS (SELECT crdb_internal.pb_to_json('desc', descriptor)-\u003e'table' AS t FROM system.descriptor WHERE id = $1::REGCLASS), columns_json AS (SELECT json_array_elements(t-\u003e'columns') AS c FROM tab_json), columns AS (SELECT c-\u003e\u003e'computeExpr' AS generation_expression, c-\u003e\u003e'name' AS column_name, c-\u003e\u003e'id' AS ordinal FROM columns_json) SELECT generation_expression FROM columns WHERE generation_expression IS NOT NULL",
"queryArgs": [
"public.table_w0_3"
],
"result": []
}
],
"previousStatements": [
"ALTER TABLE public.table_w0_3 DROP COLUMN col_w0_3_w1_5"
],
"usesDeclarativeSchemaChanger": true
}
}
schema_change_external_test.go:131:
Error Trace: pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:131
Error: Received unexpected error:
***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: column "col_w0_3_w1_5" is referenced by the primary key (SQLSTATE 42P10)
(1) forced error mark
| "fatal error when running txn"
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (2)
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3189
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:503
| github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
| github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:600
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
| pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:120
| golang.org/x/sync/errgroup.(*Group).Go.func1
| golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
Wraps: (5) ERROR: column "col_w0_3_w1_5" is referenced by the primary key (SQLSTATE 42P10)
Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
Test: TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 46
Total Schema Statements Succeeded = 46
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 36
Total Transactions Rolled Back = 32
Total Transactions Executed = 68
schema_change_external_test.go:96: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_18_of_25/test.outputs/logTestWorkload1257607946
panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload1257607946
--- FAIL: TestWorkload (13.44s)
Parameters:
attempt=1
run=18
shard=1
See also: How To Investigate a Go Test Failure (internal)
- #132904 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-24.1 release-blocker]
pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ fd4b1464dbd6e385c6e51af26fe294fd2023a259:
Parameters:
attempt=1
run=15
shard=1
Help
See also: How To Investigate a Go Test Failure (internal)
/cc @cockroachdb/sql-foundationsThis test on roachdash | Improve this report!
Jira issue: CRDB-42933