pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [duplicate key value violates unique constraint during CREATE UNIQUE INDEX]

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.13k stars 3.81k forks source link

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [duplicate key value violates unique constraint during CREATE UNIQUE INDEX] #129366

Closed github-actions[bot] closed 2 months ago

github-actions[bot] commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ ebefe416f4389f496db56c1c56898b084d1e23d1:

   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT mut FROM (SELECT json_array_elements(d-\u003e'mutations') AS mut FROM descriptors) WHERE (mut-\u003e'primaryKeySwap') IS NOT NULL)",
    "queryArgs": [
     "public.table_w1_1"
    ],
    "result": false
   },
   {
    "query": "SELECT EXISTS (SELECT * FROM [SHOW REGIONS FROM DATABASE])",
    "result": false
   }
  ],
  "previousStatements": [
   "CREATE UNIQUE INDEX index_w1_1_w0_26 ON public.table_w1_1 (col1_w1_3 ASC) STORING (col1_w1_6, col1ͫ_w1_8, col1_w1_10)"
  ]
 }
}
    schema_change_external_test.go:134: 
            Error Trace:    pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
            Error:          Received unexpected error:
                            ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "index_w1_1_w0_26" (SQLSTATE 23505)
                            (1) forced error mark
                              | "fatal error when running txn"
                              | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                            Wraps: (2)
                            Wraps: (3) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
                              | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
                              |     pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
                              | golang.org/x/sync/errgroup.(*Group).Go.func1
                              |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
                            Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "index_w1_1_w0_26" (SQLSTATE 23505)
                            Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
            Test:           TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 27
Total Schema Statements Succeeded = 27
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 20
Total Transactions Rolled Back = 19
Total Transactions Executed = 39
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/test.outputs/logTestWorkload2330105537
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload2330105537
--- FAIL: TestWorkload (11.18s)

Parameters:

attempt=1
run=1
shard=1
Help

Same failure on other branches

- #129325 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [transaction committed but schema change aborted; table was dropped] [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.2] - #128953 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed ["enum type does not exist" error code when setting a DEFAULT] [C-test-failure O-robot P-3 T-sql-foundations branch-release-23.2.10-rc]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-41506

github-actions[bot] commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 434444078f6fec9e08d180d3b07a34bc1bdb3903:

  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 0,
 "clientTimestamp": "14:25:30.225838",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE SEQUENCE IF NOT EXISTS schema_w0_106.seq_w0_107",
   "expectedExecErr": "3F000"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot create \"schema_w0_106.seq_w0_107\" because the target database or schema does not exist (SQLSTATE 3F000)"
}
{
 "workerId": 0,
 "clientTimestamp": "14:25:30.2642",
 "ops": [
  "BEGIN",
  {
   "sql": "SELECT t0.\"coL5_w1_10\" AS col0, t0.col5_w1_8 AS col1, t1.\"coL5_w1_10\" AS col2, t0.col5_w1_7 AS col3, t0.col5_w1_8 AS col4, t1.\"coL5_w1_10\" AS col5, t1.col5_w1_8 AS col6, t0.\" col5_w1_6\" AS col7, t1.col5_w1_7 AS col8, t1.col5_w1_11 AS col9 FROM public.table_w1_12 AS t0 ,public.table_w1_12 AS t1 FETCH FIRST 1 ROWS ONLY",
   "potentialExecErr": "53100,53200,57014"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 0,
 "clientTimestamp": "14:25:30.396993",
 "ops": [
  "BEGIN",
  {
   "sql": "INSERT INTO public.table_w1_12 (\" col5_w1_6\", col5_w1_7, col5_w1_8) VALUES (3614:::OID,98:::INT8,ARRAY[false,true,false]),(3220:::OID,78:::INT8,ARRAY[false,false,false,false])",
   "potentialExecErr": "23502,23505,23514"
  }
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: duplicate key value violates unique constraint \"table_w1_5_col5_w1_7_key\" (SQLSTATE 23505)"
}

Parameters:

attempt=1
run=1
shard=1
Help

Same failure on other branches

- #129462 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-24.2.1-rc release-blocker] - #129325 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [transaction committed but schema change aborted; table was dropped] [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.2] - #128953 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed ["enum type does not exist" error code when setting a DEFAULT] [C-test-failure O-robot P-3 T-sql-foundations branch-release-23.2.10-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ db79d36847c63ee90b551ac551668f010c32a8da:

   {
    "query": "SELECT EXISTS (SELECT table_name FROM information_schema.views WHERE (table_schema = $1) AND (table_name = $2))",
    "queryArgs": [
     "public",
     "view_w1_211"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "CREATE VIEW public.view_w1_211 AS SELECT schema_w0_33.table_w1_50.crdb_internal_mvcc_timestamp, schema_w0_33.table_w1_189.\"col18̧9%q_w1_190\" FROM schema_w0_33.table_w1_50, schema_w0_33.table_w1_189"
  ]
 }
}
    schema_change_external_test.go:134: 
            Error Trace:    pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
            Error:          Received unexpected error:
                            ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: relation "view_w1_211" (167): column name "crdb_internal_mvcc_timestamp" conflicts with a system column name (SQLSTATE 42701)
                            (1) forced error mark
                              | "fatal error when running txn"
                              | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                            Wraps: (2)
                            Wraps: (3) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3139
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:499
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:587
                              | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
                              |     pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
                              | golang.org/x/sync/errgroup.(*Group).Go.func1
                              |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                              | runtime.goexit
                              |     src/runtime/asm_arm64.s:1222
                            Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
                            Wraps: (5) ERROR: relation "view_w1_211" (167): column name "crdb_internal_mvcc_timestamp" conflicts with a system column name (SQLSTATE 42701)
                            Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
            Test:           TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 164
Total Schema Statements Succeeded = 164
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 106
Total Transactions Rolled Back = 83
Total Transactions Executed = 189
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload3060628944
    panic.go:626: -- test log scope end --
--- FAIL: TestWorkload (20.20s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 95dc79bf17c6a377a2cca5b93c34be8c3a9502d3:

   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT mut FROM (SELECT json_array_elements(d-\u003e'mutations') AS mut FROM descriptors) WHERE (mut-\u003e'primaryKeySwap') IS NOT NULL)",
    "queryArgs": [
     "schema_w1_3.table_w0_142"
    ],
    "result": false
   },
   {
    "query": "SELECT EXISTS (SELECT * FROM [SHOW REGIONS FROM DATABASE])",
    "result": false
   }
  ],
  "previousStatements": [
   "CREATE UNIQUE INDEX index_w0_142_w0_181 ON schema_w1_3.table_w0_142 (col142_w0_145) STORING (col142_w0_147)"
  ]
 }
}
    schema_change_external_test.go:134: 
            Error Trace:    pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
            Error:          Received unexpected error:
                            ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "index_w0_142_w0_181" (SQLSTATE 23505)
                            (1) forced error mark
                              | "fatal error when running txn"
                              | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                            Wraps: (2)
                            Wraps: (3) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
                              | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
                              |     pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
                              | golang.org/x/sync/errgroup.(*Group).Go.func1
                              |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                              | runtime.goexit
                              |     src/runtime/asm_arm64.s:1222
                            Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
                            Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "index_w0_142_w0_181" (SQLSTATE 23505)
                            Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
            Test:           TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 80
Total Schema Statements Succeeded = 80
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 53
Total Transactions Rolled Back = 48
Total Transactions Executed = 101
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload3243003916
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload3243003916
--- FAIL: TestWorkload (13.46s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

This test on roachdash | Improve this report!

github-actions[bot] commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 2e698ceaa38c49c15611db67370a6dd25c6268e5:

   },
   {
    "query": "SELECT ((CASE WHEN \"co\\\\U000E7562l119_w1_121\" IS NULL THEN e'k\\bW\\x02q':::STRING ELSE e'\"FQ!i0\\x17':::STRING END)::STRING) IS NULL AS c FROM (VALUES (e'KIqBsQ=\\rL'::NAME:::NAME, e'\\\\xff':::BYTES, e'\"FQ!i0\\x17':::STRING)) AS t (col119_w1_120, \"co\\\\U000E7562l119_w1_121\", col119_w1_122)",
    "result": false
   },
   {
    "query": "SELECT ARRAY[parent.table_schema, parent.table_name, parent.column_name, child.column_name] FROM (SELECT conname, conkey, confkey, conrelid, confrelid FROM pg_constraint WHERE (contype = 'f') AND (conrelid = 'schema_w1_58.table_w1_119'::REGCLASS::INT8)) AS con JOIN (SELECT column_name, ordinal_position, column_default FROM information_schema.columns WHERE (table_schema = 'schema_w1_58') AND (table_name = 'table_w1_119')) AS child ON conkey[1] = child.ordinal_position JOIN (SELECT pc.oid, cols.table_schema, cols.table_name, cols.column_name, cols.ordinal_position FROM pg_class AS pc JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid JOIN information_schema.columns AS cols ON ((pc.relname = cols.table_name) AND (pn.nspname = cols.table_schema))) AS parent ON ((con.confkey[1] = parent.ordinal_position) AND (con.confrelid = parent.oid)) WHERE child.column_name != 'rowid'",
    "result": null
   }
  ],
  "previousStatements": [
   "INSERT INTO schema_w1_58.table_w1_119 (col119_w1_120, \"co\\\\U000E7562l119_w1_121\") VALUES (e'\\x10Kg'::NAME:::NAME,'\\x480646a0df98':::BYTES),(e'KIqBsQ=\\rL'::NAME:::NAME,'\\xff':::BYTES)"
  ],
  "usesDeclarativeSchemaChanger": true
 }
}
    schema_change_external_test.go:134: 
            Error Trace:    pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
            Error:          Received unexpected error:
                            ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23502): validation of column "col_w0_199_w0_272" NOT NULL failed on row: "
                            (1) forced error mark
                              | "fatal error when running txn"
                              | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                            Wraps: (2)
                            Wraps: (3) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
                              | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
                              |     pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
                              | golang.org/x/sync/errgroup.(*Group).Go.func1
                              |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
                            Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23502): validation of column "col_w0_199_w0_272" NOT NULL failed on row: "
                              | col199_w0_200"='4714-11-24 BC', col199_w0_201=-97, col199_w0_202=false, "co.l199_w0_203"='01070000E0E61000000500000001040000E0E61000000600000001010000C05C03D344CF8E52402C5FA65F0725524098F39113DCC9F641C0695CC411B4F7C101010000C000B5827535B754C016D02AE7DD26564034C062939615FBC136F05E3F3B8D024201010000C0085771014ABD5240508C8F09DD9037C0B445A6F4AFC5F9C1C84FB3174C97EFC101010000C09E9639DAF5CF59C014F48B53B1434E40E030437630A4F0C124EE4472C102FB4101010000C0FC79782F7F3C6040C0E40023EF1E064032C215D3565FE0C13041A635499CFE4101010000C0EC5D0FE6C5BB614080DF1B0302274B40FEFA7760DF80FAC124C21443B90BFF4101010000E0E6100000A271036B2ABD5DC0C1EF1876361D52C0F083DC982670E4C180774F12BA71B7C101050000E0E61000000200000001020000C006000000ECD40AD73D394EC08DF45817855648C0C65DDACBF19FE0C17144D9F03CBAF1C148C2F6AD9738434016D9446CB7C149C0A6A83461B448F241B6A6F9E272EBEEC1F0A2DB9B0BB85340C7A7B4B2BACA44C0085BFA4927C201C2B6E6E9B3FE9BF9415EFEA2EE7B896440A4F519120F4C394038D85719CCBDF2410E027AF0F907F64138522A2E9EA55B40B8C9FB7C94D94F40147429576969FCC1D8DA2A7F17E4D841C48F98C641F35840E4D579064684524054C89713970BFE41563DEBE10D72024201020000C00500000098C8D5883C336440B84D1241788150C01E72AE229FA3EAC17ED470FECFC5EBC1E87083CE5CF15F40485B499BAB3728C09EED17EBB15DF441EC81979FC50BFDC1A42AF9DAFC075C40F8D1042E5160384088C5974E62FAFF41D80EA5CEBA0C00C25C39EF229B204E40D48D46B2602A3640C09606F3D38AB3C1ACBC65E58441FD41241DED94C6F64E4048026A0700ED524086F05EECA07EF24180B8F4211659C74101020000E0E610000006000000D86DE258E90232C0947415CBEB1336C0D094090436C0D041AA63DAC2B22CF641B8D407ACD5A241C0D17599422CD946C0E0F63088ADDFBFC16020E400CC6CBFC1801F7B81672F09C0CA0AC51E626E3EC070E7AABE459DCEC195BABBC0B9DB00C29E9B7750FBAD6140C39ACA3F45AA49C0AB5332E19001F3C1E76DDE666392F4C1D679E21F836A65409844F575881736C078350FE95ED4FD41B4C7E9801FFEEDC1B889BB0A49B3524016363C08649740402E3D9E56652C0142ECD20A8988C4F24101030000E0E61000000100000005000000E48DC31B48E262C0F1EF7BC598C354C0EB20ADF12265F0C1801D4D5EBF2497413034DB2157405C4088D42294307320C0FCC397ACCCD7DFC122F4245D0617F84186A29F966BE3624000467B7AB734D93FB02A80030693FC41F04C596CA880E441E8291BDDADB560C020A6F358295D1F40809755842B8DA84130FDA13F2620E3C1E48DC31B48E262C0F1EF7BC598C354C0EB20ADF12265F0C1801D4D5EBF249741', col199_w0_204=ARRAY[0.661987641233253,0.6565967691246257,1.6359130742665116,-0.20667278364053185], "c?ol1͋99_w0_205"='24:00:00', col199_w0_206=1186, col199_w0_207=ARRAY['C8A395F0/9E186E32','7F6CBBAF/99BB2183','8F813912/498A6F85','FFFFFFFF/FFFFFFFF'], "col19*9_w0_208"='{"PDHQuP": false, "cg": [{"bar": null, "vipERW": null}, [true], null, null], "foobar": true}', col199_w0_209='ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff', " col199_w0_210"='BOX(-1.744934856183839 -0.7825346227770783,1.3588369671828708 3.152259477123554)', col199_w0_211='1979-11-24', col199_w0_212='1988-10-23 02:16:49.000791', colͪ199_w0_213='17:26:29.310302', col199_w0_214='rYSvcgjwUG' <-> 'BhujmMSQs', rowid=997324429278445569, col_w0_199_w0_272=NULL (SQLSTATE 23502)
                            Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
            Test:           TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 140
Total Schema Statements Succeeded = 140
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 80
Total Transactions Rolled Back = 73
Total Transactions Executed = 153
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/test.outputs/logTestWorkload3102083793
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload3102083793
--- FAIL: TestWorkload (29.21s)

Parameters:

attempt=1
run=1
shard=1
Help

Same failure on other branches

- #129462 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.2.1-rc] - #128953 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed ["enum type does not exist" error code when setting a DEFAULT] [C-test-failure O-robot P-3 T-sql-foundations branch-release-23.2.10-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 9e48858c6c8b22af4ec1159bcff6e233e7bfddff:

   "expectedExecErr": "42P16"
  }
 ],
 "expectedExecErrors": "42P16",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot alter a table's LOCALITY if its database is not multi-region enabled (SQLSTATE 42P16)"
}
{
 "workerId": 0,
 "clientTimestamp": "06:57:13.208838",
 "ops": [
  "BEGIN",
  {
   "sql": "SELECT t0.co😰l86_w0_87 AS col0, t0.\"coL86_w0_101\" AS col1, t0.co😰l86_w0_87 AS col2, t0.col86_w0_97 AS col3, t0.col😺86_w0_98 AS col4, t0.\"co l86_w0_99\" AS col5, t0.col86_w0_91 AS col6, t0.col😺86_w0_98 AS col7, t0.\"co|l86_w0_94\" AS col8 FROM public.table_w0_86 AS t0 FETCH FIRST 1 ROWS ONLY",
   "potentialExecErr": "53100,53200,57014"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 0,
 "clientTimestamp": "06:57:13.374833",
 "ops": [
  "BEGIN",
  {
   "sql": "INSERT INTO public.table_w0_221 (IrrelevantColumnName) VALUES (\"IrrelevantValue\")",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table_w0_221\" does not exist (SQLSTATE 42P01)"
}
{
 "workerId": 0,
 "clientTimestamp": "06:57:13.484662",
 "ops": [
  "BEGIN",
  {
   "sql": "ALTER TABLE public.table_w0_171 ADD COLUMN \"col1(30_w0_144\" INT8 NOT NULL",
   "expectedExecErr": "23502,42701"
  }
 ],
 "expectedExecErrors": "23502,42701",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: column \"col1(30_w0_144\" of relation \"table_w0_171\" already exists (SQLSTATE 42701)"
}

Parameters:

attempt=1
run=4
shard=1
Help

Same failure on other branches

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 9e48858c6c8b22af4ec1159bcff6e233e7bfddff:

   {
    "query": "SELECT EXISTS (SELECT * FROM [SHOW REGIONS FROM DATABASE])",
    "result": false
   },
   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT 1 FROM descriptors WHERE ((d-\u003e'localityConfig')-\u003e'regionalByRow') IS NOT NULL)",
    "queryArgs": [
     "schema_w0_1.table_w0_70"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "ALTER TABLE schema_w0_1.table_w0_70 ADD CONSTRAINT \"table_w0_70_col2_w1_3_unique\" UNIQUE (col2_w1_3)"
  ]
 }
}
    schema_change_external_test.go:134: 
            Error Trace:    pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
            Error:          Received unexpected error:
                            ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "table_w0_70_col2_w1_3_unique" (SQLSTATE 23505)
                            (1) forced error mark
                              | "fatal error when running txn"
                              | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                            Wraps: (2)
                            Wraps: (3) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
                              | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
                              |     pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
                              | golang.org/x/sync/errgroup.(*Group).Go.func1
                              |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
                            Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "table_w0_70_col2_w1_3_unique" (SQLSTATE 23505)
                            Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
            Test:           TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 50
Total Schema Statements Succeeded = 50
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 35
Total Transactions Rolled Back = 30
Total Transactions Executed = 65
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_19_of_25/test.outputs/logTestWorkload2383049155
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload2383049155
--- FAIL: TestWorkload (16.36s)

Parameters:

attempt=1
run=19
shard=1
Help

Same failure on other branches

This test on roachdash | Improve this report!

github-actions[bot] commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 9027e081ccc98fd344402d6a5db5eb53ebfc9efe:

   {
    "query": "SELECT ARRAY[(col196_w0_204 + col196_w0_202)::INT8::STRING] AS c FROM (VALUES ('[[[[false, \"BiCOi\", true], \"c\"], [false, \"8Sd9Kgf\"], true], [{\"a\": \"9x\", \"c\": [3.0956804068942843], \"foo\": 0.012719844107316677, \"foobar\": null, \"kLcKq0fkJ\": {\"I0N\": 2.989757793601628, \"c0aN7Br7R\": \"JGTX2\"}}, [null], null, 1.7277530981657518], [true], 0.40739084313422624, null]':::JSONB, (-25):::INT8, '2022-09-18 05:05:34.000126+00':::TIMESTAMPTZ, e'\\U000E14D2\\U0003E077\\U000CD34F\\U000236DD\\U000320CA\\U000908E1' COLLATE da_DK, B'0000000000000000000000000000000', 15:::INT8, e'\\'ZBIEbVpFer\\' \\'d\\' \\'s\\':328C,902A \\'sqIeGoG\\' \\'xUVViFu\\'':::TSVECTOR, 4:::INT8, '[[0.41843095885519865, true], true]':::JSONB, e'_\\x0b':::REFCURSOR)) AS t (\" col196_w0_200\", col196_w0_202, \"\\\\U000BF73Acol😘196_w0_206\", col196_w0_198, col196_w0_203, col196_w0_204, \"CoL19 6_w0_205\", \"col%v196_w0_215\", col196_w0_199, col196_w0_197)",
    "result": [
     [
      "-10"
     ]
    ]
   },
   {
    "query": "SELECT (false = ANY (ARRAY['-10'::INT8 = 15:::INT8, '-10'::INT8 = 15:::INT8, '-10'::INT8 = 15:::INT8])) AND (count(*) = 0) FROM public.table_w0_196 WHERE col_w0_196_w0_222 = (15:::INT8)",
    "result": true
   }
  ],
  "previousStatements": [
   "INSERT INTO public.table_w0_196 (col196_w0_197, col196_w0_198, col196_w0_199, \" col196_w0_200\", col196_w0_202, col196_w0_203, col196_w0_204, \"CoL19 6_w0_205\", \"\\\\U000BF73Acol😘196_w0_206\", \"col%v196_w0_215\") VALUES (e'_\\x0b':::REFCURSOR,e'\\U000E14D2\\U0003E077\\U000CD34F\\U000236DD\\U000320CA\\U000908E1' COLLATE da_DK,'[[0.41843095885519865, true], true]':::JSONB,'[[[[false, \"BiCOi\", true], \"c\"], [false, \"8Sd9Kgf\"], true], [{\"a\": \"9x\", \"c\": [3.0956804068942843], \"foo\": 0.012719844107316677, \"foobar\": null, \"kLcKq0fkJ\": {\"I0N\": 2.989757793601628, \"c0aN7Br7R\": \"JGTX2\"}}, [null], null, 1.7277530981657518], [true], 0.40739084313422624, null]':::JSONB,(-25):::INT8,B'0000000000000000000000000000000',15:::INT8,'''ZBIEbVpFer'' ''d'' ''s'':328C,902A ''sqIeGoG'' ''xUVViFu''':::TSVECTOR,'2022-09-18 05:05:34.000126+00':::TIMESTAMPTZ,4:::INT8),(e'\\'':::REFCURSOR,e'\\U000496B1\\U00020BEA' COLLATE da_DK,'{\"AU7Pb5M6H\": [[false, null], null, {\"EZVBLwf4A\": \"bar\", \"Gq9y\": null}, \"a\", true], \"TpP\": [{\"a\": true, \"foo\": false}, false, null], \"c\": \"bar\", \"foo\": true}':::JSONB,'[[null, null], [{\"bar\": true}, {\"Drqc\": 0.38081745397589417}, false, \"a\"], null, false, null, \"foo\"]':::JSONB,80:::INT8,B'0101110101010010110101000100101',70:::INT8,'''BCzLo'':336B ''BGoLAmne'' ''FHx'':221C,278C ''LxHaax'' ''OP'':93B,421B,584A ''TxTxc'' ''hQRwBCqK'':576C,860C ''n'':979A ''oOE'':491C,755C,788C,957C ''x'':12B,51B,100C,893A':::TSVECTOR,'2029-09-17 09:16:06.000427+00':::TIMESTAMPTZ,(-115):::INT8),('9S':::REFCURSOR,e'\\U000381BF\\U000CCEC5\\U0003262B\\U000048E2\\U0006950B\\U00042098' COLLATE da_DK,'true':::JSONB,NULL,98:::INT8,B'0000111101101100010011010100100',1:::INT8,'''EjAFoTxeXa'':44C,594C,640B ''d'' ''hZxePXWI'' ''kmKsfYIc'':276A,285A,538A ''pTloGnuXs'' ''qNxIv'':414B':::TSVECTOR,'1990-03-21 00:55:03.000029+00':::TIMESTAMPTZ,70:::INT8)"
  ]
 }
}
    schema_change_external_test.go:134: 
            Error Trace:    pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
            Error:          Received unexpected error:
                            ***FAIL; Failed to receive a commit error when at least one commit error was expected
                            (1) forced error mark
                              | "fatal error when running txn"
                              | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                            Wraps: (2)
                            Wraps: (3) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:672
                              | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
                              |     pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
                              | golang.org/x/sync/errgroup.(*Group).Go.func1
                              |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (4) ***FAIL; Failed to receive a commit error when at least one commit error was expected
                            Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.leafError
            Test:           TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 176
Total Schema Statements Succeeded = 176
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 100
Total Transactions Rolled Back = 89
Total Transactions Executed = 189
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/test.outputs/logTestWorkload3989242007
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload3989242007
--- FAIL: TestWorkload (38.42s)

Parameters:

attempt=1
run=1
shard=1
Help

Same failure on other branches

This test on roachdash | Improve this report!

rafiss commented 2 months ago

Let's fix the unique constraint issue by adding the error code (23505) to the potential errors field for the create unique index / add unique constraint operation. It should be added as a "commit error" not an "exec error."

cockroach-teamcity commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 7512f81be1591cde8e3b5d34bb0b9b1101f22d24:

    "result": true
   },
   {
    "query": "SELECT ARRAY[parent.table_schema, parent.table_name, parent.column_name, child.column_name] FROM (SELECT conname, conkey, confkey, conrelid, confrelid FROM pg_constraint WHERE (contype = 'f') AND (conrelid = 'schema_w0_66.table_w0_86'::REGCLASS::INT8)) AS con JOIN (SELECT column_name, ordinal_position, column_default FROM information_schema.columns WHERE (table_schema = 'schema_w0_66') AND (table_name = 'table_w0_86')) AS child ON conkey[1] = child.ordinal_position JOIN (SELECT pc.oid, cols.table_schema, cols.table_name, cols.column_name, cols.ordinal_position FROM pg_class AS pc JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid JOIN information_schema.columns AS cols ON ((pc.relname = cols.table_name) AND (pn.nspname = cols.table_schema))) AS parent ON ((con.confkey[1] = parent.ordinal_position) AND (con.confrelid = parent.oid)) WHERE child.column_name != 'rowid'",
    "result": null
   }
  ],
  "previousStatements": [
   "INSERT INTO schema_w0_66.table_w0_86 (\"col8-6_w0_87\", col86_w0_88) VALUES ('X'::NAME:::NAME,'o'::CHAR),(e'P\\x11GU'::NAME:::NAME,')'::CHAR)"
  ],
  "usesDeclarativeSchemaChanger": true
 }
}
    schema_change_external_test.go:134: 
            Error Trace:    pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
            Error:          Received unexpected error:
                            ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: duplicate key value violates unique constraint "table_w0_86_col86_w0_88_key" (SQLSTATE 23505)
                            (1) forced error mark
                              | "fatal error when running txn"
                              | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                            Wraps: (2)
                            Wraps: (3) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3139
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:499
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:587
                              | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
                              |     pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
                              | golang.org/x/sync/errgroup.(*Group).Go.func1
                              |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
                            Wraps: (5) ERROR: duplicate key value violates unique constraint "table_w0_86_col86_w0_88_key" (SQLSTATE 23505)
                            Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
            Test:           TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 88
Total Schema Statements Succeeded = 88
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 59
Total Transactions Rolled Back = 59
Total Transactions Executed = 118
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_1_of_25/test.outputs/logTestWorkload896690560
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload896690560
--- FAIL: TestWorkload (16.58s)

Parameters:

attempt=1
run=1
shard=1
Help

Same failure on other branches

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ d95aa79edb6685f9e1572ec609afa1a518674ad7:

   {
    "query": "SELECT EXISTS (SELECT * FROM public.table_w1_86)",
    "result": false
   },
   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT mut FROM (SELECT json_array_elements(d-\u003e'mutations') AS mut FROM descriptors) WHERE (mut-\u003e'primaryKeySwap') IS NOT NULL)",
    "queryArgs": [
     "public.table_w1_86"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "ALTER TABLE public.table_w1_86 ADD COLUMN col_w1_86_w1_331 INT4 NOT NULL"
  ]
 }
}
    schema_change_external_test.go:134: 
            Error Trace:    pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
            Error:          Received unexpected error:
                            ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23502): validation of column "col_w1_86_w1_331" NOT NULL failed on row: " c\\U00106265ol8\\u23A76_w1_87"=NULL, col86_w1_88=e'\x1d', col_w1_86_w1_331=NULL (SQLSTATE 23502)
                            (1) forced error mark
                              | "fatal error when running txn"
                              | github.com/cockroachdb/errors/withstack/*withstack.withStack::
                            Wraps: (2)
                            Wraps: (3) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
                              |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
                              | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
                              |     pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
                              | golang.org/x/sync/errgroup.(*Group).Go.func1
                              |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
                            Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23502): validation of column "col_w1_86_w1_331" NOT NULL failed on row: " c\\U00106265ol8\\u23A76_w1_87"=NULL, col86_w1_88=e'\x1d', col_w1_86_w1_331=NULL (SQLSTATE 23502)
                            Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
            Test:           TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 153
Total Schema Statements Succeeded = 153
Total Schema Statement Expected Failures = 1
Total Transactions Committed = 98
Total Transactions Rolled Back = 78
Total Transactions Executed = 176
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_7_of_25/test.outputs/logTestWorkload770264460
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload770264460
--- FAIL: TestWorkload (34.32s)

Parameters:

attempt=1
run=7
shard=1
Help

Same failure on other branches

This test on roachdash | Improve this report!

blathers-crl[bot] commented 2 months ago

Based on the specified backports for linked PR #130351, I applied the following new label(s) to this issue: branch-release-24.1, branch-release-24.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}