search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.15k stars 3.81k forks source link

roachtest: schemachange/random-load failed #78493

Closed cockroach-teamcity closed 2 years ago

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ 9cf10ddee27563508b759df66ce8685bdca07bfb:

          | I220325 12:47:10.150209 217 workload/pgx_helpers.go:79  [-] 550  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-west1" already added to database (SQLSTATE 42710) pid:1708094 sql:ALTER DATABASE schemachange ADD REGION "us-west1"]
          | I220325 12:47:10.208316 218 workload/pgx_helpers.go:79  [-] 551  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema2045.table2366" does not exist (SQLSTATE 42P01) pid:1391653 sql:ALTER TABLE schema2045.table2366 ALTER COLUMN "IrrelevantColumnName" DROP NOT NULL]
          | I220325 12:47:10.491804 214 workload/pgx_helpers.go:79  [-] 552  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot drop relation "table2338" because view "view2399" depends on it (SQLSTATE 2BP01) pid:1822751 sql:DROP TABLE schema1.table2338 RESTRICT]
          | I220325 12:47:10.505332 213 workload/pgx_helpers.go:79  [-] 553  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema1.table2410" does not exist (SQLSTATE 42P01) pid:1683956 sql:CREATE INDEX "IrrelevantName" ON schema1.table2410 ("IrrelevantColumn" ASC)]
          | I220325 12:47:10.712385 210 workload/pgx_helpers.go:79  [-] 554  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema1328.table2411" does not exist (SQLSTATE 42P01) pid:1056444 sql:DROP INDEX schema1328.table2411@"IrrelevantIndexName"]
          | I220325 12:47:10.712975 219 workload/pgx_helpers.go:79  [-] 555  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot drop schema "public" (SQLSTATE 3F000) pid:1513287 sql:DROP SCHEMA "public" CASCADE]
          | I220325 12:47:11.042586 226 workload/pgx_helpers.go:79  [-] 556  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema1257.table2414" does not exist (SQLSTATE 42P01) pid:1732232 sql:CREATE SEQUENCE IF NOT EXISTS schema768.seq2413 OWNED BY schema1257.table2414."IrrelevantColumnName"]
          | I220325 12:47:11.142487 216 workload/pgx_helpers.go:79  [-] 557  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: internal error: relation "view2399" (687): depends-on relation "table1495" (481) has no corresponding depended-on-by back reference (SQLSTATE XX000) pid:1331731 sql:DROP VIEW schema1.view2399]
          | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
          | Expected errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema1" "view2399"]] :true
          | QUERY ["\n\tSELECT EXISTS(\n        SELECT fd.descriptor_name\n          FROM crdb_internal.forward_dependencies AS fd\n         WHERE fd.descriptor_id\n               = (\n                    SELECT c.oid\n                      FROM pg_catalog.pg_class AS c\n                      JOIN pg_catalog.pg_namespace AS ns ON\n                            ns.oid = c.relnamespace\n                     WHERE c.relname = $1 AND ns.nspname = $2\n                )\n           AND fd.descriptor_id != fd.dependedonby_id\n           AND fd.dependedonby_type != 'sequence'\n       )\n\t" ["view2399" "schema1"]] :false
          | ===========================Previous statements [DROP VIEW schema1.view2399]: ERROR: internal error: relation "view2399" (687): depends-on relation "table1495" (481) has no corresponding depended-on-by back reference (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | \n                              FROM descriptors\\n                             WHERE (descriptor-\u003e'table') IS NOT NULL\\n                        ),\\n       dependent AS (\\n                    SELECT (ref-\u003e\u003e'id')::INT8 AS id FROM table_references\\n                 ),\\n       referenced_descriptors AS (\\n                                SELECT json_array_elements_text(\\n                                        descriptor-\u003e'type'-\u003e'referencingDescriptorIds'\\n                                       )::INT8 AS id\\n                                  FROM types\\n                              )\\nSELECT EXISTS(\\n        SELECT *\\n          FROM system.namespace\\n         WHERE id IN (SELECT id FROM referenced_descriptors)\\n           AND \\\"parentSchemaID\\\" NOT IN (SELECT id FROM schema_id)\\n           AND id NOT IN (SELECT id FROM dependent)\\n       );\" [\"public\"]] :false\n===========================Previous statements [ALTER VIEW schema1.view2399 RENAME TO schema1.view2408 DROP SCHEMA \"public\" CASCADE]: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:47:08.144504",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SEQUENCE IF NOT EXISTS schema768.seq2413 OWNED BY schema1257.table2414.\"IrrelevantColumnName\""
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42P01===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema768\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT sequence_name\\n    FROM information_schema.sequences\\n   WHERE sequence_schema = $1\\n     AND sequence_name = $2\\n   )\" [\"schema768\" \"seq2413\"]] :false\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema1257\" \"table2414\"]] :false\n===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema768.seq2413 OWNED BY schema1257.table2414.\"IrrelevantColumnName\"]: ERROR: relation \"schema1257.table2414\" does not exist (SQLSTATE 42P01)"
          | }
          |   190.0s        0            9.0            6.2     71.3   1409.3   1409.3   1409.3 opOk
          |   190.0s        0            1.0            1.1   7516.2   7516.2   7516.2   7516.2 txnCmtErr
          |   190.0s        0            1.0            0.4   2684.4   2684.4   2684.4   2684.4 txnOk
          |   190.0s        0            8.0            3.9      5.5     79.7     79.7     79.7 txnRbk
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:47:07.145531",
          |  "ops": [
          |   "BEGIN",
          |   "DROP VIEW schema1.view2399"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema1\" \"view2399\"]] :true\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"view2399\" \"schema1\"]] :false\n===========================Previous statements [DROP VIEW schema1.view2399]: ERROR: internal error: relation \"view2399\" (687): depends-on relation \"table1495\" (481) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #63518 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-63484] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-14156

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ 7fbd04334b4a5cd369d9df1afdcfa3f49707989d:

          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  = $1\\n     AND table_name = $2\\n   )\" [\"schema270\" \"table1311\"]] :false\n===========================Previous statements [CREATE TABLE schema270.table1311 AS SELECT schema270.table445.col445_452, schema270.table445.col445_451, schema270.table445.col445_446, schema270.table445.col445_454, schema270.table677.col677_682, schema270.table445.col445_456, schema270.table445.col445_454, schema270.table445.col445_448, schema270.table445.col445_452, schema270.table445.col445_447, schema270.table445.col445_453 FROM schema270.table445, schema270.table677, schema270.table445]: ERROR: source name \"table445\" specified more than once (missing AS clause) (SQLSTATE 42712)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:45:19.381601",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE",
          |   "DROP TABLE public.table988 RESTRICT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=6ac66d99 key=/Table/15/1/747858589922656257/0 pri=0.04025004 epo=1 ts=1648298739.245114491,1 min=1648298719.381539203,0 seq=27} lock=true stat=PENDING rts=1648298719.488989659,0 wto=false gul=1648298719.881539203,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:45:38.254826",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "ERROR: restart transaction: crdb_internal.validate_multi_region_zone_configs(): TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: read at time 1648298738.254743780,0 encountered previous write with future timestamp 1648298738.254743780,1 within uncertainty interval ``t \u003c= (local=1648298738.287253684,0, global=1648298738.754743780,0)``; observed timestamps: [{1 1648298738.254743780,0} {2 1648298738.287253684,0} {3 1648298738.299072860,0}]: \"sql txn\" meta={id=55a877bf key=/Min pri=0.04365202 epo=0 ts=1648298738.254743780,0 min=1648298738.254743780,0 seq=0} lock=false stat=PENDING rts=1648298738.254743780,0 wto=false gul=1648298738.754743780,0 (SQLSTATE 40001)"
          | }
          |   135.0s        0            3.0            4.6    486.5  10737.4  10737.4  10737.4 opOk
          |   135.0s        0            0.0            0.8      0.0      0.0      0.0      0.0 txnCmtErr
          |   135.0s        0            0.0            0.3      0.0      0.0      0.0      0.0 txnOk
          |   135.0s        0            4.0            3.3    469.8    738.2    738.2    738.2 txnRbk
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:45:36.781447",
          |  "ops": [
          |   "BEGIN",
          |   "DROP VIEW IF EXISTS public.view1272 RESTRICT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"view1272\"]] :true\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"view1272\" \"public\"]] :false\n===========================Previous statements [DROP VIEW IF EXISTS public.view1272 RESTRICT]: ERROR: internal error: relation \"view1272\" (520): depends-on relation \"table677\" (333) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #63518 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-63484] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ 1226d6f5d25b6bd41264da64fe7e7ec0db1848f2:

          |   "BEGIN",
          |   "CREATE TABLE schema8394.table8395 AS SELECT schema4945.table7810.col7810_7817, schema4945.table7810.col7810_7811, schema4945.table7810.col7810_7818, schema4945.table7810.col7810_7815, public.table4622.col4622_4627 FROM schema4945.table7810, public.table4622"
          |  ],
          |  "expectedExecErrors": "3F000",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 3F000===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema4945\" \"table7810\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table4622\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema8394\"]] :false\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema8394\" \"table8395\"]] :false\n===========================Previous statements [CREATE TABLE schema8394.table8395 AS SELECT schema4945.table7810.col7810_7817, schema4945.table7810.col7810_7811, schema4945.table7810.col7810_7818, schema4945.table7810.col7810_7815, public.table4622.col4622_4627 FROM schema4945.table7810, public.table4622]: ERROR: cannot create \"schema8394.table8395\" because the target database or schema does not exist (SQLSTATE 3F000)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:17:07.888467",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": ""
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:17:04.267682",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER VIEW schema8173.view8320 RENAME TO schema8173.view8378",
          |   "DROP TABLE schema6413.table7808",
          |   "DROP SCHEMA \"schema8173\" CASCADE"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): \"sql txn\" meta={id=6620424a key=/Table/3/1/2184/2/1 pri=0.01731738 epo=0 ts=1648387029.176493946,1 min=1648387024.267607481,0 seq=32} lock=true stat=ABORTED rts=1648387024.267607481,0 wto=false gul=1648387024.767607481,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:17:08.537488",
          |  "ops": [
          |   "BEGIN",
          |   "DROP VIEW IF EXISTS schema4945.view8379 RESTRICT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema4945\" \"view8379\"]] :true\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"view8379\" \"schema4945\"]] :false\n===========================Previous statements [DROP VIEW IF EXISTS schema4945.view8379 RESTRICT]: ERROR: internal error: relation \"view8379\" (2208): depends-on relation \"table7810\" (2058) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #63518 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-63484] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ b5e0ec1844e63c4dc43dc7a9b1605b37c270ecbe:

          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ts', crdb_internal.validate_multi_region_zone_configs()",
          |   "CREATE TABLE public.table988 (col988_989 REGCLASS NULL, col988_990 TIME NOT NULL, col988_991 FLOAT8 NOT NULL, col988_992 BOX2D, col988_993 BYTES NOT NULL, col988_994 JSONB, col988_995 INT8 NOT NULL, col988_996 TIMESTAMP NULL, col988_997 GEOMETRY NOT NULL, PRIMARY KEY (col988_990 ASC, col988_995 DESC, col988_993 DESC), UNIQUE (col988_991 DESC, col988_989 ASC) STORING (col988_992, col988_994, col988_997), INVERTED INDEX (col988_994 DESC), FAMILY (col988_991), FAMILY (col988_992, col988_993), FAMILY (col988_994), FAMILY (col988_995, col988_996), FAMILY (col988_989, col988_990, col988_997))",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: collecting namespace table entries: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): \"sql txn\" meta={id=883124a6 key=/NamespaceTable/30/1/104/105/\"table988\"/4/1 pri=0.01393813 epo=0 ts=1648471120.689090510,2 min=1648471118.320208465,0 seq=8} lock=true stat=ABORTED rts=1648471118.320208465,0 wto=false gul=1648471118.820208465,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:38:40.259699",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE INDEX \"IrrelevantName\" ON schema1026.table1027 (\"IrrelevantColumn\" ASC)"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42P01===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema1026\" \"table1027\"]] :false\n===========================Previous statements [CREATE INDEX \"IrrelevantName\" ON schema1026.table1027 (\"IrrelevantColumn\" ASC)]: ERROR: relation \"schema1026.table1027\" does not exist (SQLSTATE 42P01)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:38:39.673375",
          |  "ops": [
          |   "BEGIN",
          |   "DROP SEQUENCE schema585.seq702",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: intent on key /Table/3/1/296/2/1): \"sql txn\" meta={id=8769ec31 key=/NamespaceTable/30/1/104/268/\"seq702\"/4/1 pri=0.01674374 epo=0 ts=1648471120.659334209,2 min=1648471119.673298405,0 seq=14} lock=true stat=PENDING rts=1648471119.673298405,0 wto=false gul=1648471120.173298405,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:38:39.497246",
          |  "ops": [
          |   "BEGIN",
          |   "DROP VIEW public.view983 CASCADE"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"view983\"]] :true\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"view983\" \"public\"]] :false\n===========================Previous statements [DROP VIEW public.view983 CASCADE]: ERROR: internal error: relation \"view983\" (364): depends-on relation \"table194\" (168) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #63518 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-63484] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ fed021b0766c184d231b9a67e5c90d36e81f7c0a:

          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | scriptor-\u003e'table') IS NOT NULL\\n                        ),\\n       dependent AS (\\n                    SELECT (ref-\u003e\u003e'id')::INT8 AS id FROM table_references\\n                 ),\\n       referenced_descriptors AS (\\n                                SELECT json_array_elements_text(\\n                                        descriptor-\u003e'type'-\u003e'referencingDescriptorIds'\\n                                       )::INT8 AS id\\n                                  FROM types\\n                              )\\nSELECT EXISTS(\\n        SELECT *\\n          FROM system.namespace\\n         WHERE id IN (SELECT id FROM referenced_descriptors)\\n           AND \\\"parentSchemaID\\\" NOT IN (SELECT id FROM schema_id)\\n           AND id NOT IN (SELECT id FROM dependent)\\n       );\" [\"public\"]] :false\n===========================Previous statements [ALTER DATABASE schemachange SURVIVE ZONE FAILURE DROP SCHEMA \"public\" CASCADE]: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
          | }
          |    23.0s        0           10.0           11.6     71.3   1610.6   1610.6   1610.6 opOk
          |    23.0s        0            2.0            2.0   1409.3   1946.2   1946.2   1946.2 txnCmtErr
          |    23.0s        0            0.0            0.9      0.0      0.0      0.0      0.0 txnOk
          |    23.0s        0            4.0            9.8      0.9    402.7    402.7    402.7 txnRbk
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:45:56.450138",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\"",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": ""
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:45:59.251825",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SCHEMA IF NOT EXISTS schema630 AUTHORIZATION root",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: intent on key /Table/3/1/241/2/1): \"sql txn\" meta={id=510aad6a key=/Table/15/1/748708044831752193/0 pri=0.00255993 epo=0 ts=1648557963.943964940,1 min=1648557959.251740108,0 seq=15} lock=true stat=PENDING rts=1648557959.251740108,0 wto=false gul=1648557959.751740108,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:46:02.371217",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER VIEW public.view638 RENAME TO public.view680"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.views \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"view638\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"public\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.views \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"view680\"]] :false\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"view638\" \"public\"]] :false\n===========================Previous statements [ALTER VIEW public.view638 RENAME TO public.view680]: ERROR: internal error: relation \"view638\" (258): depends-on relation \"table581\" (241) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #63518 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-63484] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

This test on roachdash | Improve this report!

ajwerner commented 2 years ago

@fqazi do you understand the source of the failure here?

fqazi commented 2 years ago

Still trying to root the cause, but we are missing a backreference from the table to the view. I don't have a full root cause yet :(

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ ff4e23e53388d3639f4b1b23b66e87d436556a8f:

          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:2020
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:160
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:866
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_120713.935380893_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | =map[args:[] err:ERROR: relation "schemachange.schema5765.table6463" already exists (SQLSTATE 42P07) pid:1642916 sql:CREATE TABLE schema5765.table6463 AS SELECT schema3406.table5822.col5822_5836, schema3406.table5822.col5822_5825, schema3406.table5822.col5822_5828, schema3406.table5822.col5822_5835, schema3406.table5822.col5822_5834, schema3406.table5822.col5822_5830, schema3406.table5822.col5822_5826, schema5765.table6463.col6463_6464, schema5765.table6463.col6463_6465 FROM schema3406.table5822, schema5765.table6463]
          | I220330 12:33:30.836479 219 workload/pgx_helpers.go:79  [-] 1597  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema5297.table6821" does not exist (SQLSTATE 42P01) pid:1688315 sql:CREATE TABLE schema2964.table6823 AS SELECT schema5765.table6463.col6463_6464, schema5765.table6463.col6463_6465, "IrrelevantColumnName" FROM schema5765.table6463, schema5297.table6821]
          | I220330 12:33:32.135854 217 workload/pgx_helpers.go:79  [-] 1598  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "europe-west2" already added to database (SQLSTATE 42710) pid:1741689 sql:ALTER DATABASE schemachange ADD REGION "europe-west2"]
          | I220330 12:33:33.019666 216 workload/pgx_helpers.go:79  [-] 1599  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: internal error: relation "view6789" (1744): depends-on relation "table6463" (1675) has no corresponding depended-on-by back reference (SQLSTATE XX000) pid:2081735 sql:ALTER VIEW schema5765.view6789 RENAME TO schema5765.view6829]
          | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
          | Expected errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.views \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema5765" "view6789"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT schema_name\n\t\tFROM information_schema.schemata\n   WHERE schema_name = $1\n\t)" ["schema5765"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.views \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema5765" "view6829"]] :false
          | QUERY ["\n\tSELECT EXISTS(\n        SELECT fd.descriptor_name\n          FROM crdb_internal.forward_dependencies AS fd\n         WHERE fd.descriptor_id\n               = (\n                    SELECT c.oid\n                      FROM pg_catalog.pg_class AS c\n                      JOIN pg_catalog.pg_namespace AS ns ON\n                            ns.oid = c.relnamespace\n                     WHERE c.relname = $1 AND ns.nspname = $2\n                )\n           AND fd.descriptor_id != fd.dependedonby_id\n           AND fd.dependedonby_type != 'sequence'\n       )\n\t" ["view6789" "schema5765"]] :false
          | ===========================Previous statements [CREATE TABLE IF NOT EXISTS schema3793.table6806 (col6806_6807 REGCLASS NOT NULL, col6806_6808 TIMETZ NOT NULL, col6806_6809 INT4, col6806_6810 GEOMETRY NOT NULL, col6806_6811 GEOMETRY, col6806_6812 UUID, col6806_6813 GEOMETRY NULL, col6806_6814 STRING AS (lower(CAST(col6806_6811 AS STRING))) VIRTUAL, col6806_6815 STRING NOT NULL AS (lower(CAST(col6806_6810 AS STRING))) STORED, col6806_6816 STRING NOT NULL AS (lower(CAST(col6806_6810 AS STRING))) STORED, col6806_6817 STRING AS (lower(CAST(col6806_6812 AS STRING))) STORED, col6806_6818 STRING NOT NULL AS (CASE WHEN col6806_6807 IS NULL THEN e'IT\n5\x0f':::STRING ELSE '':::STRING END) STORED, col6806_6819 STRING NOT NULL AS (lower(CAST(col6806_6808 AS STRING))) STORED, FAMILY (col6806_6816), FAMILY (col6806_6812), FAMILY (col6806_6807, col6806_6809), FAMILY (col6806_6818), FAMILY (col6806_6819), FAMILY (col6806_6817), FAMILY (col6806_6813, col6806_6810), FAMILY (col6806_6815, col6806_6811, col6806_6808)) ALTER DATABASE schemachange PRIMARY REGION "us-east1" ALTER VIEW schema5765.view6789 RENAME TO schema5765.view6829]: ERROR: internal error: relation "view6789" (1744): depends-on relation "table6463" (1675) has no corresponding depended-on-by back reference (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 48988696588156929/0 pri=0.01190705 epo=0 ts=1648643612.630089619,0 min=1648643610.485885276,0 seq=15} lock=true stat=ABORTED rts=1648643610.485885276,0 wto=false gul=1648643610.985885276,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:33:27.482237",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE TABLE IF NOT EXISTS schema3793.table6806 (col6806_6807 REGCLASS NOT NULL, col6806_6808 TIMETZ NOT NULL, col6806_6809 INT4, col6806_6810 GEOMETRY NOT NULL, col6806_6811 GEOMETRY, col6806_6812 UUID, col6806_6813 GEOMETRY NULL, col6806_6814 STRING AS (lower(CAST(col6806_6811 AS STRING))) VIRTUAL, col6806_6815 STRING NOT NULL AS (lower(CAST(col6806_6810 AS STRING))) STORED, col6806_6816 STRING NOT NULL AS (lower(CAST(col6806_6810 AS STRING))) STORED, col6806_6817 STRING AS (lower(CAST(col6806_6812 AS STRING))) STORED, col6806_6818 STRING NOT NULL AS (CASE WHEN col6806_6807 IS NULL THEN e'IT\\n5\\x0f':::STRING ELSE '':::STRING END) STORED, col6806_6819 STRING NOT NULL AS (lower(CAST(col6806_6808 AS STRING))) STORED, FAMILY (col6806_6816), FAMILY (col6806_6812), FAMILY (col6806_6807, col6806_6809), FAMILY (col6806_6818), FAMILY (col6806_6819), FAMILY (col6806_6817), FAMILY (col6806_6813, col6806_6810), FAMILY (col6806_6815, col6806_6811, col6806_6808))",
          |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
          |   "ALTER VIEW schema5765.view6789 RENAME TO schema5765.view6829"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.views \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema5765\" \"view6789\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema5765\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.views \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema5765\" \"view6829\"]] :false\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"view6789\" \"schema5765\"]] :false\n===========================Previous statements [CREATE TABLE IF NOT EXISTS schema3793.table6806 (col6806_6807 REGCLASS NOT NULL, col6806_6808 TIMETZ NOT NULL, col6806_6809 INT4, col6806_6810 GEOMETRY NOT NULL, col6806_6811 GEOMETRY, col6806_6812 UUID, col6806_6813 GEOMETRY NULL, col6806_6814 STRING AS (lower(CAST(col6806_6811 AS STRING))) VIRTUAL, col6806_6815 STRING NOT NULL AS (lower(CAST(col6806_6810 AS STRING))) STORED, col6806_6816 STRING NOT NULL AS (lower(CAST(col6806_6810 AS STRING))) STORED, col6806_6817 STRING AS (lower(CAST(col6806_6812 AS STRING))) STORED, col6806_6818 STRING NOT NULL AS (CASE WHEN col6806_6807 IS NULL THEN e'IT\\n5\\x0f':::STRING ELSE '':::STRING END) STORED, col6806_6819 STRING NOT NULL AS (lower(CAST(col6806_6808 AS STRING))) STORED, FAMILY (col6806_6816), FAMILY (col6806_6812), FAMILY (col6806_6807, col6806_6809), FAMILY (col6806_6818), FAMILY (col6806_6819), FAMILY (col6806_6817), FAMILY (col6806_6813, col6806_6810), FAMILY (col6806_6815, col6806_6811, col6806_6808)) ALTER DATABASE schemachange PRIMARY REGION \"us-east1\" ALTER VIEW schema5765.view6789 RENAME TO schema5765.view6829]: ERROR: internal error: relation \"view6789\" (1744): depends-on relation \"table6463\" (1675) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ a88367d7bd168a3614b7946cd7b77bb9c6374a30:

        Wraps: (2) output in run_123351.822425814_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | re than once (missing AS clause) (SQLSTATE 42712) pid:1901873 sql:CREATE TABLE schema1409.table3686 AS SELECT schema1409.table3286.col3286_3290, schema1409.table3286.col3286_3287, schema1409.table3286.col3286_3288, schema1409.table3286.col3286_3293, schema1409.table3286.col3286_3296, schema1409.table3286.col3286_3287, schema1409.table3286.col3286_3292, schema1409.table3286.col3286_3289, schema1409.table3286.col3286_3294, schema1409.table3286.col3286_3288 FROM schema1409.table3286, schema1409.table3286]
          | I220331 12:43:56.163027 218 workload/pgx_helpers.go:79  [-] 876  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "public.table3662" does not exist (SQLSTATE 42P01) pid:1130251 sql:ALTER TABLE public.table3662 ALTER COLUMN "IrrelevantColumnName" DROP NOT NULL]
          | I220331 12:43:56.202812 214 workload/pgx_helpers.go:79  [-] 877  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-east1" already added to database (SQLSTATE 42710) pid:2071879 sql:ALTER DATABASE schemachange ADD REGION "us-east1"]
          | I220331 12:43:56.522899 213 workload/pgx_helpers.go:79  [-] 878  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema2865.table3684" does not exist (SQLSTATE 42P01) pid:1808857 sql:ALTER TABLE schema2865.table3684 ADD COLUMN IrrelevantColumnName string]
          | I220331 12:43:56.619613 216 workload/pgx_helpers.go:79  [-] 879  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "public.table3698" does not exist (SQLSTATE 42P01) pid:2026070 sql:ALTER TABLE public.table3698 RENAME TO public.table3699]
          | I220331 12:43:56.964187 217 workload/pgx_helpers.go:79  [-] 880  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "public.table3702" does not exist (SQLSTATE 42P01) pid:1996346 sql:ALTER INDEX public.table3702@"IrrelevantConstraintName" RENAME TO "OtherConstraintName"]
          | I220331 12:43:57.559350 171 workload/pgx_helpers.go:79  [-] 881  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: source name "table3504" specified more than once (missing AS clause) (SQLSTATE 42712) pid:1598010 sql:CREATE TABLE schema1409.table3703 AS SELECT public.table3504.col3504_3520, public.table3504.col3504_3519, public.table3504.col3504_3517, public.table3504.col3504_3521, public.table3504.col3504_3514, public.table3504.col3504_3515, public.table3504.col3504_3508, public.table3504.col3504_3513, public.table3504.col3504_3515, public.table3504.col3504_3514, public.table3504.col3504_3510 FROM public.table3504, public.table3504]
          | I220331 12:43:58.428092 173 workload/pgx_helpers.go:79  [-] 882  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: internal error: relation "view3685" (1041): depends-on relation "table3504" (1008) has no corresponding depended-on-by back reference (SQLSTATE XX000) pid:1447066 sql:DROP VIEW schema2865.view3685]
          | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
          | Expected errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema2865" "view3685"]] :true
          | QUERY ["\n\tSELECT EXISTS(\n        SELECT fd.descriptor_name\n          FROM crdb_internal.forward_dependencies AS fd\n         WHERE fd.descriptor_id\n               = (\n                    SELECT c.oid\n                      FROM pg_catalog.pg_class AS c\n                      JOIN pg_catalog.pg_namespace AS ns ON\n                            ns.oid = c.relnamespace\n                     WHERE c.relname = $1 AND ns.nspname = $2\n                )\n           AND fd.descriptor_id != fd.dependedonby_id\n           AND fd.dependedonby_type != 'sequence'\n       )\n\t" ["view3685" "schema2865"]] :false
          | ===========================Previous statements [ALTER SEQUENCE schema1409.seq3282 RENAME TO schema1409.seq3688 DROP VIEW schema2865.view3685]: ERROR: internal error: relation "view3685" (1041): depends-on relation "table3504" (1008) has no corresponding depended-on-by back reference (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | TABASE schemachange PRIMARY REGION \"europe-west2\"",
          |   "CREATE TABLE schema1409.table3703 AS SELECT public.table3504.col3504_3520, public.table3504.col3504_3519, public.table3504.col3504_3517, public.table3504.col3504_3521, public.table3504.col3504_3514, public.table3504.col3504_3515, public.table3504.col3504_3508, public.table3504.col3504_3513, public.table3504.col3504_3515, public.table3504.col3504_3514, public.table3504.col3504_3510 FROM public.table3504, public.table3504"
          |  ],
          |  "expectedExecErrors": "42701,42712",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42701,42712===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table3504\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table3504\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema1409\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema1409\" \"table3703\"]] :false\n===========================Previous statements [ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\" CREATE TABLE schema1409.table3703 AS SELECT public.table3504.col3504_3520, public.table3504.col3504_3519, public.table3504.col3504_3517, public.table3504.col3504_3521, public.table3504.col3504_3514, public.table3504.col3504_3515, public.table3504.col3504_3508, public.table3504.col3504_3513, public.table3504.col3504_3515, public.table3504.col3504_3514, public.table3504.col3504_3510 FROM public.table3504, public.table3504]: ERROR: source name \"table3504\" specified more than once (missing AS clause) (SQLSTATE 42712)"
          | }
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |   606.0s        0            2.0            2.7    453.0   6442.5   6442.5   6442.5 opOk
          |   606.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnCmtErr
          |   606.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnOk
          |   606.0s        0            1.0            2.0    184.5    184.5    184.5    184.5 txnRbk
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:43:53.515115",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER SEQUENCE schema1409.seq3282 RENAME TO schema1409.seq3688",
          |   "DROP VIEW schema2865.view3685"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema2865\" \"view3685\"]] :true\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"view3685\" \"schema2865\"]] :false\n===========================Previous statements [ALTER SEQUENCE schema1409.seq3282 RENAME TO schema1409.seq3688 DROP VIEW schema2865.view3685]: ERROR: internal error: relation \"view3685\" (1041): depends-on relation \"table3504\" (1008) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ 6a34ce7658342f4de148e5f23e986f21f45a25cd:

          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_120931.676449259_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | p[args:[] err:ERROR: cannot drop relation "table6740" because view "view9977" depends on it (SQLSTATE 2BP01) pid:1682825 sql:DROP TABLE IF EXISTS schema3882.table6740 RESTRICT]
          | I220403 12:30:34.613908 188 workload/pgx_helpers.go:79  [-] 2216  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-east1" already added to database (SQLSTATE 42710) pid:2024699 sql:ALTER DATABASE schemachange ADD REGION "us-east1"]
          | I220403 12:30:34.764145 232 workload/pgx_helpers.go:79  [-] 2217  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot create "schema10012.view10013" because the target database or schema does not exist (SQLSTATE 3F000) pid:1773681 sql:CREATE VIEW schema10012.view10013 AS SELECT schema4327.table8076.col6192_6195, schema4327.table8076.col7569_7587, schema4327.table8076.col6457_6469, schema4327.table8076.col6457_6466, schema4327.table8076.col6192_6193, schema4327.table8076.col6457_6460, schema4327.table8076.col6457_6461, schema4327.table8076.col7569_7576, schema4327.table8076.col6457_6470, schema4327.table8076.col6457_6462, schema4327.table8076.col6192_6197, schema4327.table8076.col6457_6463, schema4327.table8076.col6457_6459, schema4327.table8076.col6457_6471, schema4327.table8076.col6457_6464, schema4327.table8076.col6457_6472, schema4327.table8076.col6457_6458, schema4327.table8076.col6457_6468, schema4327.table8076.col6457_6465, schema4327.table8076.col6457_6468, schema4327.table8076.col6457_6472, schema4327.table8076.col6457_6458, schema4327.table8076.col6457_6459, schema4327.table8076.col6457_6470, schema4327.table8076.col6457_6462, schema480.table6585.col6192_6193 FROM schema4327.table8076, schema4327.table8076, schema480.table6585]
          | I220403 12:30:35.870176 191 workload/pgx_helpers.go:79  [-] 2218  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: internal error: error resolving dependent view ID 2559: relation "view9977" (2559): depends-on relation "table8887" (2296) has no corresponding depended-on-by back reference (SQLSTATE XX000) pid:2060252 sql:DROP TABLE IF EXISTS schema480.table6585]
          | I220403 12:30:35.907036 234 workload/pgx_helpers.go:79  [-] 2219  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schemachange.schema3882.table6740" already exists (SQLSTATE 42P07) pid:1704579 sql:CREATE TABLE schema3882.table6740 AS SELECT schema4327.table8076.col6457_6460, schema4327.table8076.col6192_6195, schema4327.table8076.col6457_6458, schema4327.table8076.col6457_6468, schema4327.table8076.col7569_7587, schema4327.table8076.col6457_6461, schema4327.table8076.col6457_6464, schema4327.table8076.col6457_6463, schema4327.table8076.col6457_6459, schema4327.table8076.col6457_6469, schema4327.table8076.col6457_6470, schema4327.table8076.col6457_6465 FROM schema4327.table8076]
          | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
          | Expected errors: 2BP01===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema480" "table6585"]] :true
          | QUERY ["\n\tSELECT EXISTS(\n        SELECT fd.descriptor_name\n          FROM crdb_internal.forward_dependencies AS fd\n         WHERE fd.descriptor_id\n               = (\n                    SELECT c.oid\n                      FROM pg_catalog.pg_class AS c\n                      JOIN pg_catalog.pg_namespace AS ns ON\n                            ns.oid = c.relnamespace\n                     WHERE c.relname = $1 AND ns.nspname = $2\n                )\n           AND fd.descriptor_id != fd.dependedonby_id\n           AND fd.dependedonby_type != 'sequence'\n       )\n\t" ["table6585" "schema480"]] :true
          | ===========================Previous statements [ALTER DATABASE schemachange PRIMARY REGION "europe-west2" DROP TABLE IF EXISTS schema480.table6585]: ERROR: internal error: error resolving dependent view ID 2559: relation "view9977" (2559): depends-on relation "table8887" (2296) has no corresponding depended-on-by back reference (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  stat=PENDING rts=1648989031.011548255,0 wto=false gul=1648989031.511548255,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:30:33.846732",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE TABLE schema29.table10034 AS SELECT schema5757.table7569.col7569_7579, schema5757.table7569.col7569_7572, schema3882.table6740.col6740_6744 FROM schema5757.table7569, schema3882.table6740",
          |   "CREATE TABLE schema3882.table6740 AS SELECT schema4327.table8076.col6457_6460, schema4327.table8076.col6192_6195, schema4327.table8076.col6457_6458, schema4327.table8076.col6457_6468, schema4327.table8076.col7569_7587, schema4327.table8076.col6457_6461, schema4327.table8076.col6457_6464, schema4327.table8076.col6457_6463, schema4327.table8076.col6457_6459, schema4327.table8076.col6457_6469, schema4327.table8076.col6457_6470, schema4327.table8076.col6457_6465 FROM schema4327.table8076"
          |  ],
          |  "expectedExecErrors": "42P07",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42P07===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema4327\" \"table8076\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema3882\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3882\" \"table6740\"]] :true\n===========================Previous statements [CREATE TABLE schema29.table10034 AS SELECT schema5757.table7569.col7569_7579, schema5757.table7569.col7569_7572, schema3882.table6740.col6740_6744 FROM schema5757.table7569, schema3882.table6740 CREATE TABLE schema3882.table6740 AS SELECT schema4327.table8076.col6457_6460, schema4327.table8076.col6192_6195, schema4327.table8076.col6457_6458, schema4327.table8076.col6457_6468, schema4327.table8076.col7569_7587, schema4327.table8076.col6457_6461, schema4327.table8076.col6457_6464, schema4327.table8076.col6457_6463, schema4327.table8076.col6457_6459, schema4327.table8076.col6457_6469, schema4327.table8076.col6457_6470, schema4327.table8076.col6457_6465 FROM schema4327.table8076]: ERROR: relation \"schemachange.schema3882.table6740\" already exists (SQLSTATE 42P07)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:30:34.047118",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"",
          |   "DROP TABLE IF EXISTS schema480.table6585"
          |  ],
          |  "expectedExecErrors": "2BP01",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: 2BP01===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema480\" \"table6585\"]] :true\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"table6585\" \"schema480\"]] :true\n===========================Previous statements [ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\" DROP TABLE IF EXISTS schema480.table6585]: ERROR: internal error: error resolving dependent view ID 2559: relation \"view9977\" (2559): depends-on relation \"table8887\" (2296) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on release-22.1 @ 1ec23d5938e3e0a2dc4deff3fc05c2ea6b33cd80:

          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | .table4916, schema2500.table4646]
          | I220406 12:23:53.336260 191 workload/pgx_helpers.go:79  [-] 1168  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-west1" already added to database (SQLSTATE 42710) pid:2084787 sql:ALTER DATABASE schemachange ADD REGION "us-west1"]
          | I220406 12:23:54.883622 217 workload/pgx_helpers.go:79  [-] 1169  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema1526.table4919" does not exist (SQLSTATE 42P01) pid:1412958 sql:CREATE TABLE schema2500.table4920 AS SELECT "IrrelevantColumnName" FROM schema1526.table4919]
          | I220406 12:23:55.599156 214 workload/pgx_helpers.go:79  [-] 1170  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema1526.table4910" does not exist (SQLSTATE 42P01) pid:1808047 sql:ALTER INDEX schema1526.table4910@"IrrelevantConstraintName" RENAME TO "OtherConstraintName"]
          | I220406 12:23:55.607890 212 workload/pgx_helpers.go:79  [-] 1171  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema2500.seq4908" does not exist (SQLSTATE 42P01) pid:1099908 sql:ALTER SEQUENCE schema2500.seq4908 RENAME TO schema2500.seq4909]
          | I220406 12:23:55.659892 219 workload/pgx_helpers.go:79  [-] 1172  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-east1" already added to database (SQLSTATE 42710) pid:1730612 sql:ALTER DATABASE schemachange ADD REGION "us-east1"]
          | I220406 12:23:55.790314 187 workload/pgx_helpers.go:79  [-] 1173  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot create "schema4944.enum4945" because the target database or schema does not exist (SQLSTATE 3F000) pid:1151616 sql:CREATE TYPE schema4944.enum4945 AS ENUM ('fda', 'afffs', 'fsad')]
          | I220406 12:23:55.800519 184 workload/pgx_helpers.go:79  [-] 1174  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot create "schema4946.view4947" because the target database or schema does not exist (SQLSTATE 3F000) pid:1140688 sql:CREATE VIEW schema4946.view4947 AS SELECT schema920.table1583.col1583_1590, schema920.table1583.col1583_1588, schema920.table1583.col1583_1595, schema920.table1583.col1583_1599, schema920.table1583.col1583_1586, schema920.table1583.col1583_1600, schema920.table1583.col1583_1589, schema920.table1583.col1583_1585, schema920.table977.col977_978, schema920.table977.col977_980, schema2500.table4646.col4646_4649, schema2500.table4646.col4646_4648 FROM schema920.table1583, schema920.table977, schema2500.table4646]
          | I220406 12:23:56.067200 188 workload/pgx_helpers.go:79  [-] 1175  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: internal error: relation "view4914" (1452): depends-on relation "table4646" (1381) has no corresponding depended-on-by back reference (SQLSTATE XX000) pid:1462025 sql:DROP VIEW schema1526.view4914 CASCADE]
          | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
          | Expected errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema1526" "view4914"]] :true
          | QUERY ["\n\tSELECT EXISTS(\n        SELECT fd.descriptor_name\n          FROM crdb_internal.forward_dependencies AS fd\n         WHERE fd.descriptor_id\n               = (\n                    SELECT c.oid\n                      FROM pg_catalog.pg_class AS c\n                      JOIN pg_catalog.pg_namespace AS ns ON\n                            ns.oid = c.relnamespace\n                     WHERE c.relname = $1 AND ns.nspname = $2\n                )\n           AND fd.descriptor_id != fd.dependedonby_id\n           AND fd.dependedonby_type != 'sequence'\n       )\n\t" ["view4914" "schema1526"]] :false
          | ===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() DROP VIEW schema1526.view4914 CASCADE]: ERROR: internal error: relation "view4914" (1452): depends-on relation "table4646" (1381) has no corresponding depended-on-by back reference (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | se or schema does not exist (SQLSTATE 3F000)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:23:47.316166",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE VIEW schema4946.view4947 AS SELECT schema920.table1583.col1583_1590, schema920.table1583.col1583_1588, schema920.table1583.col1583_1595, schema920.table1583.col1583_1599, schema920.table1583.col1583_1586, schema920.table1583.col1583_1600, schema920.table1583.col1583_1589, schema920.table1583.col1583_1585, schema920.table977.col977_978, schema920.table977.col977_980, schema2500.table4646.col4646_4649, schema2500.table4646.col4646_4648 FROM schema920.table1583, schema920.table977, schema2500.table4646"
          |  ],
          |  "expectedExecErrors": "3F000",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 3F000===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema920\" \"table1583\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema920\" \"table977\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema2500\" \"table4646\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema4946\"]] :false\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.views \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema4946\" \"view4947\"]] :false\n===========================Previous statements [CREATE VIEW schema4946.view4947 AS SELECT schema920.table1583.col1583_1590, schema920.table1583.col1583_1588, schema920.table1583.col1583_1595, schema920.table1583.col1583_1599, schema920.table1583.col1583_1586, schema920.table1583.col1583_1600, schema920.table1583.col1583_1589, schema920.table1583.col1583_1585, schema920.table977.col977_978, schema920.table977.col977_980, schema2500.table4646.col4646_4649, schema2500.table4646.col4646_4648 FROM schema920.table1583, schema920.table977, schema2500.table4646]: ERROR: cannot create \"schema4946.view4947\" because the target database or schema does not exist (SQLSTATE 3F000)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "12:23:52.338883",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
          |   "DROP VIEW schema1526.view4914 CASCADE"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema1526\" \"view4914\"]] :true\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"view4914\" \"schema1526\"]] :false\n===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() DROP VIEW schema1526.view4914 CASCADE]: ERROR: internal error: relation \"view4914\" (1452): depends-on relation \"table4646\" (1381) has no corresponding depended-on-by back reference (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2] - #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1] - #56235 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master]

This test on roachdash | Improve this report!

fqazi commented 2 years ago

This seems to resolved, no new failures observed