search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

roachtest: schemachange/random-load failed #82133

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 662cc5c3070e6d64d155a9cc9f33253ee5d99ee9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:884: output in run_124321.258348568_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1947
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:884
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_124321.258348568_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 09FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);" []] :false
          | QUERY ["SELECT (((col8964_8969, col8964_8968))::STRING) IS NULL AS c FROM ( VALUES('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);" []] :false
          | QUERY ["SELECT (((col8964_8969))::STRING) IS NULL AS c FROM ( VALUES(NULL,'0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY)) AS t(col8964_8969,col8964_8968);" []] :true
          | QUERY ["\n    WITH tab_json AS (\n                    SELECT crdb_internal.pb_to_json(\n                            'desc',\n                            descriptor\n                           )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema4785.table8964"]] :[["(col8964_8969)"] ["(CASE WHEN col8964_8969 IS NULL THEN e'\\'':::STRING ELSE e'\\x1e!#':::STRING END)"] ["(col8964_8969)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema4785.table8964'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema4785'\n\t\t           AND table_name = 'table8964'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [INSERT INTO schema4785.table8964 (col8964_8968,col8964_8969) VALUES ('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)]: ERROR: duplicate key value violates unique constraint "table8964_expr_key" (SQLSTATE 23505)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 2C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);\" []] :false\nQUERY [\"SELECT (((col8964_8969))::STRING) IS NULL AS c FROM ( VALUES(NULL,'0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY)) AS t(col8964_8969,col8964_8968);\" []] :true\nQUERY [\"\\n    WITH tab_json AS (\\n                    SELECT crdb_internal.pb_to_json(\\n                            'desc',\\n                            descriptor\\n                           )-\u003e'table' AS t\\n                      FROM system.descriptor\\n                     WHERE id = $1::REGCLASS\\n                  ),\\n         columns_json AS (\\n                        SELECT json_array_elements(t-\u003e'columns') AS c FROM tab_json\\n                      ),\\n         columns AS (\\n                    SELECT (c-\u003e\u003e'id')::INT8 AS col_id,\\n                           IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema4785.table8964\"]] :[[\"(col8964_8969)\"] [\"(CASE WHEN col8964_8969 IS NULL THEN e'\\\\'':::STRING ELSE e'\\\\x1e!#':::STRING END)\"] [\"(col8964_8969)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema4785.table8964'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema4785'\\n\\t\\t           AND table_name = 'table8964'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO schema4785.table8964 (col8964_8968,col8964_8969) VALUES ('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)]: ERROR: duplicate key value violates unique constraint \"table8964_expr_key\" (SQLSTATE 23505)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-16222

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 2c08debd6d8c89019f26a3e58cf3b5d3d97b6495:

          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1948
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:884
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_131707.590160241_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ema5510.seq5511" because the target database or schema does not exist (SQLSTATE 3F000) pid:1700748 sql:CREATE SEQUENCE IF NOT EXISTS schema5510.seq5511 time:474.303µs]
          | I220606 13:34:50.243786 129 workload/pgx_helpers.go:79  [-] 1344  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-west1" already added to database (SQLSTATE 42710) pid:1291274 sql:ALTER DATABASE schemachange ADD REGION "us-west1" time:10.037632ms]
          | I220606 13:34:50.274010 212 workload/pgx_helpers.go:79  [-] 1345  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema5513.table5547" does not exist (SQLSTATE 42P01) pid:1393661 sql:ALTER TABLE schema5513.table5547 DROP CONSTRAINT IrrelevantConstraintName time:88.671403ms]
          | I220606 13:34:50.285571 211 workload/pgx_helpers.go:79  [-] 1346  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema1129.table5506" does not exist (SQLSTATE 42P01) pid:1781910 sql:CREATE TABLE schema4074.table5549 AS SELECT "IrrelevantColumnName" FROM schema1129.table5506 time:2.147877ms]
          | I220606 13:34:50.778757 221 workload/pgx_helpers.go:79  [-] 1347  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot create "schema5555.table5556" because the target database or schema does not exist (SQLSTATE 3F000) pid:1694075 sql:CREATE TABLE schema5555.table5556 AS SELECT public.table907.col907_911, public.table907.col907_916, public.table907.col907_918, public.table907.col907_917, public.table907.col907_910, public.table907.col907_908, public.table907.col907_915, public.table907.col907_909, public.table907.col907_919, public.table907.col907_914, schema3441.table4899.col907_910 FROM public.table907, schema3441.table4899 time:826.696µs]
          | I220606 13:34:51.287192 228 workload/pgx_helpers.go:79  [-] 1348  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schemachange.schema4074.view5371" already exists (SQLSTATE 42P07) pid:1091750 sql:CREATE VIEW schema4074.view5371 AS SELECT schema3224.table5225.col5225_5235, schema3224.table5225.col5225_5229, schema3224.table5225.col5225_5231, schema3224.table5225.col5225_5238, schema3224.table5225.col5225_5228, schema3224.table5225.col5225_5239, schema3224.table5225.col5225_5243 FROM schema3224.table5225 time:3.018861ms]
          | I220606 13:34:51.492491 225 workload/pgx_helpers.go:79  [-] 1349  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot change schema of table with RENAME (SQLSTATE 42602) pid:1191175 sql:ALTER SEQUENCE schema5048.seq5471 RENAME TO public.seq5561 time:60.185157ms]
          | I220606 13:34:52.023311 217 workload/pgx_helpers.go:79  [-] 1350  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-east1" already added to database (SQLSTATE 42710) pid:2075137 sql:ALTER DATABASE schemachange ADD REGION "us-east1" time:2.901426ms]
          | I220606 13:34:52.308661 210 workload/pgx_helpers.go:79  [-] 1351  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema4812.table5559" does not exist (SQLSTATE 42P01) pid:1700748 sql:CREATE INDEX "IrrelevantName" ON schema4812.table5559 ("IrrelevantColumn" ASC) time:41.18307ms]
          | I220606 13:34:52.330158 211 workload/pgx_helpers.go:79  [-] 1352  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema4812.table5558" does not exist (SQLSTATE 42P01) pid:1781910 sql:ALTER TABLE schema4812.table5558 ALTER COLUMN IrrelevantColumnName SET DEFAULT "IrrelevantValue" time:39.693446ms]
          | I220606 13:34:52.362434 224 workload/pgx_helpers.go:79  [-] 1353  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3224.seq5560" does not exist (SQLSTATE 42P01) pid:2090407 sql:DROP SEQUENCE schema3224.seq5560 time:7.329099ms]
          | I220606 13:34:52.618296 214 workload/pgx_helpers.go:79  [-] 1354  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "seq5548" (1453): expected matching namespace entry, found none (SQLSTATE XXUUU) pid:1792622 sql:commit time:221.080649ms]
          | Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: relation "seq5548" (1453): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ntTimestamp": "13:34:46.174947",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER SEQUENCE schema5048.seq5471 RENAME TO schema5048.seq5548",
          |   "INSERT INTO schema3224.table5225 (col5225_5227,col5225_5228,col5225_5229,col5225_5230,col5225_5231,col5225_5232,col5225_5233,col5225_5234,col5225_5235,col5225_5236,col5225_5237,col5225_5238,col5225_5239,col5225_5240) VALUES ('01070000A0E610000000000000':::GEOGRAPHY,'16:37:35.406972-04:01:00':::TIMETZ,1005:::OID,'BOX(-10 -10,10 10)':::BOX2D,2202:::OID,'1978-06-17':::DATE,B'11101111100101000010',0.4085620270341669:::FLOAT8,3781804427339826273:::INT8,true,'01030000000100000006000000B83122E5F06AD0415F6CB2E1115BF2C1563D18105319F2412008A153A25DD2C1229210CA8811E6C1DAA3881587FF0042F8802F502B6BDDC1706BA8A0C068F041CD61513922B3FFC1887B15B196EBE941B83122E5F06AD0415F6CB2E1115BF2C1':::GEOMETRY,'\"':::STRING,'d8ff:e1d1:15f7:566a:64cd:4f43:484f:a837/107':::INET,4096:::OID),(NULL,'09:42:03.309393-03:14:00':::TIMETZ,1700:::OID,'BOX(-0.7915415944938847 0.42891940566383313,0.5517028864132273 2.0529120145413016)':::BOX2D,4089:::OID,NULL,B'01000000111100001011',3.4028234663852886e+38:::FLOAT8,NULL,false,'01030000000100000005000000A416EF1108A0FD4112E10F5E0426F1C1CA6DF4B8D0840042B4E3B211F9F90042AA9865CA1476FAC1206559BC313C0042DEE78754221C02C2443CCBA59169FB41A416EF1108A0FD4112E10F5E0426F1C1':::GEOMETRY,e'\\x16\\x04:\\x0f\\x1a,':::STRING:::NAME,'5509:6236:8491:e2a6:3b56:7362:aaf1:e17e/35':::INET,1183:::OID),('0103000060E61000000100000008000000C843ED93E4DF3DC00A4A04E50D643EC088C40695BAADE7C1D8A78A5C056A37C064CE405B158330C05C186F68B67DE24140213C8F87FE2B403012A55257AC124014CB71E49A0CFB41C63506C7A8F66440404F7D6985FDF8BF925EB7F62215FBC13879BB9F68AC46408A85758C8770514050F8D66CAE6DDAC1BA115C1B0E2E62C000C63528B35D3B401050CE3188A0CEC164989AB35A1762C08CC92C1630DE34406CDFF3D82EA5E5C1C843ED93E4DF3DC00A4A04E50D643EC088C40695BAADE7C1':::GEOGRAPHY,'00:32:01.204457+11:24:00':::TIMETZ,21:::OID,'BOX(-1.77477918270027 -1.6240761077574726,0.020742573623563132 1.2915659074199173)':::BOX2D,1560:::OID,'1991-03-16':::DATE,B'00100111100001110101','NaN':::FLOAT8,3686258930713013282:::INT8,true,'0107000080030000000106000080020000000103000080010000000B000000A67037E88E6BE1C1B0026A812DDFD0C1C1438BDA445BFCC1D0EFEEF16FF5EA4120A5E761368EF6C10E9F2B2AF9C1EBC116C39756D9E3F1419E2D7332A1A401C2C4BC39B2D981ED418075DE55DE40FF4174CEAE9AB69CFFC160B0075379D4DB41504E41C55B5BFB417C6A86C89B6EF0C14CF62559FC14EAC1F02B15BAB87AFC418CFD52730575E8411E7E7DD0D8F5F8C100DDFA57B6E2E2414AC9391FB29F004294EF04784F0DDEC180D648A5E9D1B4C1BCAA0305F8B7EC4148666C9EA8C0F641F89CD8C88C4AD2C1DC536BFE7A75E64198B1794ADB58FFC14E827E947693FFC15ED1103A3087F3413927F83089DBF0C1A67037E88E6BE1C1B0026A812DDFD0C1C1438BDA445BFCC10103000080010000000A0000004EABCDDBADE2F8C168AC3CB055E6E6C1A00F5A7DDECB00422009E62A3012F84142C6F34D2FC3FBC17A438C041F20024202A0F6F8647EF941E282934B0F44F8C138596CD28954F24148BDC46E9F30EB4148E17AC531D0E9C17C3090E65A0BF141BC6977AFEBC4004200A6128CAB289AC1B8AF61B92BE1F0C112B233B638C4EEC1F4A3AF694442FF41D432A404FEDDF8413EAABD1184AEF7C11499C827C849F841D8F8524937A2FC41E91254ABB40701C23AE1FA8296C3F8410A1AAA50A911F8C105A8406A45D700C2607F0E9D55DBB44154C59BBC63D2DFC14EABCDDBADE2F8C168AC3CB055E6E6C1A00F5A7DDECB004201050000800300000001020000800200000098DD3FE649AEF14187877C11A94101C2A2521F665BCFE0C1E6FB831F99FAF7410060B7732296B9C1247BFBF71F8EF3C1010200008002000000B00E2BBD425DEA41984F43B5F6C9D541EC9203B5B1C7FC41B01910D1F083FD41D41923CD4AB9E3414CB017A99699F6410102000080020000008046481EA5A9D4C168E6DFB0B30EE0C120C01D56D72A01C20089EDED1E359041A0C150215D43CCC1AC65D520D8B4F2C10101000080C4B3BE80F571FAC1B2E4669A809400C26CF11E9A678EF741':::GEOMETRY,e'\\'\\x15B !\\f\u0026^[':::STRING:::NAME,'55.253.119.32/16':::INET,1560:::OID)",
          |   "ALTER VIEW schema4074.view5371 RENAME TO schema4074.view5563",
          |   "CREATE TYPE schema4074.enum5564 AS ENUM ('asda', 'f', 'dsdf')",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: relation \"seq5548\" (1453): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 175eebd03e5f2674a2d1f6bfea8bd72f131a6532:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:884: output in run_130727.839700873_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1948
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:884
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_130727.839700873_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | S (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema3146.table5807"]] :[["(CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\x0e\\\\\\bG':::STRING END)"]]
          | QUERY ["SELECT EXISTS ( SELECT * FROM schema3146.table5807 WHERE (CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\x0e\\\\\\bG':::STRING END)= ( SELECT  (CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\x0e\\\\\\bG':::STRING END) FROM (VALUES( ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL]) ) AS T(col5807_5810) ) )" []] :false
          | QUERY ["SELECT array[((CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\x0e\\\\\\bG':::STRING END))::STRING] FROM (VALUES(ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL]) ) AS T(col5807_5810)" []] :[["@\x0e\\\bG"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema3146.table5807'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema3146'\n\t\t           AND table_name = 'table5807'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [INSERT INTO schema3146.table5807 (col5807_5810) VALUES (ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL])]: ERROR: duplicate key value violates unique constraint "table5807_expr_key" (SQLSTATE 23505)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |    IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema3146.table5807\"]] :[[\"(CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\\\x0e\\\\\\\\\\\\bG':::STRING END)\"]]\nQUERY [\"SELECT EXISTS ( SELECT * FROM schema3146.table5807 WHERE (CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\\\x0e\\\\\\\\\\\\bG':::STRING END)= ( SELECT  (CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\\\x0e\\\\\\\\\\\\bG':::STRING END) FROM (VALUES( ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL]) ) AS T(col5807_5810) ) )\" []] :false\nQUERY [\"SELECT array[((CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\\\x0e\\\\\\\\\\\\bG':::STRING END))::STRING] FROM (VALUES(ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL]) ) AS T(col5807_5810)\" []] :[[\"@\\x0e\\\\\\bG\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema3146.table5807'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema3146'\\n\\t\\t           AND table_name = 'table5807'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO schema3146.table5807 (col5807_5810) VALUES (ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL])]: ERROR: duplicate key value violates unique constraint \"table5807_expr_key\" (SQLSTATE 23505)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ c070f2c70e0c92573a99dcb80b8a0cb0532bd06f:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:884: output in run_133618.008142896_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1949
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:884
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_133618.008142896_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ["(5\x11k\x16,06:43:38.98882-00:11:00,5\x11k\x16)"]]
          | QUERY ["SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6557, col6553_6559, col6553_6556)= ( SELECT  (col6553_6557, col6553_6559, col6553_6556) FROM (VALUES( '06:43:38.98882-00:11:00':::TIMETZ,e'5\\x11k\\x16':::STRING,e'5\\x11k\\x16':::STRING) ) AS T(col6553_6557,col6553_6559,col6553_6556) ) )" []] :false
          | QUERY ["SELECT array[((col6553_6557, col6553_6559, col6553_6556))::STRING] FROM (VALUES('06:43:38.98882-00:11:00':::TIMETZ,e'5\\x11k\\x16':::STRING,e'5\\x11k\\x16':::STRING) ) AS T(col6553_6557,col6553_6559,col6553_6556)" []] :[["(06:43:38.98882-00:11:00,5\x11k\x16,5\x11k\x16)"]]
          | QUERY ["SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (lower(CAST(col6553_6558 AS STRING)), col6553_6556)= ( SELECT  (lower(CAST(col6553_6558 AS STRING)), col6553_6556) FROM (VALUES( B'11101111011001100101101000010100',e'5\\x11k\\x16':::STRING) ) AS T(col6553_6558,col6553_6556) ) )" []] :false
          | QUERY ["SELECT array[((lower(CAST(col6553_6558 AS STRING)), col6553_6556))::STRING] FROM (VALUES(B'11101111011001100101101000010100',e'5\\x11k\\x16':::STRING) ) AS T(col6553_6558,col6553_6556)" []] :[["(11101111011001100101101000010100,5\x11k\x16)"]]
          | QUERY ["SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6559)= ( SELECT  (col6553_6559) FROM (VALUES( e'5\\x11k\\x16':::STRING) ) AS T(col6553_6559) ) )" []] :false
          | QUERY ["SELECT array[((col6553_6559))::STRING] FROM (VALUES(e'5\\x11k\\x16':::STRING) ) AS T(col6553_6559)" []] :[["5\x11k\x16"]]
          | QUERY ["SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6559, col6553_6556)= ( SELECT  (col6553_6559, col6553_6556) FROM (VALUES( e'5\\x11k\\x16':::STRING,e'5\\x11k\\x16':::STRING) ) AS T(col6553_6559,col6553_6556) ) )" []] :false
          | QUERY ["SELECT array[((col6553_6559, col6553_6556))::STRING] FROM (VALUES(e'5\\x11k\\x16':::STRING,e'5\\x11k\\x16':::STRING) ) AS T(col6553_6559,col6553_6556)" []] :[["(5\x11k\x16,5\x11k\x16)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema6147.table6553'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema6147' AND\n\t\t\t\t\t\t\t\t    table_name='table6553' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema6147.table6553', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema6147'\n\t\t           AND table_name = 'table6553'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[["schema6147" "table6553" "col6553_6559" "col6553_6556"]]
          | QUERY ["\n\t    SELECT count(*) = 0 from schema6147.table6553\n\t    WHERE col6553_6559 = e'5\\x11k\\x16':::STRING\n\t" []] :true
          | ===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema6147.table6553 (col6553_6556,col6553_6557,col6553_6558) VALUES (e'5\x11k\x16':::STRING,'06:43:38.98882-00:11:00':::TIMETZ,B'11101111011001100101101000010100')]
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | :43:38.98882-00:11:00':::TIMETZ,e'5\\\\x11k\\\\x16':::STRING,e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6557,col6553_6559,col6553_6556) ) )\" []] :false\nQUERY [\"SELECT array[((col6553_6557, col6553_6559, col6553_6556))::STRING] FROM (VALUES('06:43:38.98882-00:11:00':::TIMETZ,e'5\\\\x11k\\\\x16':::STRING,e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6557,col6553_6559,col6553_6556)\" []] :[[\"(06:43:38.98882-00:11:00,5\\x11k\\x16,5\\x11k\\x16)\"]]\nQUERY [\"SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (lower(CAST(col6553_6558 AS STRING)), col6553_6556)= ( SELECT  (lower(CAST(col6553_6558 AS STRING)), col6553_6556) FROM (VALUES( B'11101111011001100101101000010100',e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6558,col6553_6556) ) )\" []] :false\nQUERY [\"SELECT array[((lower(CAST(col6553_6558 AS STRING)), col6553_6556))::STRING] FROM (VALUES(B'11101111011001100101101000010100',e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6558,col6553_6556)\" []] :[[\"(11101111011001100101101000010100,5\\x11k\\x16)\"]]\nQUERY [\"SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6559)= ( SELECT  (col6553_6559) FROM (VALUES( e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6559) ) )\" []] :false\nQUERY [\"SELECT array[((col6553_6559))::STRING] FROM (VALUES(e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6559)\" []] :[[\"5\\x11k\\x16\"]]\nQUERY [\"SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6559, col6553_6556)= ( SELECT  (col6553_6559, col6553_6556) FROM (VALUES( e'5\\\\x11k\\\\x16':::STRING,e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6559,col6553_6556) ) )\" []] :false\nQUERY [\"SELECT array[((col6553_6559, col6553_6556))::STRING] FROM (VALUES(e'5\\\\x11k\\\\x16':::STRING,e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6559,col6553_6556)\" []] :[[\"(5\\x11k\\x16,5\\x11k\\x16)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema6147.table6553'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema6147' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table6553' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema6147.table6553', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema6147'\\n\\t\\t           AND table_name = 'table6553'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[[\"schema6147\" \"table6553\" \"col6553_6559\" \"col6553_6556\"]]\nQUERY [\"\\n\\t    SELECT count(*) = 0 from schema6147.table6553\\n\\t    WHERE col6553_6559 = e'5\\\\x11k\\\\x16':::STRING\\n\\t\" []] :true\n===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema6147.table6553 (col6553_6556,col6553_6557,col6553_6558) VALUES (e'5\\x11k\\x16':::STRING,'06:43:38.98882-00:11:00':::TIMETZ,B'11101111011001100101101000010100')]"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

          | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
          | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT sequence_name\n    FROM information_schema.sequences\n   WHERE sequence_schema = $1\n     AND sequence_name = $2\n   )" ["schema3710" "seq4194"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT schema_name\n\t\tFROM information_schema.schemata\n   WHERE schema_name = $1\n\t)" ["schema3710"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT sequence_name\n    FROM information_schema.sequences\n   WHERE sequence_schema = $1\n     AND sequence_name = $2\n   )" ["schema3710" "seq4498"]] :false
          | ===========================Previous statements [CREATE TYPE schema3710.enum4454 AS ENUM ('a', 'ssadda', 'fa') INSERT INTO schema3074.table3873 (col3873_3874,col3873_3875,col3873_3876,col3873_3877,col3873_3878,col3873_3879,col3873_3880) VALUES (e'\x05\x1cM\x0b$':::STRING:::NAME,'t':::STRING,1266:::OID,'9c9bada6-c05c-430a-b6b8-a0d51db4dc99':::UUID,0.08652160551068466:::FLOAT8,26:::OID,(-0.2958253258539698):::FLOAT8) ALTER SEQUENCE schema3710.seq4194 RENAME TO schema3710.seq4498]: ERROR: relation "schema3710.seq4194" does not exist (SQLSTATE 42P01)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | , FAMILY (col4479_4485, col4479_4496), FAMILY (col4479_4486), FAMILY (col4479_4491), FAMILY (col4479_4493, col4479_4482, col4479_4494, col4479_4481, col4479_4488))",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: intent on key /Table/3/1/104/2/1): \"sql txn\" meta={id=fd808be9 key=/Table/1001/1/0.669857324225041/\"0\\x03Y\\t\\x1cx$SB\"/18/0 pri=0.05514920 epo=0 ts=1655560956.503322420,2 min=1655560953.301654010,0 seq=29} lock=true stat=PENDING rts=1655560953.301654010,0 wto=false gul=1655560953.801654010,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:02:35.489648",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SEQUENCE IF NOT EXISTS schema4462.seq4463"
          |  ],
          |  "expectedExecErrors": "3F000",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 3F000Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema4462\"]] :false\nQUERY [\"SELECT EXISTS (\\n\\tSELECT sequence_name\\n    FROM information_schema.sequences\\n   WHERE sequence_schema = $1\\n     AND sequence_name = $2\\n   )\" [\"schema4462\" \"seq4463\"]] :false\n===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema4462.seq4463]: ERROR: cannot create \"schema4462.seq4463\" because the target database or schema does not exist (SQLSTATE 3F000)"
          | }
          |   733.0s        0            7.0            3.8     96.5   4563.4   4563.4   4563.4 opOk
          |   733.0s        0            2.0            0.8   6710.9   6710.9   6710.9   6710.9 txnCmtErr
          |   733.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnOk
          |   733.0s        0            1.0            2.2     21.0     21.0     21.0     21.0 txnRbk
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:02:29.392667",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE TYPE schema3710.enum4454 AS ENUM ('a', 'ssadda', 'fa')",
          |   "INSERT INTO schema3074.table3873 (col3873_3874,col3873_3875,col3873_3876,col3873_3877,col3873_3878,col3873_3879,col3873_3880) VALUES (e'\\x05\\x1cM\\x0b$':::STRING:::NAME,'t':::STRING,1266:::OID,'9c9bada6-c05c-430a-b6b8-a0d51db4dc99':::UUID,0.08652160551068466:::FLOAT8,26:::OID,(-0.2958253258539698):::FLOAT8)",
          |   "ALTER SEQUENCE schema3710.seq4194 RENAME TO schema3710.seq4498"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT sequence_name\\n    FROM information_schema.sequences\\n   WHERE sequence_schema = $1\\n     AND sequence_name = $2\\n   )\" [\"schema3710\" \"seq4194\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema3710\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT sequence_name\\n    FROM information_schema.sequences\\n   WHERE sequence_schema = $1\\n     AND sequence_name = $2\\n   )\" [\"schema3710\" \"seq4498\"]] :false\n===========================Previous statements [CREATE TYPE schema3710.enum4454 AS ENUM ('a', 'ssadda', 'fa') INSERT INTO schema3074.table3873 (col3873_3874,col3873_3875,col3873_3876,col3873_3877,col3873_3878,col3873_3879,col3873_3880) VALUES (e'\\x05\\x1cM\\x0b$':::STRING:::NAME,'t':::STRING,1266:::OID,'9c9bada6-c05c-430a-b6b8-a0d51db4dc99':::UUID,0.08652160551068466:::FLOAT8,26:::OID,(-0.2958253258539698):::FLOAT8) ALTER SEQUENCE schema3710.seq4194 RENAME TO schema3710.seq4498]: ERROR: relation \"schema3710.seq4194\" does not exist (SQLSTATE 42P01)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:897: output in run_130204.189452170_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1957
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:897
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_130204.189452170_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | :00,11100101111010100001011100101111101)"]]
          | QUERY ["SELECT num_nulls(col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501) > 0 FROM (VALUES(B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM public.table3500 WHERE (col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501)= ( SELECT  (col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501) FROM (VALUES( B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507) ) )" []] :false
          | QUERY ["SELECT array[((col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501))::STRING] FROM (VALUES(B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507)" []] :[["(25,20:06:57.841197,\"2001-10-16 07:31:34.000275\",5117979915624448840,12:07:52.582766-06:40:00,10000100000011001001110010111010100)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table3500'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table3500' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table3500', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table3500'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [CREATE TABLE public.table4712 AS SELECT public.table3500.col3500_3501 FROM public.table3500 INSERT INTO public.table3500 (col3500_3501,col3500_3502,col3500_3503,col3500_3504,col3500_3505,col3500_3506,col3500_3507) VALUES (B'10101000111100111101101010101000101','03:39:40.426812':::TIME,(-7841707273834830937):::INT8,'1994-09-15 09:14:57.00021':::TIMESTAMP,NULL,90004:::OID,'04:22:35.192972+14:48:00':::TIMETZ),(B'11100101111010100001011100101111101','07:28:52.590851':::TIME,32767:::INT8,'2006-09-01 16:29:13.000445':::TIMESTAMP,(-2147483648):::INT8,1700:::OID,'12:19:14.113199-13:37:00':::TIMETZ),(B'10000100000011001001110010111010100','20:06:57.841197':::TIME,1292809320963222986:::INT8,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,25:::OID,'12:07:52.582766-06:40:00':::TIMETZ)]: ERROR: relation "table4712" (1203): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 00_3501) \u003e 0 FROM (VALUES(B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM public.table3500 WHERE (col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501)= ( SELECT  (col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501) FROM (VALUES( B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507) ) )\" []] :false\nQUERY [\"SELECT array[((col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501))::STRING] FROM (VALUES(B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507)\" []] :[[\"(25,20:06:57.841197,\\\"2001-10-16 07:31:34.000275\\\",5117979915624448840,12:07:52.582766-06:40:00,10000100000011001001110010111010100)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table3500'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table3500' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table3500', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table3500'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [CREATE TABLE public.table4712 AS SELECT public.table3500.col3500_3501 FROM public.table3500 INSERT INTO public.table3500 (col3500_3501,col3500_3502,col3500_3503,col3500_3504,col3500_3505,col3500_3506,col3500_3507) VALUES (B'10101000111100111101101010101000101','03:39:40.426812':::TIME,(-7841707273834830937):::INT8,'1994-09-15 09:14:57.00021':::TIMESTAMP,NULL,90004:::OID,'04:22:35.192972+14:48:00':::TIMETZ),(B'11100101111010100001011100101111101','07:28:52.590851':::TIME,32767:::INT8,'2006-09-01 16:29:13.000445':::TIMESTAMP,(-2147483648):::INT8,1700:::OID,'12:19:14.113199-13:37:00':::TIMETZ),(B'10000100000011001001110010111010100','20:06:57.841197':::TIME,1292809320963222986:::INT8,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,25:::OID,'12:07:52.582766-06:40:00':::TIMETZ)]: ERROR: relation \"table4712\" (1203): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 377b113bd7e7b679c55a0f93e0d8ab5dbb6f2e97:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:897: output in run_132814.766426103_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1957
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:897
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_132814.766426103_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | death:
          | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT schema_name\n\t\tFROM information_schema.schemata\n   WHERE schema_name = $1\n\t)" ["schema1970"]] :false
          | ===========================Previous statements [CREATE TABLE IF NOT EXISTS schema877.table1931 (col1931_1933 REGPROCEDURE NULL, col1931_1934 TIMETZ, col1931_1935 REGROLE NOT NULL, col1931_1936 TIMESTAMP NOT NULL, col1931_1937 REGROLE, col1931_1938 INT8 NOT NULL, col1931_1939 TIMESTAMP NULL, col1931_1940 "char", col1931_1941 STRING AS (CASE WHEN col1931_1937 IS NULL THEN e' \x0e4HQ/':::STRING ELSE e'k\x1b@\f':::STRING END) STORED, INDEX (col1931_1934, (col1931_1938 + (-7086260798774548313):::INT8), col1931_1933 ASC, col1931_1938 ASC, lower(CAST(col1931_1934 AS STRING)), col1931_1937), UNIQUE (col1931_1940 ASC, col1931_1934 DESC, col1931_1937, col1931_1933 ASC), UNIQUE (col1931_1940 ASC, col1931_1934 DESC, col1931_1938, col1931_1941 DESC, col1931_1936, col1931_1937 DESC), INDEX (col1931_1941 DESC, col1931_1937 DESC, col1931_1935 ASC, col1931_1938 ASC, col1931_1936, col1931_1933 ASC, col1931_1934, col1931_1939 DESC), INDEX (col1931_1936 ASC, col1931_1941, col1931_1937 DESC, col1931_1935, col1931_1939, col1931_1934 ASC, col1931_1940, col1931_1933), UNIQUE (col1931_1939, col1931_1934, col1931_1940 ASC) STORING (col1931_1933, col1931_1935, col1931_1938, col1931_1941), INDEX (col1931_1937 ASC, col1931_1934 ASC, col1931_1936 ASC, col1931_1935 ASC), INDEX (col1931_1938 DESC, col1931_1937 ASC, col1931_1933 DESC, col1931_1935 ASC, col1931_1939 DESC, col1931_1936 DESC, col1931_1941), FAMILY (col1931_1936, col1931_1939, col1931_1935, col1931_1941, col1931_1933), FAMILY (col1931_1937), FAMILY (col1931_1940, col1931_1934, col1931_1938)) INSERT INTO schema877.table1258 (col1258_1259,col1258_1260,col1258_1261,col1258_1262,col1258_1263,col1258_1264,col1258_1265,col1258_1266,col1258_1267,col1258_1268,col1258_1269,col1258_1270,col1258_1271,col1258_1272,col1258_1273,col1258_1274,col1258_1275) VALUES ('-56 years -103 days -08:24:29.822279':::INTERVAL,e'%MEc\x10A':::STRING:::NAME,'f4a0:bb31:5dbe:5224:45f:63bf:cbb2:eb71/87':::INET,1266:::OID,1266:::OID,10.228008859700175:::DECIMAL,'[[[["a", [], {}, []], true], []], {"a": false, "c": {"bar": []}, "foobar": {}}, []]':::JSONB,ARRAY['1989-03-06':::DATE,'1984-04-24':::DATE,'1984-07-20':::DATE,'1990-06-09':::DATE,'1979-08-25':::DATE],90002:::OID,'*':::STRING,e'{"?=\\\\<oIaR3h{": {"&|f&n+D(~<S": {"a": [], "baz": [false]}}, "bG(.-": []}':::JSONB,1560:::OID,1.7976931348623157e+308:::FLOAT8,1186:::OID,4089:::OID,1114:::OID,3802:::OID),('-30 years -3 mons -536 days -01:13:14.812631':::INTERVAL,'':::STRING:::NAME,'40.131.66.109/15':::INET,26:::OID,2202:::OID,(-2.667423998350947061E+37):::DECIMAL,e'{"8E9O:I#h4": {"r8jsH]``": {}}, ">L*me[$\\"A%a_": [{}, []], "EOG": [[{}, {}], "b:$*\\"=W,55q", "-Z9t9-1%n5"], "F.X~pIxo$": [], "``ng(|\\"}9zh#Y": {}, "a": true, "c": [[{}], []]}':::JSONB,ARRAY['1988-01-07':::DATE,'1970-05-04':::DATE],18:::OID,'3':::STRING,e'{">TFhOr": [{"Ud42HZe": 4.367809801176824}, []], "H$#OD": {}, "a": {}, "b": {" \\"|{+O;!41F": {"bar": []}, "c": [null], "foobar": []}, "bar": "baz"}':::JSONB,17:::OID,(-1.2201161276955275):::FLOAT8,90004:::OID,90000:::OID,17:::OID,869:::OID),('41 years 1 mon 429 days 17:18:59.405336':::INTERVAL,e'PXt\r\x16]':::STRING:::NAME,'7162:1d9c:2cd:9e6e:63a4:208d:a0eb:c4d9/15':::INET,700:::OID,21:::OID,(-7.531607206776994903E+23):::DECIMAL,e'[{"F(o~F/x": [{"c": {}}], "bar": {"Y)3(\\"q\\\\^Yg(": {}}, "foo": [null, [false], null]}, true, null]':::JSONB,ARRAY['1974-02-18':::DATE],90004:::OID,e'\x13':::STRING,e'{"& MtAo3-{x": {"&:&GZNCs12i": {}}, "a\'S@;;)wci8": [[[{"#gD^__E\\"": true, "Dlj": "bj!/nJ"}, [null], []]], "c", []], "wMi08n": {"b": {}}}':::JSONB,17:::OID,(-2.5003141290217457):::FLOAT8,1266:::OID,2205:::OID,21:::OID,701:::OID) CREATE SCHEMA schema1970 AUTHORIZATION root]: ERROR: relation "table1931" (572): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema1970\"]] :false\n===========================Previous statements [CREATE TABLE IF NOT EXISTS schema877.table1931 (col1931_1933 REGPROCEDURE NULL, col1931_1934 TIMETZ, col1931_1935 REGROLE NOT NULL, col1931_1936 TIMESTAMP NOT NULL, col1931_1937 REGROLE, col1931_1938 INT8 NOT NULL, col1931_1939 TIMESTAMP NULL, col1931_1940 \"char\", col1931_1941 STRING AS (CASE WHEN col1931_1937 IS NULL THEN e' \\x0e4HQ/':::STRING ELSE e'k\\x1b@\\f':::STRING END) STORED, INDEX (col1931_1934, (col1931_1938 + (-7086260798774548313):::INT8), col1931_1933 ASC, col1931_1938 ASC, lower(CAST(col1931_1934 AS STRING)), col1931_1937), UNIQUE (col1931_1940 ASC, col1931_1934 DESC, col1931_1937, col1931_1933 ASC), UNIQUE (col1931_1940 ASC, col1931_1934 DESC, col1931_1938, col1931_1941 DESC, col1931_1936, col1931_1937 DESC), INDEX (col1931_1941 DESC, col1931_1937 DESC, col1931_1935 ASC, col1931_1938 ASC, col1931_1936, col1931_1933 ASC, col1931_1934, col1931_1939 DESC), INDEX (col1931_1936 ASC, col1931_1941, col1931_1937 DESC, col1931_1935, col1931_1939, col1931_1934 ASC, col1931_1940, col1931_1933), UNIQUE (col1931_1939, col1931_1934, col1931_1940 ASC) STORING (col1931_1933, col1931_1935, col1931_1938, col1931_1941), INDEX (col1931_1937 ASC, col1931_1934 ASC, col1931_1936 ASC, col1931_1935 ASC), INDEX (col1931_1938 DESC, col1931_1937 ASC, col1931_1933 DESC, col1931_1935 ASC, col1931_1939 DESC, col1931_1936 DESC, col1931_1941), FAMILY (col1931_1936, col1931_1939, col1931_1935, col1931_1941, col1931_1933), FAMILY (col1931_1937), FAMILY (col1931_1940, col1931_1934, col1931_1938)) INSERT INTO schema877.table1258 (col1258_1259,col1258_1260,col1258_1261,col1258_1262,col1258_1263,col1258_1264,col1258_1265,col1258_1266,col1258_1267,col1258_1268,col1258_1269,col1258_1270,col1258_1271,col1258_1272,col1258_1273,col1258_1274,col1258_1275) VALUES ('-56 years -103 days -08:24:29.822279':::INTERVAL,e'%MEc\\x10A':::STRING:::NAME,'f4a0:bb31:5dbe:5224:45f:63bf:cbb2:eb71/87':::INET,1266:::OID,1266:::OID,10.228008859700175:::DECIMAL,'[[[[\"a\", [], {}, []], true], []], {\"a\": false, \"c\": {\"bar\": []}, \"foobar\": {}}, []]':::JSONB,ARRAY['1989-03-06':::DATE,'1984-04-24':::DATE,'1984-07-20':::DATE,'1990-06-09':::DATE,'1979-08-25':::DATE],90002:::OID,'*':::STRING,e'{\"?=\\\\\\\\\u003coIaR3h{\": {\"\u0026|f\u0026n+D(~\u003cS\": {\"a\": [], \"baz\": [false]}}, \"bG(.-\": []}':::JSONB,1560:::OID,1.7976931348623157e+308:::FLOAT8,1186:::OID,4089:::OID,1114:::OID,3802:::OID),('-30 years -3 mons -536 days -01:13:14.812631':::INTERVAL,'':::STRING:::NAME,'40.131.66.109/15':::INET,26:::OID,2202:::OID,(-2.667423998350947061E+37):::DECIMAL,e'{\"8E9O:I#h4\": {\"r8jsH]``\": {}}, \"\u003eL*me[$\\\\\"A%a_\": [{}, []], \"EOG\": [[{}, {}], \"b:$*\\\\\"=W,55q\", \"-Z9t9-1%n5\"], \"F.X~pIxo$\": [], \"``ng(|\\\\\"}9zh#Y\": {}, \"a\": true, \"c\": [[{}], []]}':::JSONB,ARRAY['1988-01-07':::DATE,'1970-05-04':::DATE],18:::OID,'3':::STRING,e'{\"\u003eTFhOr\": [{\"Ud42HZe\": 4.367809801176824}, []], \"H$#OD\": {}, \"a\": {}, \"b\": {\" \\\\\"|{+O;!41F\": {\"bar\": []}, \"c\": [null], \"foobar\": []}, \"bar\": \"baz\"}':::JSONB,17:::OID,(-1.2201161276955275):::FLOAT8,90004:::OID,90000:::OID,17:::OID,869:::OID),('41 years 1 mon 429 days 17:18:59.405336':::INTERVAL,e'PXt\\r\\x16]':::STRING:::NAME,'7162:1d9c:2cd:9e6e:63a4:208d:a0eb:c4d9/15':::INET,700:::OID,21:::OID,(-7.531607206776994903E+23):::DECIMAL,e'[{\"F(o~F/x\": [{\"c\": {}}], \"bar\": {\"Y)3(\\\\\"q\\\\\\\\^Yg(\": {}}, \"foo\": [null, [false], null]}, true, null]':::JSONB,ARRAY['1974-02-18':::DATE],90004:::OID,e'\\x13':::STRING,e'{\"\u0026 MtAo3-{x\": {\"\u0026:\u0026GZNCs12i\": {}}, \"a\\'S@;;)wci8\": [[[{\"#gD^__E\\\\\"\": true, \"Dlj\": \"bj!/nJ\"}, [null], []]], \"c\", []], \"wMi08n\": {\"b\": {}}}':::JSONB,17:::OID,(-2.5003141290217457):::FLOAT8,1266:::OID,2205:::OID,21:::OID,701:::OID) CREATE SCHEMA schema1970 AUTHORIZATION root]: ERROR: relation \"table1931\" (572): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 7af6c0d3e2085cafda90d5f0b15d06f6a360f154:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_131241.151871516_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1964
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_131241.151871516_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n\t\t" ["schema2091.table3716"]] :[]
          | QUERY ["\nWITH tab_json AS (\n                    SELECT crdb_internal.pb_to_json(\n                            'desc',\n                            descriptor\n                           )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n\t\t" ["schema2091.table3716"]] :[]
          | QUERY ["\n    WITH tab_json AS (\n                    SELECT crdb_internal.pb_to_json(\n                            'desc',\n                            descriptor\n                           )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema2091.table3716"]] :[]
          | Stmts: 
          | []
          | : ERROR: unknown schema "[1188]" (SQLSTATE 3F000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 01000080A29617CD28C8FDC1627898F1282EF241B841D34E6BE4FA4101010000809839950618CDED4160857E101C24BBC1FC7E53303852E5C10101000080303FD618B326024280075B9A0C46D7C1C08355394470CE4101010000809C21F0974AD1F74108310A4D79DDE54178925ADF3880DB4101010000803E0E14320B6C00C2E04934AC4145D7415EA68AA1E964F9C101030000800100000004000000C821B775C34ED2C180C56B09953FBA41F63A7C9FF886F3C18E273EE93B2FEDC1D0893EE2103DD6C1A6217CDC84EEF1412A84143B585A0142E0E2D363B506E241F45720A5B45AD9C1C821B775C34ED2C180C56B09953FBA41F63A7C9FF886F3C10104000080070000000101000080FE4E13446283F7418A9038588589E9C1FC82A2C6797DE5410101000080F464A09C909AE941D3C2F91EF60AF6C10C73262E987FF2410101000080885C7962E6D1DBC17092A066127CDA41C020A318E3F3F241010100008080D1D19EDF2AA5419C5595385F94F4416CBF5D2944CBE9C1010100008002B09C88DEAEF1C130588FEC278ACBC1988AEDEC9D4AF9410101000080D4E3DD372AC7E041EAC2910E956FF241468A35CA1301F9C10101000080F898C41EA824FB41C871C433ABFAFB41AEBDB3DDEF17F8C10104000080070000000101000080C052FECD37D2A941E848DA4603DCF841C830E144A160E24101010000809EC8EF6CE253EEC1D05DE650C651E1C17C8B67DEA06FF6410101000080947083658299E9419A6F6EC000B8EEC13C553E43F3F801C20101000080B0438FBC09DCD4C12C56EEFB60A2F44107C6C6A500BFF6C10101000080805C36645612A24168F4CC65ADC8D4C1C08AB16535ABC54101010000802B64FF4EFA4402C2D8A7732A9D63E14100F9705EB8AF88410101000080806F5C1EDDEFBDC1C4E18C04102BFB4181EBAFC5617402C2010200008003000000D872FEAF5365DEC1A083B9B84852C54158DD6247D464FAC1B0B47E0F865ED8417072065A0CD9C5411F0826C63686F2C184E66AAE95B5E1C10E51DDAE51C6014284517B19BEAAE341':::GEOMETRY,1083:::OID,19:::OID,0.6792910099029541:::FLOAT8,NULL,1082:::OID,(-661791291):::INT8){
          |  "workerId": 0,
          |  "clientTimestamp": "13:32:13.391106",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE TABLE schema2091.table3716 AS SELECT schema2091.table3716.col2719_2728, schema2091.table3716.col2719_2728, schema5028.table6347.col6347_6348, schema5028.table6347.col6347_6358, schema5028.table6347.col6347_6357, schema5028.table6347.col6347_6360, schema5028.table6347.col6347_6350, schema5028.table6347.col6347_6355, schema5028.table6347.col6347_6353, schema5028.table6347.col6347_6351, schema5028.table6347.col6347_6354 FROM schema2091.table3716, schema2091.table3716, schema5028.table6347"
          |  ],
          |  "expectedExecErrors": "42701,42712,42P07",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42701,42712,42P07Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema2091\" \"table3716\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema2091\" \"table3716\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema5028\" \"table6347\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema2091\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema2091\" \"table3716\"]] :true\n===========================Previous statements [CREATE TABLE schema2091.table3716 AS SELECT schema2091.table3716.col2719_2728, schema2091.table3716.col2719_2728, schema5028.table6347.col6347_6348, schema5028.table6347.col6347_6358, schema5028.table6347.col6347_6357, schema5028.table6347.col6347_6360, schema5028.table6347.col6347_6350, schema5028.table6347.col6347_6355, schema5028.table6347.col6347_6353, schema5028.table6347.col6347_6351, schema5028.table6347.col6347_6354 FROM schema2091.table3716, schema2091.table3716, schema5028.table6347]: ERROR: source name \"table3716\" specified more than once (missing AS clause) (SQLSTATE 42712)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ e1e99da678e29db4ef21813423f012273c99add2:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    test_runner.go:1027,test_runner.go:926: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 4e8134033890c956d39020f28a0e99b21b32649f:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133240.358591906_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_133240.358591906_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 2291)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM schema895.table2289 WHERE (col2289_2291)= ( SELECT  (col2289_2291) FROM (VALUES( B'0111111111111111111111111111111111111') ) AS T(col2289_2291) ) )" []] :true
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema895.table2289'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema895' AND\n\t\t\t\t\t\t\t\t    table_name='table2289' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema895.table2289', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema895'\n\t\t           AND table_name = 'table2289'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[["schema895" "table2289" "col2289_2291" "col2289_2291"]]
          | ===========================Previous statements [INSERT INTO schema3085.table3840 (col3840_3843,col3840_3844,col3840_3845,col3840_3846,col3840_3847,col3840_3848,col3840_3849,col3840_3850,col3840_3851,col3840_3852,col3840_3853,col3840_3854,col3840_3855,col3840_3856,col3840_3857,col3840_3858,col3840_3859,col3840_3860,col3840_3861) VALUES (3803780901110908529:::INT8,3802:::OID,true,'14:12:36.797101-12:24:00':::TIMETZ,1:::INT8,ARRAY[B'11100111101101000001011010111000110010101111',B'00010001001100111000001010111101100110010001',B'10011011001011101111010110111011101111101111',B'00101000001011010011100110111111000001010101'],1083:::OID,1560:::OID,24:::OID,26:::OID,'z'::CHAR,'-59 years -811 days -02:04:13.458369':::INTERVAL,'\x4280454255a8c6':::BYTES,'122.23.1.169/1':::INET,'46 years 10 mons 356 days 12:26:51.38384':::INTERVAL,'oz0?2$k'::NAME:::NAME,e'\x03rFm'::VARCHAR,'o>,nK'::STRING,'3000-01-01 00:00:00':::TIMESTAMP),((-6087578964869502355):::INT8,2205:::OID,true,'04:01:08.380617+04:14:00':::TIMETZ,282654241:::INT8,ARRAY[B'01100100101110010101100001100100000000001101',B'01100011111100110010011010010110011100101011'],20:::OID,19:::OID,90002:::OID,19:::OID,e'\x1c'::CHAR,'36 years 8 mons 297 days 15:12:15.082235':::INTERVAL,'\xb305bb41ba07b8':::BYTES,'a953:3d0f:ee2f:6d2f:508a:93a0:bc9b:79f4/59':::INET,'-60 years -1 mons -164 days -18:38:14.39587':::INTERVAL,e'LE\x14'::NAME:::NAME,e'\x167C%\x17v f'::VARCHAR,e'\''::STRING,'1984-12-15 20:21:10.00023':::TIMESTAMP),(4027233331045315694:::INT8,90000:::OID,false,'09:45:48.944205+12:40:00':::TIMETZ,(-507780718):::INT8,ARRAY[B'00110011010001000101011111110111001000011100',B'01111111111111111111111111111111111111111111'],17:::OID,26:::OID,1114:::OID,2206:::OID,e'\f'::CHAR,'-23 years -3 mons -705 days -14:36:11.562614':::INTERVAL,'\x9c4581':::BYTES,'152.51.89.37/2':::INET,'-4 years -2 mons -217 days -14:28:10.220544':::INTERVAL,e'\x1e@M'::NAME:::NAME,e'\x11F\x14\x0fnD%'::VARCHAR,NULL,'1984-08-30 01:27:04.000602':::TIMESTAMP) INSERT INTO schema895.table2289 (col2289_2290,col2289_2291) VALUES (1560:::OID,B'0111111111111111111111111111111111111')]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  B'0111111111111111111111111111111111111') ) AS T(col2289_2291) ) )\" []] :true\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema895.table2289'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema895' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table2289' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema895.table2289', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema895'\\n\\t\\t           AND table_name = 'table2289'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[[\"schema895\" \"table2289\" \"col2289_2291\" \"col2289_2291\"]]\n===========================Previous statements [INSERT INTO schema3085.table3840 (col3840_3843,col3840_3844,col3840_3845,col3840_3846,col3840_3847,col3840_3848,col3840_3849,col3840_3850,col3840_3851,col3840_3852,col3840_3853,col3840_3854,col3840_3855,col3840_3856,col3840_3857,col3840_3858,col3840_3859,col3840_3860,col3840_3861) VALUES (3803780901110908529:::INT8,3802:::OID,true,'14:12:36.797101-12:24:00':::TIMETZ,1:::INT8,ARRAY[B'11100111101101000001011010111000110010101111',B'00010001001100111000001010111101100110010001',B'10011011001011101111010110111011101111101111',B'00101000001011010011100110111111000001010101'],1083:::OID,1560:::OID,24:::OID,26:::OID,'z'::CHAR,'-59 years -811 days -02:04:13.458369':::INTERVAL,'\\x4280454255a8c6':::BYTES,'122.23.1.169/1':::INET,'46 years 10 mons 356 days 12:26:51.38384':::INTERVAL,'oz0?2$k'::NAME:::NAME,e'\\x03rFm'::VARCHAR,'o\u003e,nK'::STRING,'3000-01-01 00:00:00':::TIMESTAMP),((-6087578964869502355):::INT8,2205:::OID,true,'04:01:08.380617+04:14:00':::TIMETZ,282654241:::INT8,ARRAY[B'01100100101110010101100001100100000000001101',B'01100011111100110010011010010110011100101011'],20:::OID,19:::OID,90002:::OID,19:::OID,e'\\x1c'::CHAR,'36 years 8 mons 297 days 15:12:15.082235':::INTERVAL,'\\xb305bb41ba07b8':::BYTES,'a953:3d0f:ee2f:6d2f:508a:93a0:bc9b:79f4/59':::INET,'-60 years -1 mons -164 days -18:38:14.39587':::INTERVAL,e'LE\\x14'::NAME:::NAME,e'\\x167C%\\x17v f'::VARCHAR,e'\\''::STRING,'1984-12-15 20:21:10.00023':::TIMESTAMP),(4027233331045315694:::INT8,90000:::OID,false,'09:45:48.944205+12:40:00':::TIMETZ,(-507780718):::INT8,ARRAY[B'00110011010001000101011111110111001000011100',B'01111111111111111111111111111111111111111111'],17:::OID,26:::OID,1114:::OID,2206:::OID,e'\\f'::CHAR,'-23 years -3 mons -705 days -14:36:11.562614':::INTERVAL,'\\x9c4581':::BYTES,'152.51.89.37/2':::INET,'-4 years -2 mons -217 days -14:28:10.220544':::INTERVAL,e'\\x1e@M'::NAME:::NAME,e'\\x11F\\x14\\x0fnD%'::VARCHAR,NULL,'1984-08-30 01:27:04.000602':::TIMESTAMP) INSERT INTO schema895.table2289 (col2289_2290,col2289_2291) VALUES (1560:::OID,B'0111111111111111111111111111111111111')]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 3c9b17113488d2ee6929936aa6ec48396f3ed71c:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_134252.570148638_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_134252.570148638_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  (CASE WHEN col54_60 IS NULL THEN '+':::STRING ELSE e'R\\x0e':::STRING END, col54_55, col54_60, col54_65, lower(CAST(col54_63 AS STRING)), col54_58, col54_63, col54_66, col54_56, col54_59) FROM (VALUES( e'p\\x7f|H1\\x0byn':::STRING,'15:03:13.818412':::TIME,'-2000-01-01 00:00:00':::TIMESTAMP,'u;RF':::STRING,24:::OID,19:::OID,'\\x98f4c5d53a22':::BYTES,'*'::NAME:::NAME) ) AS T(col54_65,col54_63,col54_58,col54_66,col54_56,col54_59,col54_60,col54_55) ) )" []] :false
          | QUERY ["SELECT array[((CASE WHEN col54_60 IS NULL THEN '+':::STRING ELSE e'R\\x0e':::STRING END, col54_55, col54_60, col54_65, lower(CAST(col54_63 AS STRING)), col54_58, col54_63, col54_66, col54_56, col54_59))::STRING] FROM (VALUES(e'p\\x7f|H1\\x0byn':::STRING,'15:03:13.818412':::TIME,'-2000-01-01 00:00:00':::TIMESTAMP,'u;RF':::STRING,24:::OID,19:::OID,'\\x98f4c5d53a22':::BYTES,'*'::NAME:::NAME) ) AS T(col54_65,col54_63,col54_58,col54_66,col54_56,col54_59,col54_60,col54_55)" []] :[["(R\x0e,*,\"\\\\x98f4c5d53a22\",\"p\u007f|H1\vyn\",15:03:13.818412,\"-2000-01-01 00:00:00\",15:03:13.818412,u;RF,24,19)"]]
          | QUERY ["SELECT num_nulls(col54_59) > 0 FROM (VALUES(19:::OID) ) AS T(col54_59)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM public.table54 WHERE (col54_59)= ( SELECT  (col54_59) FROM (VALUES( 19:::OID) ) AS T(col54_59) ) )" []] :false
          | QUERY ["SELECT array[((col54_59))::STRING] FROM (VALUES(19:::OID) ) AS T(col54_59)" []] :[["19"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table54'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table54' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table54', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table54'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[["public" "table54" "col54_59" "col54_59"]]
          | ===========================Previous statements [DROP SCHEMA "schema607" CASCADE INSERT INTO public.table54 (col54_55,col54_56,col54_57,col54_58,col54_59,col54_60,col54_61,col54_62,col54_63) VALUES (e'}>\'e$\'\x04'::NAME:::NAME,1114:::OID,'+'::"char",'1987-05-24 02:15:25.000317':::TIMESTAMP,1700:::OID,'\xb3ea8d':::BYTES,(-0.705607090841737):::FLOAT8,'4d5b9439-605d-4b4c-bfc8-41c9d454f0c3':::UUID,'11:53:14.69639':::TIME),(NULL,1186:::OID,'o'::"char",'1999-11-15 07:53:18.000567':::TIMESTAMP,1186:::OID,'\xda565e79c4b8':::BYTES,0.28914216093517053:::FLOAT8,'8771a065-8648-4901-9747-c082ee793943':::UUID,'23:32:57.012975':::TIME),('*'::NAME:::NAME,24:::OID,'{'::"char",'-2000-01-01 00:00:00':::TIMESTAMP,19:::OID,'\x98f4c5d53a22':::BYTES,(-1.0715612012272748):::FLOAT8,'c6dbc4a2-43e5-497e-a368-10696211875e':::UUID,'15:03:13.818412':::TIME)]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 775898849214332929 could not be loaded: job with ID 775898849214332929 does not exist (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | \\\\x7f|H1\\\\x0byn':::STRING,'15:03:13.818412':::TIME,'-2000-01-01 00:00:00':::TIMESTAMP,'u;RF':::STRING,24:::OID,19:::OID,'\\\\x98f4c5d53a22':::BYTES,'*'::NAME:::NAME) ) AS T(col54_65,col54_63,col54_58,col54_66,col54_56,col54_59,col54_60,col54_55) ) )\" []] :false\nQUERY [\"SELECT array[((CASE WHEN col54_60 IS NULL THEN '+':::STRING ELSE e'R\\\\x0e':::STRING END, col54_55, col54_60, col54_65, lower(CAST(col54_63 AS STRING)), col54_58, col54_63, col54_66, col54_56, col54_59))::STRING] FROM (VALUES(e'p\\\\x7f|H1\\\\x0byn':::STRING,'15:03:13.818412':::TIME,'-2000-01-01 00:00:00':::TIMESTAMP,'u;RF':::STRING,24:::OID,19:::OID,'\\\\x98f4c5d53a22':::BYTES,'*'::NAME:::NAME) ) AS T(col54_65,col54_63,col54_58,col54_66,col54_56,col54_59,col54_60,col54_55)\" []] :[[\"(R\\x0e,*,\\\"\\\\\\\\x98f4c5d53a22\\\",\\\"p\\u007f|H1\\vyn\\\",15:03:13.818412,\\\"-2000-01-01 00:00:00\\\",15:03:13.818412,u;RF,24,19)\"]]\nQUERY [\"SELECT num_nulls(col54_59) \u003e 0 FROM (VALUES(19:::OID) ) AS T(col54_59)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM public.table54 WHERE (col54_59)= ( SELECT  (col54_59) FROM (VALUES( 19:::OID) ) AS T(col54_59) ) )\" []] :false\nQUERY [\"SELECT array[((col54_59))::STRING] FROM (VALUES(19:::OID) ) AS T(col54_59)\" []] :[[\"19\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table54'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table54' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table54', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table54'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[[\"public\" \"table54\" \"col54_59\" \"col54_59\"]]\n===========================Previous statements [DROP SCHEMA \"schema607\" CASCADE INSERT INTO public.table54 (col54_55,col54_56,col54_57,col54_58,col54_59,col54_60,col54_61,col54_62,col54_63) VALUES (e'}\u003e\\'e$\\'\\x04'::NAME:::NAME,1114:::OID,'+'::\"char\",'1987-05-24 02:15:25.000317':::TIMESTAMP,1700:::OID,'\\xb3ea8d':::BYTES,(-0.705607090841737):::FLOAT8,'4d5b9439-605d-4b4c-bfc8-41c9d454f0c3':::UUID,'11:53:14.69639':::TIME),(NULL,1186:::OID,'o'::\"char\",'1999-11-15 07:53:18.000567':::TIMESTAMP,1186:::OID,'\\xda565e79c4b8':::BYTES,0.28914216093517053:::FLOAT8,'8771a065-8648-4901-9747-c082ee793943':::UUID,'23:32:57.012975':::TIME),('*'::NAME:::NAME,24:::OID,'{'::\"char\",'-2000-01-01 00:00:00':::TIMESTAMP,19:::OID,'\\x98f4c5d53a22':::BYTES,(-1.0715612012272748):::FLOAT8,'c6dbc4a2-43e5-497e-a368-10696211875e':::UUID,'15:03:13.818412':::TIME)]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 775898849214332929 could not be loaded: job with ID 775898849214332929 does not exist (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 33d70998719051ee058bc9e516afa238ea7b7451:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133321.381602086_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_133321.381602086_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | col1437_1442);" []] :[["[{\"7qanfbm *87\": {\"/n5#\": []}, \"d+ub0ui\": {\"93yb!q*%l2\": 1.1152646744807915, \"baz\": \"ir8m!_<k1\"}, \"v%viii3\": [{\"baz\": {\"foo\": []}, \"fpfbqqyjf(f\": {}}], \"a\": {}, \"foo\": {\"b\": {\"k&8ormm>al\": true, \"vruw{o;\": \"m~w:\\\\&l\"}}}, {}]"]]
          | QUERY ["SELECT array[(CASE WHEN col1437_1444 IS NULL THEN e'``W\\x0fk':::STRING ELSE 'gH':::STRING END)::STRING::STRING] AS c FROM ( VALUES('2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\"7QANfbm *87\": {\"/N5#\": []}, \"D+uB0Ui\": {\"93yB!q*%l2\": 1.1152646744807915, \"baz\": \"IR8m!_<K1\"}, \"V%viIi3\": [{\"baz\": {\"foo\": []}, \"fPfBqqYjf(F\": {}}], \"a\": {}, \"foo\": {\"b\": {\"K&8ORMM>al\": true, \"VRUw{O;\": \"m~W:\\\\\\\\&L\"}}}, {}]':::JSONB,701:::OID)) AS t(col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1438,col1437_1439);" []] :[["gH"]]
          | QUERY ["SELECT array[(CASE WHEN col1437_1444 IS NULL THEN e'?=\\f\\x1eEx':::STRING ELSE e'<\\x03bbX\\x19a':::STRING END)::STRING::STRING] AS c FROM ( VALUES('1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\"7QANfbm *87\": {\"/N5#\": []}, \"D+uB0Ui\": {\"93yB!q*%l2\": 1.1152646744807915, \"baz\": \"IR8m!_<K1\"}, \"V%viIi3\": [{\"baz\": {\"foo\": []}, \"fPfBqqYjf(F\": {}}], \"a\": {}, \"foo\": {\"b\": {\"K&8ORMM>al\": true, \"VRUw{O;\": \"m~W:\\\\\\\\&L\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP)) AS t(col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1438,col1437_1439,col1437_1440);" []] :[["<\x03bbX\x19a"]]
          | QUERY ["SELECT (((col1437_1443, col1437_1439, col1437_1447, col1437_1442 + (-2780600270711972420):::INT8, col1437_1445, col1437_1440, col1437_1441, col1437_1446))::STRING) IS NULL AS c FROM ( VALUES(e'[{\"7QANfbm *87\": {\"/N5#\": []}, \"D+uB0Ui\": {\"93yB!q*%l2\": 1.1152646744807915, \"baz\": \"IR8m!_<K1\"}, \"V%viIi3\": [{\"baz\": {\"foo\": []}, \"fPfBqqYjf(F\": {}}], \"a\": {}, \"foo\": {\"b\": {\"K&8ORMM>al\": true, \"VRUw{O;\": \"m~W:\\\\\\\\&L\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\"7qanfbm *87\": {\"/n5#\": []}, \"d+ub0ui\": {\"93yb!q*%l2\": 1.1152646744807915, \"baz\": \"ir8m!_<k1\"}, \"v%viii3\": [{\"baz\": {\"foo\": []}, \"fpfbqqyjf(f\": {}}], \"a\": {}, \"foo\": {\"b\": {\"k&8ormm>al\": true, \"vruw{o;\": \"m~w:\\\\\\\\&l\"}}}, {}]':::STRING,'gH':::STRING,e'<\\x03bbX\\x19a':::STRING)) AS t(col1437_1438,col1437_1439,col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1445,col1437_1446,col1437_1447);" []] :false
          | ===========================Previous statements [CREATE TYPE schema3287.enum3465 AS ENUM ('da') INSERT INTO public.table1437 (col1437_1438,col1437_1439,col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444) VALUES ('[[{}], [{"5%ZxBJy4&2o": true, "JnF6": null}, [], {"9jCt-4%Q": null}, 1.9554469664025482, "~/sDY>", {}], [{"m``d9k] ^P": 1.9124982004432292}, {}, {}], false]':::JSONB,1005:::OID,'1978-11-11 07:01:55.000027':::TIMESTAMP,NULL,(-3046394468882016911):::INT8,']xi'::STRING,'1973-10-16':::DATE),('[[[null, {"foobar": {}}], {"T>~q:": null}, {"D8(C;Laz<": {}, "foo": {}, "foobar": []}, true], {"}uEUn": true}]':::JSONB,2202:::OID,'-4713-11-24 00:00:00':::TIMESTAMP,'2009-04-27 21:35:11.00064+00:00':::TIMESTAMPTZ,(-5174918294687126107):::INT8,e'$n\x07d\x12\x1dr_+'::STRING,'1985-12-10':::DATE),(e'[{"7QANfbm *87": {"/N5#": []}, "D+uB0Ui": {"93yB!q*%l2": 1.1152646744807915, "baz": "IR8m!_<K1"}, "V%viIi3": [{"baz": {"foo": []}, "fPfBqqYjf(F": {}}], "a": {}, "foo": {"b": {"K&8ORMM>al": true, "VRUw{O;": "m~W:\\\\&L"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | Z,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\\\"7QANfbm *87\\\": {\\\"/N5#\\\": []}, \\\"D+uB0Ui\\\": {\\\"93yB!q*%l2\\\": 1.1152646744807915, \\\"baz\\\": \\\"IR8m!_\u003cK1\\\"}, \\\"V%viIi3\\\": [{\\\"baz\\\": {\\\"foo\\\": []}, \\\"fPfBqqYjf(F\\\": {}}], \\\"a\\\": {}, \\\"foo\\\": {\\\"b\\\": {\\\"K\u00268ORMM\u003eal\\\": true, \\\"VRUw{O;\\\": \\\"m~W:\\\\\\\\\\\\\\\\\u0026L\\\"}}}, {}]':::JSONB,701:::OID)) AS t(col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1438,col1437_1439);\" []] :[[\"gH\"]]\nQUERY [\"SELECT array[(CASE WHEN col1437_1444 IS NULL THEN e'?=\\\\f\\\\x1eEx':::STRING ELSE e'\u003c\\\\x03bbX\\\\x19a':::STRING END)::STRING::STRING] AS c FROM ( VALUES('1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\\\"7QANfbm *87\\\": {\\\"/N5#\\\": []}, \\\"D+uB0Ui\\\": {\\\"93yB!q*%l2\\\": 1.1152646744807915, \\\"baz\\\": \\\"IR8m!_\u003cK1\\\"}, \\\"V%viIi3\\\": [{\\\"baz\\\": {\\\"foo\\\": []}, \\\"fPfBqqYjf(F\\\": {}}], \\\"a\\\": {}, \\\"foo\\\": {\\\"b\\\": {\\\"K\u00268ORMM\u003eal\\\": true, \\\"VRUw{O;\\\": \\\"m~W:\\\\\\\\\\\\\\\\\u0026L\\\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP)) AS t(col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1438,col1437_1439,col1437_1440);\" []] :[[\"\u003c\\x03bbX\\x19a\"]]\nQUERY [\"SELECT (((col1437_1443, col1437_1439, col1437_1447, col1437_1442 + (-2780600270711972420):::INT8, col1437_1445, col1437_1440, col1437_1441, col1437_1446))::STRING) IS NULL AS c FROM ( VALUES(e'[{\\\"7QANfbm *87\\\": {\\\"/N5#\\\": []}, \\\"D+uB0Ui\\\": {\\\"93yB!q*%l2\\\": 1.1152646744807915, \\\"baz\\\": \\\"IR8m!_\u003cK1\\\"}, \\\"V%viIi3\\\": [{\\\"baz\\\": {\\\"foo\\\": []}, \\\"fPfBqqYjf(F\\\": {}}], \\\"a\\\": {}, \\\"foo\\\": {\\\"b\\\": {\\\"K\u00268ORMM\u003eal\\\": true, \\\"VRUw{O;\\\": \\\"m~W:\\\\\\\\\\\\\\\\\u0026L\\\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\\\"7qanfbm *87\\\": {\\\"/n5#\\\": []}, \\\"d+ub0ui\\\": {\\\"93yb!q*%l2\\\": 1.1152646744807915, \\\"baz\\\": \\\"ir8m!_\u003ck1\\\"}, \\\"v%viii3\\\": [{\\\"baz\\\": {\\\"foo\\\": []}, \\\"fpfbqqyjf(f\\\": {}}], \\\"a\\\": {}, \\\"foo\\\": {\\\"b\\\": {\\\"k\u00268ormm\u003eal\\\": true, \\\"vruw{o;\\\": \\\"m~w:\\\\\\\\\\\\\\\\\u0026l\\\"}}}, {}]':::STRING,'gH':::STRING,e'\u003c\\\\x03bbX\\\\x19a':::STRING)) AS t(col1437_1438,col1437_1439,col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1445,col1437_1446,col1437_1447);\" []] :false\n===========================Previous statements [CREATE TYPE schema3287.enum3465 AS ENUM ('da') INSERT INTO public.table1437 (col1437_1438,col1437_1439,col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444) VALUES ('[[{}], [{\"5%ZxBJy4\u00262o\": true, \"JnF6\": null}, [], {\"9jCt-4%Q\": null}, 1.9554469664025482, \"~/sDY\u003e\", {}], [{\"m``d9k] ^P\": 1.9124982004432292}, {}, {}], false]':::JSONB,1005:::OID,'1978-11-11 07:01:55.000027':::TIMESTAMP,NULL,(-3046394468882016911):::INT8,']xi'::STRING,'1973-10-16':::DATE),('[[[null, {\"foobar\": {}}], {\"T\u003e~q:\": null}, {\"D8(C;Laz\u003c\": {}, \"foo\": {}, \"foobar\": []}, true], {\"}uEUn\": true}]':::JSONB,2202:::OID,'-4713-11-24 00:00:00':::TIMESTAMP,'2009-04-27 21:35:11.00064+00:00':::TIMESTAMPTZ,(-5174918294687126107):::INT8,e'$n\\x07d\\x12\\x1dr_+'::STRING,'1985-12-10':::DATE),(e'[{\"7QANfbm *87\": {\"/N5#\": []}, \"D+uB0Ui\": {\"93yB!q*%l2\": 1.1152646744807915, \"baz\": \"IR8m!_\u003cK1\"}, \"V%viIi3\": [{\"baz\": {\"foo\": []}, \"fPfBqqYjf(F\": {}}], \"a\": {}, \"foo\": {\"b\": {\"K\u00268ORMM\u003eal\": true, \"VRUw{O;\": \"m~W:\\\\\\\\\u0026L\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 03d5260a467b8640de41ee08054c1ce05f91966c:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_131334.550474315_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_131334.550474315_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | TABLE schema1323.table3498 AS SELECT "IrrelevantColumnName", schema1487.table3124.col180_185, schema1487.table3124.col180_184, schema1487.table3124.col180_183, schema1487.table3124.col1201_1213, schema1487.table3124.col1201_1208, schema1487.table3124.col180_181, schema1487.table3124.col180_182, schema1487.table3124.col1201_1203, schema1487.table3124.col1201_1205, schema1487.table3124.col1201_1212, schema1487.table3124.col1201_1206, schema1487.table3124.col1201_1204, schema1487.table3124.col1201_1202, schema1487.table3124.col180_186, schema1487.table3124.col1201_1209, schema1487.table3124.col1201_1207, public.table72.col72_75 FROM schema3021.table3497, schema1487.table3124, public.table72 time:1.405343ms]
          | I220707 13:21:50.609828 213 workload/pgx_helpers.go:79  [-] 980  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema524.table3502" does not exist (SQLSTATE 42P01) pid:1693724 sql:INSERT INTO schema524.table3502 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:175.276332ms]
          | I220707 13:21:52.384812 219 workload/pgx_helpers.go:79  [-] 981  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3249.table3495" does not exist (SQLSTATE 42P01) pid:1448082 sql:ALTER TABLE schema3249.table3495 RENAME COLUMN "IrrelevantColumnName" TO "OtherIrrelevantName" time:2.199876ms]
          | I220707 13:21:53.111826 221 workload/pgx_helpers.go:79  [-] 982  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "seq3501" (886): expected matching namespace entry, found none (SQLSTATE XXUUU) pid:2086717 sql:commit time:263.093072ms]
          | Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:
          | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: ===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema1809.seq3501 INSERT INTO schema1487.table3124 (col180_185,col180_184,col180_186,col180_182,col180_181,col180_183,col1201_1208,col1201_1203,col1201_1213,col1201_1209,col1201_1204,col1201_1202,col1201_1205,col1201_1207,col1201_1206,col1201_1212) VALUES (NULL,17:::OID,'7/W^8'::STRING,B'11110101101010101001000110111111001011',(-1.6509901285171509):::FLOAT8,'\x00':::BYTES,(-1.582002878189087):::FLOAT8,NULL,''::STRING,NULL,5.632314407969060020E+23:::DECIMAL,'01050000E0E61000000200000001020000C008000000D8C755C4DAD853401C246E55AE0633C0AB04032F7680F0C18037C72FF101D741AE82D011E19065409C84A160D1B239C0943CE6628327F9418E82F872DA500242843F62C66F7E6140CC2F0879766B4F40C447261AAC57FE4104917AC3470BF9C1E8F87A21AF973D40141A06396AB94340160F6E1354070042624A878C3FA30042801F4699B0C9FB3F66876B3BEC4440405C7CB9ECB874FC41CC3A6D6D7058F641D092588EB51E21C0E06A2DE039433540D1846FA08699F2C100A0D710F955EA419277E50BA56154C0AC41D993DEF73A40B448F0F8EF90EC4185BBE5FBED90F6C1156B3305049B5CC0B43E2051971F3B406049B55702C5C4C180443E9F35C0B7C101020000C002000000DD1457968EBE50C08B3CBF07482E56C0A82148463003D9C12336CD57187DFEC1800D11CC2ADB2D40CF41D5A7C9A950C090EED075DDB0CBC1E8B4062CB3560242':::GEOGRAPHY,'\x9c9b2d48':::BYTES,26:::OID,(-0.8428827144913749):::FLOAT8,e',\' '::STRING),('18:49:08.988719':::TIME,1186:::OID,e'u\x1a\b'::STRING,B'11111100110101100101010110110010111000',NULL,'\x435923fc88ac13':::BYTES,0.6958065032958984:::FLOAT8,''::NAME:::NAME,e'g\x05pEG\x1d,\x03\x19'::STRING,ARRAY['EZZ':::STRING:::NAME,e'\x11qO\x19#p':::STRING:::NAME],8.924687560078392295E+20:::DECIMAL,'01040000E0E61000000400000001010000C040E2BA2BB3B53140A8F644901D11504074581180C5EF01422A8C02C4A686F04101010000C04727831D17E054C090B8184DB93B504010A2F7F5419AFBC1F65792F8EDD8F1C101010000C056E73B71EF466140D0FF1F760B264F40748672390D02F9414A5BC295C670E7C101010000C0A02EF3A4F3F2204040217795C5304D4020768B48A887F741E60A7EE68383F0C1':::GEOGRAPHY,'\x200953d461a9cb96':::BYTES,90004:::OID,(-1.005883226858427):::FLOAT8,e'%+!;6H\x0f\x10'::STRING) SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()]: ERROR: relation "seq3501" (886): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 327F9418E82F872DA500242843F62C66F7E6140CC2F0879766B4F40C447261AAC57FE4104917AC3470BF9C1E8F87A21AF973D40141A06396AB94340160F6E1354070042624A878C3FA30042801F4699B0C9FB3F66876B3BEC4440405C7CB9ECB874FC41CC3A6D6D7058F641D092588EB51E21C0E06A2DE039433540D1846FA08699F2C100A0D710F955EA419277E50BA56154C0AC41D993DEF73A40B448F0F8EF90EC4185BBE5FBED90F6C1156B3305049B5CC0B43E2051971F3B406049B55702C5C4C180443E9F35C0B7C101020000C002000000DD1457968EBE50C08B3CBF07482E56C0A82148463003D9C12336CD57187DFEC1800D11CC2ADB2D40CF41D5A7C9A950C090EED075DDB0CBC1E8B4062CB3560242':::GEOGRAPHY,'\\x9c9b2d48':::BYTES,26:::OID,(-0.8428827144913749):::FLOAT8,e',\\' '::STRING),('18:49:08.988719':::TIME,1186:::OID,e'u\\x1a\\b'::STRING,B'11111100110101100101010110110010111000',NULL,'\\x435923fc88ac13':::BYTES,0.6958065032958984:::FLOAT8,''::NAME:::NAME,e'g\\x05pEG\\x1d,\\x03\\x19'::STRING,ARRAY['EZZ':::STRING:::NAME,e'\\x11qO\\x19#p':::STRING:::NAME],8.924687560078392295E+20:::DECIMAL,'01040000E0E61000000400000001010000C040E2BA2BB3B53140A8F644901D11504074581180C5EF01422A8C02C4A686F04101010000C04727831D17E054C090B8184DB93B504010A2F7F5419AFBC1F65792F8EDD8F1C101010000C056E73B71EF466140D0FF1F760B264F40748672390D02F9414A5BC295C670E7C101010000C0A02EF3A4F3F2204040217795C5304D4020768B48A887F741E60A7EE68383F0C1':::GEOGRAPHY,'\\x200953d461a9cb96':::BYTES,90004:::OID,(-1.005883226858427):::FLOAT8,e'%+!;6H\\x0f\\x10'::STRING)",
          |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: ===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema1809.seq3501 INSERT INTO schema1487.table3124 (col180_185,col180_184,col180_186,col180_182,col180_181,col180_183,col1201_1208,col1201_1203,col1201_1213,col1201_1209,col1201_1204,col1201_1202,col1201_1205,col1201_1207,col1201_1206,col1201_1212) VALUES (NULL,17:::OID,'7/W^8'::STRING,B'11110101101010101001000110111111001011',(-1.6509901285171509):::FLOAT8,'\\x00':::BYTES,(-1.582002878189087):::FLOAT8,NULL,''::STRING,NULL,5.632314407969060020E+23:::DECIMAL,'01050000E0E61000000200000001020000C008000000D8C755C4DAD853401C246E55AE0633C0AB04032F7680F0C18037C72FF101D741AE82D011E19065409C84A160D1B239C0943CE6628327F9418E82F872DA500242843F62C66F7E6140CC2F0879766B4F40C447261AAC57FE4104917AC3470BF9C1E8F87A21AF973D40141A06396AB94340160F6E1354070042624A878C3FA30042801F4699B0C9FB3F66876B3BEC4440405C7CB9ECB874FC41CC3A6D6D7058F641D092588EB51E21C0E06A2DE039433540D1846FA08699F2C100A0D710F955EA419277E50BA56154C0AC41D993DEF73A40B448F0F8EF90EC4185BBE5FBED90F6C1156B3305049B5CC0B43E2051971F3B406049B55702C5C4C180443E9F35C0B7C101020000C002000000DD1457968EBE50C08B3CBF07482E56C0A82148463003D9C12336CD57187DFEC1800D11CC2ADB2D40CF41D5A7C9A950C090EED075DDB0CBC1E8B4062CB3560242':::GEOGRAPHY,'\\x9c9b2d48':::BYTES,26:::OID,(-0.8428827144913749):::FLOAT8,e',\\' '::STRING),('18:49:08.988719':::TIME,1186:::OID,e'u\\x1a\\b'::STRING,B'11111100110101100101010110110010111000',NULL,'\\x435923fc88ac13':::BYTES,0.6958065032958984:::FLOAT8,''::NAME:::NAME,e'g\\x05pEG\\x1d,\\x03\\x19'::STRING,ARRAY['EZZ':::STRING:::NAME,e'\\x11qO\\x19#p':::STRING:::NAME],8.924687560078392295E+20:::DECIMAL,'01040000E0E61000000400000001010000C040E2BA2BB3B53140A8F644901D11504074581180C5EF01422A8C02C4A686F04101010000C04727831D17E054C090B8184DB93B504010A2F7F5419AFBC1F65792F8EDD8F1C101010000C056E73B71EF466140D0FF1F760B264F40748672390D02F9414A5BC295C670E7C101010000C0A02EF3A4F3F2204040217795C5304D4020768B48A887F741E60A7EE68383F0C1':::GEOGRAPHY,'\\x200953d461a9cb96':::BYTES,90004:::OID,(-1.005883226858427):::FLOAT8,e'%+!;6H\\x0f\\x10'::STRING) SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()]: ERROR: relation \"seq3501\" (886): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 6374bd83f192bd509e50313acb431690b9c3f8ae:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_125958.784508835_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_125958.784508835_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 798, schema3074.table3756.col791_799, schema3074.table3756.col791_801 FROM schema1377.table1895, schema1470.table1789, schema3074.table3756 time:420.426951ms]
          | I220710 13:12:54.503558 219 workload/pgx_helpers.go:79  [-] 1019  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema320.table3760" does not exist (SQLSTATE 42P01) pid:1473806 sql:INSERT INTO schema320.table3760 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:645.140776ms]
          | I220710 13:13:01.622696 217 workload/pgx_helpers.go:79  [-] 1020  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: command is too large: 76517260 bytes (max: 67108864) (SQLSTATE XXUUU) pid:1610991 sql:CREATE TABLE schema3074.table3379 AS SELECT schema320.table998.col998_1000, schema320.table998.col998_1012, schema320.table998.col998_1009, schema1864.table2162.col94_102, public.table791.col791_801, public.table791.col791_795, public.table791.col791_803 FROM schema320.table998, schema1864.table2162, public.table791 time:2m1.258635508s]
          | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
          | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema320" "table998"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema1864" "table2162"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["public" "table791"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT schema_name\n\t\tFROM information_schema.schemata\n   WHERE schema_name = $1\n\t)" ["schema3074"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema3074" "table3379"]] :false
          | ===========================Previous statements [INSERT INTO schema1377.table1895 (col998_1004,col998_1000,col998_1008,col998_1011,col998_1005,col998_1002,col619_624,col619_629,col619_625,col619_620,col94_95,col94_101,col94_103,col94_97,col94_104,col94_99,col94_96) VALUES ((-696188699):::INT8,'11:14:49.454686-07:14:00':::TIMETZ,'8``<'::STRING,e'Z\r\x1e\x1dZ}FG'::STRING,ARRAY['3a3cd0cb-521d-409c-a898-bd2a8afcf300':::UUID],e'\x18'::"char",90000:::OID,'05:14:32.31676':::TIME,'1986-04-15 20:35:43.000845':::TIMESTAMP,false,'0107000040020000000101000040EC604F0AECDCFC41007E15ECEA94FE41A0684E824462FD41010400004006000000010100004016463B769AC6F2C12F23BAE2C4F1FDC193AD289D8DACFAC10101000040CC7880A9328EE9C14A56AC04F89501C2C0883A51297702420101000040165956328D62F4C110D875FC867BDE41366C309E6152F0410101000040BAF400D8D52AF9C1507F2B5CD5C6014280EE96EA9A4DD2C101010000407037928DA1A9CD41A84C7FC39747DFC13031F0C80F96D3C101010000402864A66658AFF5C1E4ED038EF544FE41D133FBEEF52AFEC1':::GEOMETRY,0.32986159773781043:::FLOAT8,0.39415409726750117:::FLOAT8,1.4332882165908813:::FLOAT8,(-0.014903925608096669):::FLOAT8,(-1.1400883197784424):::FLOAT8,'0103000020E6100000010000000D000000C83DA5BD33F746C0484D4DACEF7430C0A54C7CCCF24B5EC034F3A49943954CC0B8C58FF6482E42C099B78D2B426844C0802DF9517CFCFB3FE0066F6871BC47C03C19B18532C34F406CEC2D93DC2354C0C6C0F279146050408846A8BC88E052C09061F4792E71574048661376BA0C2340B602EBBDBFCF6240088ACB47A4A43440F4B6844936D8424028D78AD376984C40F071FE691B59264070D10F1123D154409988036663C361C0D6D199755F9555402F68E5C6982C66C0F4AFF45EF4054040C83DA5BD33F746C0484D4DACEF7430C0':::GEOGRAPHY) CREATE TABLE schema3074.table3379 AS SELECT schema320.table998.col998_1000, schema320.table998.col998_1012, schema320.table998.col998_1009, schema1864.table2162.col94_102, public.table791.col791_801, public.table791.col791_795, public.table791.col791_803 FROM schema320.table998, schema1864.table2162, public.table791]: ERROR: command is too large: 76517260 bytes (max: 67108864) (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 13:::FLOAT8,(-0.014903925608096669):::FLOAT8,(-1.1400883197784424):::FLOAT8,'0103000020E6100000010000000D000000C83DA5BD33F746C0484D4DACEF7430C0A54C7CCCF24B5EC034F3A49943954CC0B8C58FF6482E42C099B78D2B426844C0802DF9517CFCFB3FE0066F6871BC47C03C19B18532C34F406CEC2D93DC2354C0C6C0F279146050408846A8BC88E052C09061F4792E71574048661376BA0C2340B602EBBDBFCF6240088ACB47A4A43440F4B6844936D8424028D78AD376984C40F071FE691B59264070D10F1123D154409988036663C361C0D6D199755F9555402F68E5C6982C66C0F4AFF45EF4054040C83DA5BD33F746C0484D4DACEF7430C0':::GEOGRAPHY)",
          |   "CREATE TABLE schema3074.table3379 AS SELECT schema320.table998.col998_1000, schema320.table998.col998_1012, schema320.table998.col998_1009, schema1864.table2162.col94_102, public.table791.col791_801, public.table791.col791_795, public.table791.col791_803 FROM schema320.table998, schema1864.table2162, public.table791"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema320\" \"table998\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema1864\" \"table2162\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table791\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema3074\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3074\" \"table3379\"]] :false\n===========================Previous statements [INSERT INTO schema1377.table1895 (col998_1004,col998_1000,col998_1008,col998_1011,col998_1005,col998_1002,col619_624,col619_629,col619_625,col619_620,col94_95,col94_101,col94_103,col94_97,col94_104,col94_99,col94_96) VALUES ((-696188699):::INT8,'11:14:49.454686-07:14:00':::TIMETZ,'8``\u003c'::STRING,e'Z\\r\\x1e\\x1dZ}FG'::STRING,ARRAY['3a3cd0cb-521d-409c-a898-bd2a8afcf300':::UUID],e'\\x18'::\"char\",90000:::OID,'05:14:32.31676':::TIME,'1986-04-15 20:35:43.000845':::TIMESTAMP,false,'0107000040020000000101000040EC604F0AECDCFC41007E15ECEA94FE41A0684E824462FD41010400004006000000010100004016463B769AC6F2C12F23BAE2C4F1FDC193AD289D8DACFAC10101000040CC7880A9328EE9C14A56AC04F89501C2C0883A51297702420101000040165956328D62F4C110D875FC867BDE41366C309E6152F0410101000040BAF400D8D52AF9C1507F2B5CD5C6014280EE96EA9A4DD2C101010000407037928DA1A9CD41A84C7FC39747DFC13031F0C80F96D3C101010000402864A66658AFF5C1E4ED038EF544FE41D133FBEEF52AFEC1':::GEOMETRY,0.32986159773781043:::FLOAT8,0.39415409726750117:::FLOAT8,1.4332882165908813:::FLOAT8,(-0.014903925608096669):::FLOAT8,(-1.1400883197784424):::FLOAT8,'0103000020E6100000010000000D000000C83DA5BD33F746C0484D4DACEF7430C0A54C7CCCF24B5EC034F3A49943954CC0B8C58FF6482E42C099B78D2B426844C0802DF9517CFCFB3FE0066F6871BC47C03C19B18532C34F406CEC2D93DC2354C0C6C0F279146050408846A8BC88E052C09061F4792E71574048661376BA0C2340B602EBBDBFCF6240088ACB47A4A43440F4B6844936D8424028D78AD376984C40F071FE691B59264070D10F1123D154409988036663C361C0D6D199755F9555402F68E5C6982C66C0F4AFF45EF4054040C83DA5BD33F746C0484D4DACEF7430C0':::GEOGRAPHY) CREATE TABLE schema3074.table3379 AS SELECT schema320.table998.col998_1000, schema320.table998.col998_1012, schema320.table998.col998_1009, schema1864.table2162.col94_102, public.table791.col791_801, public.table791.col791_795, public.table791.col791_803 FROM schema320.table998, schema1864.table2162, public.table791]: ERROR: command is too large: 76517260 bytes (max: 67108864) (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 86e007dcb5cbde3501f138c1d768519db3487857:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_125752.352067431_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_125752.352067431_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 0103000020E61000000100000004000000B0D170B1610738C064D2E19FE0E139C09C2834EC1406574018DE66554F5631C069F66A83341360C0CA3FBD1D39F95440B0D170B1610738C064D2E19FE0E139C00105000020E610000006000000010200000003000000BCE7B97429F763C07409BA8DB9F25340704175B927B365C0FAC54AA1310256402EEE8CBEE10C58C0885C4AE641452E4001020000000300000094ED5B12925755407835053E35B83240104C01E1BC6662C0F8C95868263C534064FD6A32D5B860C0643436DAE6BA3F4001020000000800000046036576BC5F4BC0966B7D4782684EC0FCD5819D26735B4033C24CB7DEE152C04CC953C186EC514038EA55AB6E223FC0DCBC8F3C8C616140F7F4E95FC64642C0B045E7F4246C5F409AD9E7024D843EC086D23F6C4AEB6040603F70C6D00C15C000E9F9F23B540C40189E94957B314A408EC492CBCF6861C04A138ED0B5EF5140010200000002000000100E68721AB42D40CCA759A99EFD454080605376B9F234C09A9CB143CD33544001020000000300000090D8410CD85F2BC0592053C86F3A51C0BC3AE4D03383594088C3F2B9A4BD44C0EECFE37451B8634074901788195249C001020000000300000065C6486463E05BC0224209BC1AA14DC070507BAC13EC2FC06FD6743BF6B84CC014060FFFED134840C14BBA26C4D251C00106000020E6100000040000000103000000010000000B0000009CC8C65B8ADE58C014DC09ED2A3C54C0D83FB51098945B40DE729D290B5B53C0D8B3BAA867FD514060F56E6278131CC092A57444ADB5644010F7C4FC99101B40B84920DADE1E6140708D7A9F4E8821405872A9CDA08E5740B009151CDFC55040792E8868305465C04C769F2AA6EB5540DC0665BEAF925AC0A4E540A7EB8049400CEF272C310962C04800334E091841406BF817F9468E62C0C0D509C3ED902A409CC8C65B8ADE58C014DC09ED2A3C54C001030000000100000005000000000DBB3909AA114040B1AF96A1A03BC00C2F5BC998C55240655BC1A653DD41C0CC7210D4DB855E4084E76FC27BA83040686C8D3099D162C0007E550B29654B40000DBB3909AA114040B1AF96A1A03BC00103000000010000000500000000E111871FF8294052CEB06AD67B50C03877DE107CAD5B4047465A4F8B9455C0586FD4F591EB56405E16E489A2A846C07D2C4F05D97551C0E8C538EBC2AD444000E111871FF8294052CEB06AD67B50C0010300000001000000070000007A15058EDF2D62C070876A7D7FF811C0B84703B38A7866C09140270B35D955C000358B1B9CDF034073D083F9ACC355C0386793E1EC923740A8B7884655E04840F8C41679A30B46C0200E44CD6ED336406048C9AF28BB28C00E714FF9726156407A15058EDF2D62C070876A7D7FF811C00103000020E61000000100000006000000B42CFE0341FE49C0C819F3FF139031C0D4D19BB73C1D5140D48CF9D5C4CF4DC058DF313065863D4058A3BCAB506127C080BB7ACCA34C5840A08E7F12FCE92E4080043E04A96528C0143DC41DCE683D40B42CFE0341FE49C0C819F3FF139031C00102000020E610000004000000B4E9FC1AB14F554010D97AB8A76B1A4024264FD309655340760E5AE3DA9746C0D8865CAABFE85940B8AE3609E8302DC0CC2BF0B9324F6040009C6F9A00C6F8BF0102000020E61000000300000064CD7C2CEE224FC00A2639362F6042C07CA6FE325A5258401CA9EA145C4F51C0C8602468B143604024C163ED56FD46C0':::GEOGRAPHY,'-74 years -8 mons -8 days -00:29:52.117558':::INTERVAL,90004:::OID,869:::OID,ARRAY[7763035483519866360:::INT8,2147483647:::INT8],2951:::OID),('0101000020E6100000DCCFB6F269475040D899CDC9C1013B40':::GEOGRAPHY,'63 years 2 mons 535 days 21:55:46.356649':::INTERVAL,1184:::OID,20:::OID,ARRAY[(-1402262343001151102):::INT8],1043:::OID) CREATE TABLE IF NOT EXISTS schema2821.table2877 (col2877_2879 "char" NOT NULL, col2877_2880 UUID NOT NULL, col2877_2881 DATE NOT NULL, col2877_2882 REGROLE NULL, col2877_2883 INT8 NOT NULL, col2877_2884 GEOMETRY, col2877_2885 VARCHAR NOT NULL, col2877_2886 DECIMAL NOT NULL, col2877_2887 INT4 NULL, col2877_2888 STRING, col2877_2889 REGNAMESPACE NULL, col2877_2890 REGPROC, col2877_2891 GEOGRAPHY NOT NULL, col2877_2892 DECIMAL NOT NULL AS (col2877_2886 + (-7.597922143869865637E+22):::DECIMAL) VIRTUAL, col2877_2893 INT8 NULL AS (col2877_2887 + (-532726378):::INT8) STORED, col2877_2894 STRING NULL AS (CASE WHEN col2877_2889 IS NULL THEN e'CbS\x1c\x14x':::STRING ELSE e'<,;aA\b\x05':::STRING END) VIRTUAL) SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema1281.table2726 (col358_362,col358_366,col358_360,col358_364,col358_365,col358_361,col358_363,col479_480) VALUES (1000:::OID,NULL,'-43 years -10 mons -603 days -10:45:44.351933':::INTERVAL,1186:::OID,e'\U00002603'::STRING,19:::OID,ARRAY[6020803765411495318:::INT8,(-6918499188159710745):::INT8],'1971-01-26':::DATE)]: ERROR: relation "seq2871" (804): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 00100000004000000B0D170B1610738C064D2E19FE0E139C09C2834EC1406574018DE66554F5631C069F66A83341360C0CA3FBD1D39F95440B0D170B1610738C064D2E19FE0E139C00105000020E610000006000000010200000003000000BCE7B97429F763C07409BA8DB9F25340704175B927B365C0FAC54AA1310256402EEE8CBEE10C58C0885C4AE641452E4001020000000300000094ED5B12925755407835053E35B83240104C01E1BC6662C0F8C95868263C534064FD6A32D5B860C0643436DAE6BA3F4001020000000800000046036576BC5F4BC0966B7D4782684EC0FCD5819D26735B4033C24CB7DEE152C04CC953C186EC514038EA55AB6E223FC0DCBC8F3C8C616140F7F4E95FC64642C0B045E7F4246C5F409AD9E7024D843EC086D23F6C4AEB6040603F70C6D00C15C000E9F9F23B540C40189E94957B314A408EC492CBCF6861C04A138ED0B5EF5140010200000002000000100E68721AB42D40CCA759A99EFD454080605376B9F234C09A9CB143CD33544001020000000300000090D8410CD85F2BC0592053C86F3A51C0BC3AE4D03383594088C3F2B9A4BD44C0EECFE37451B8634074901788195249C001020000000300000065C6486463E05BC0224209BC1AA14DC070507BAC13EC2FC06FD6743BF6B84CC014060FFFED134840C14BBA26C4D251C00106000020E6100000040000000103000000010000000B0000009CC8C65B8ADE58C014DC09ED2A3C54C0D83FB51098945B40DE729D290B5B53C0D8B3BAA867FD514060F56E6278131CC092A57444ADB5644010F7C4FC99101B40B84920DADE1E6140708D7A9F4E8821405872A9CDA08E5740B009151CDFC55040792E8868305465C04C769F2AA6EB5540DC0665BEAF925AC0A4E540A7EB8049400CEF272C310962C04800334E091841406BF817F9468E62C0C0D509C3ED902A409CC8C65B8ADE58C014DC09ED2A3C54C001030000000100000005000000000DBB3909AA114040B1AF96A1A03BC00C2F5BC998C55240655BC1A653DD41C0CC7210D4DB855E4084E76FC27BA83040686C8D3099D162C0007E550B29654B40000DBB3909AA114040B1AF96A1A03BC00103000000010000000500000000E111871FF8294052CEB06AD67B50C03877DE107CAD5B4047465A4F8B9455C0586FD4F591EB56405E16E489A2A846C07D2C4F05D97551C0E8C538EBC2AD444000E111871FF8294052CEB06AD67B50C0010300000001000000070000007A15058EDF2D62C070876A7D7FF811C0B84703B38A7866C09140270B35D955C000358B1B9CDF034073D083F9ACC355C0386793E1EC923740A8B7884655E04840F8C41679A30B46C0200E44CD6ED336406048C9AF28BB28C00E714FF9726156407A15058EDF2D62C070876A7D7FF811C00103000020E61000000100000006000000B42CFE0341FE49C0C819F3FF139031C0D4D19BB73C1D5140D48CF9D5C4CF4DC058DF313065863D4058A3BCAB506127C080BB7ACCA34C5840A08E7F12FCE92E4080043E04A96528C0143DC41DCE683D40B42CFE0341FE49C0C819F3FF139031C00102000020E610000004000000B4E9FC1AB14F554010D97AB8A76B1A4024264FD309655340760E5AE3DA9746C0D8865CAABFE85940B8AE3609E8302DC0CC2BF0B9324F6040009C6F9A00C6F8BF0102000020E61000000300000064CD7C2CEE224FC00A2639362F6042C07CA6FE325A5258401CA9EA145C4F51C0C8602468B143604024C163ED56FD46C0':::GEOGRAPHY,'-74 years -8 mons -8 days -00:29:52.117558':::INTERVAL,90004:::OID,869:::OID,ARRAY[7763035483519866360:::INT8,2147483647:::INT8],2951:::OID),('0101000020E6100000DCCFB6F269475040D899CDC9C1013B40':::GEOGRAPHY,'63 years 2 mons 535 days 21:55:46.356649':::INTERVAL,1184:::OID,20:::OID,ARRAY[(-1402262343001151102):::INT8],1043:::OID) CREATE TABLE IF NOT EXISTS schema2821.table2877 (col2877_2879 \"char\" NOT NULL, col2877_2880 UUID NOT NULL, col2877_2881 DATE NOT NULL, col2877_2882 REGROLE NULL, col2877_2883 INT8 NOT NULL, col2877_2884 GEOMETRY, col2877_2885 VARCHAR NOT NULL, col2877_2886 DECIMAL NOT NULL, col2877_2887 INT4 NULL, col2877_2888 STRING, col2877_2889 REGNAMESPACE NULL, col2877_2890 REGPROC, col2877_2891 GEOGRAPHY NOT NULL, col2877_2892 DECIMAL NOT NULL AS (col2877_2886 + (-7.597922143869865637E+22):::DECIMAL) VIRTUAL, col2877_2893 INT8 NULL AS (col2877_2887 + (-532726378):::INT8) STORED, col2877_2894 STRING NULL AS (CASE WHEN col2877_2889 IS NULL THEN e'CbS\\x1c\\x14x':::STRING ELSE e'\u003c,;aA\\b\\x05':::STRING END) VIRTUAL) SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema1281.table2726 (col358_362,col358_366,col358_360,col358_364,col358_365,col358_361,col358_363,col479_480) VALUES (1000:::OID,NULL,'-43 years -10 mons -603 days -10:45:44.351933':::INTERVAL,1186:::OID,e'\\U00002603'::STRING,19:::OID,ARRAY[6020803765411495318:::INT8,(-6918499188159710745):::INT8],'1971-01-26':::DATE)]: ERROR: relation \"seq2871\" (804): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ e9ee21860458d997a8155734dc608cfcd050ef24:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_122518.447822259_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_122518.447822259_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | (VALUES( 19:::OID,24:::OID,e'\\x14'::\"char\",e':\\x07v#;\\x7fO':::STRING,0.48219192028045654:::FLOAT8,1043:::OID,'47.206.210.205/7':::INET,1014:::OID,'0.3708638548851013'::FLOAT8,'infinity':::DATE,18:::OID) ) AS T(col1449_1457,col1449_1450,col1449_1460,col1449_1464,col1449_1459,col1449_1461,col1449_1463,col1449_1456,col1449_1465,col1449_1454,col1449_1452) ) )" []] :false
          | QUERY ["SELECT array[((col1449_1450, col1449_1460, col1449_1464, col1449_1456, col1449_1465, col1449_1454, col1449_1452, col1449_1459, col1449_1461, col1449_1463, col1449_1457))::STRING] FROM (VALUES(19:::OID,24:::OID,e'\\x14'::\"char\",e':\\x07v#;\\x7fO':::STRING,0.48219192028045654:::FLOAT8,1043:::OID,'47.206.210.205/7':::INET,1014:::OID,'0.3708638548851013'::FLOAT8,'infinity':::DATE,18:::OID) ) AS T(col1449_1457,col1449_1450,col1449_1460,col1449_1464,col1449_1459,col1449_1461,col1449_1463,col1449_1456,col1449_1465,col1449_1454,col1449_1452)" []] :[["(24,\x14,:\av#;\u007fO,1014,0.3708638548851013,infinity,18,0.48219192028045654,1043,47.206.210.205/7,19)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema1251.table1449'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema1251' AND\n\t\t\t\t\t\t\t\t    table_name='table1449' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema1251.table1449', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema1251'\n\t\t           AND table_name = 'table1449'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[["schema1251" "table1449" "col1449_1452" "col1449_1452"]]
          | ===========================Previous statements [INSERT INTO schema1251.table1449 (col1449_1450,col1449_1451,col1449_1452,col1449_1453,col1449_1454,col1449_1455,col1449_1456,col1449_1457,col1449_1458,col1449_1459,col1449_1460,col1449_1461,col1449_1462,col1449_1463) VALUES (2205:::OID,24:::OID,17:::OID,NULL,'1977-08-18':::DATE,B'01111111111111111111111111111111111111111111111',90004:::OID,1114:::OID,24:::OID,(-0.3453367352485657):::FLOAT8,'5'::"char",90002:::OID,'\x95f6db6933b67c4ab4':::BYTES,'d01:835c:62d7:6c27:881f:37d6:1faf:e314/69':::INET),(1266:::OID,1700:::OID,1700:::OID,'22:53:21.995695':::TIME,'1972-01-16':::DATE,B'10110000000011101110101111010011001111111100110',23:::OID,1560:::OID,3802:::OID,(-0.09712880849838257):::FLOAT8,'g'::"char",2950:::OID,'\x73446952616f':::BYTES,'140.99.124.168/9':::INET) INSERT INTO schema1251.table1449 (col1449_1450,col1449_1451,col1449_1452,col1449_1453,col1449_1454,col1449_1455,col1449_1456,col1449_1457,col1449_1458,col1449_1459,col1449_1460,col1449_1461,col1449_1462,col1449_1463) VALUES (24:::OID,18:::OID,18:::OID,'11:00:19.220298':::TIME,'infinity':::DATE,B'10011100011010111110111011011010011001110001111',1014:::OID,19:::OID,18:::OID,0.48219192028045654:::FLOAT8,e'\x14'::"char",1043:::OID,'\xeb':::BYTES,'47.206.210.205/7':::INET)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 08638548851013'::FLOAT8,'infinity':::DATE,18:::OID) ) AS T(col1449_1457,col1449_1450,col1449_1460,col1449_1464,col1449_1459,col1449_1461,col1449_1463,col1449_1456,col1449_1465,col1449_1454,col1449_1452) ) )\" []] :false\nQUERY [\"SELECT array[((col1449_1450, col1449_1460, col1449_1464, col1449_1456, col1449_1465, col1449_1454, col1449_1452, col1449_1459, col1449_1461, col1449_1463, col1449_1457))::STRING] FROM (VALUES(19:::OID,24:::OID,e'\\\\x14'::\\\"char\\\",e':\\\\x07v#;\\\\x7fO':::STRING,0.48219192028045654:::FLOAT8,1043:::OID,'47.206.210.205/7':::INET,1014:::OID,'0.3708638548851013'::FLOAT8,'infinity':::DATE,18:::OID) ) AS T(col1449_1457,col1449_1450,col1449_1460,col1449_1464,col1449_1459,col1449_1461,col1449_1463,col1449_1456,col1449_1465,col1449_1454,col1449_1452)\" []] :[[\"(24,\\x14,:\\av#;\\u007fO,1014,0.3708638548851013,infinity,18,0.48219192028045654,1043,47.206.210.205/7,19)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema1251.table1449'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema1251' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table1449' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema1251.table1449', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema1251'\\n\\t\\t           AND table_name = 'table1449'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[[\"schema1251\" \"table1449\" \"col1449_1452\" \"col1449_1452\"]]\n===========================Previous statements [INSERT INTO schema1251.table1449 (col1449_1450,col1449_1451,col1449_1452,col1449_1453,col1449_1454,col1449_1455,col1449_1456,col1449_1457,col1449_1458,col1449_1459,col1449_1460,col1449_1461,col1449_1462,col1449_1463) VALUES (2205:::OID,24:::OID,17:::OID,NULL,'1977-08-18':::DATE,B'01111111111111111111111111111111111111111111111',90004:::OID,1114:::OID,24:::OID,(-0.3453367352485657):::FLOAT8,'5'::\"char\",90002:::OID,'\\x95f6db6933b67c4ab4':::BYTES,'d01:835c:62d7:6c27:881f:37d6:1faf:e314/69':::INET),(1266:::OID,1700:::OID,1700:::OID,'22:53:21.995695':::TIME,'1972-01-16':::DATE,B'10110000000011101110101111010011001111111100110',23:::OID,1560:::OID,3802:::OID,(-0.09712880849838257):::FLOAT8,'g'::\"char\",2950:::OID,'\\x73446952616f':::BYTES,'140.99.124.168/9':::INET) INSERT INTO schema1251.table1449 (col1449_1450,col1449_1451,col1449_1452,col1449_1453,col1449_1454,col1449_1455,col1449_1456,col1449_1457,col1449_1458,col1449_1459,col1449_1460,col1449_1461,col1449_1462,col1449_1463) VALUES (24:::OID,18:::OID,18:::OID,'11:00:19.220298':::TIME,'infinity':::DATE,B'10011100011010111110111011011010011001110001111',1014:::OID,19:::OID,18:::OID,0.48219192028045654:::FLOAT8,e'\\x14'::\"char\",1043:::OID,'\\xeb':::BYTES,'47.206.210.205/7':::INET)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 0e816efe5443764591a944edcb385c7b2762523e:

        Wraps: (2) output in run_131023.344529848_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | -] 1195  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot add region "europe-west2" to database schemachange (SQLSTATE 42P12) pid:1488520 sql:ALTER DATABASE schemachange ADD REGION "europe-west2" time:64.34414ms]
          | I220716 13:16:05.773356 239 workload/pgx_helpers.go:79  [-] 1196  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot drop schema "public" (SQLSTATE 3F000) pid:1527331 sql:DROP SCHEMA "public" CASCADE time:1.338759ms]
          | I220716 13:16:05.887759 226 workload/pgx_helpers.go:79  [-] 1197  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: type "schemachange.schema1605.enum2798" already exists (SQLSTATE 42710) pid:2032719 sql:CREATE TYPE schema1605.enum2798 AS ENUM ('a') time:528.728157ms]
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198  pgx logger [error]: Query logParams=map[args:[public.table3548] err:ERROR: error in argument for $1: relation "public.table3548" does not exist (SQLSTATE 42P01) pid:1226535 sql:
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +SELECT array_agg(index_name)
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +  FROM (
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +SELECT
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +  index_name
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +FROM
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +  crdb_internal.table_indexes
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +WHERE
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +  index_type = 'primary'
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +  AND descriptor_id = $1::REGCLASS
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +       );
          | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +  ]
          | I220716 13:16:06.720284 232 workload/pgx_helpers.go:79  [-] 1199  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: duplicate key value violates unique constraint "table3231_col3231_3240_col3231_3237_key" (SQLSTATE 23505) pid:1218423 sql:INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1983-07-28 23:42:56.000958':::TIMESTAMP,2202:::OID,''::STRING,'BOX(-0.8028798948299167 0.4866737668766704,0.25842982473176523 1.256152449899192)':::BOX2D,'{"b": [true, [{}, {"c": []}, true]], "bar": null, "baz": [[]], "foo": "$Kk7U&Sf,/", "|iHKGe&Y|)i": 0.21917494336233792}':::JSONB,90002:::OID) time:134.01211ms]
          | Error: ***UNEXPECTED ERROR; Failed to generate a random operation
          |  OpGen log: 
          | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["public" "table3548"]] :true
          | Stmts: 
          | [INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1992-05-03 17:51:29.00024':::TIMESTAMP,869:::OID,e'\x1bo|QL>'::STRING,'BOX(-1.48848753568185 0.11413130096700558,-0.5829904756554781 1.2114179120739301)':::BOX2D,e'[{"\'-{\\"#qb$": [], "c": {"Zo&U": {"(;19~W5": 0.3397086645906202, "a": []}, "bar": {"{![Cg": {}}}, "foo": {"b": {}, "c": {"H)a7I": []}}}, {"Rf\\\\X": false, "}p>&I\'AiYD,{": {}}]':::JSONB,90004:::OID) INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1971-02-05 02:00:12.000346':::TIMESTAMP,1043:::OID,e'[rO\x15iq<W~'::STRING,'BOX(-0.9235027037328147 -1.2006760750506533,-0.8793129890189155 0.17191664750169722)':::BOX2D,'[[{}], [{"b": {}, "bar": {}, "u/RSD.r``}": [true]}, []], [{}], []]':::JSONB,17:::OID),('1974-08-28 05:54:34.000015':::TIMESTAMP,19:::OID,''::STRING,NULL,'1':::JSONB,1560:::OID)]
          | : scanStringArray "\nSELECT array_agg(index_name)\n  FROM (\nSELECT\n\tindex_name\nFROM\n\tcrdb_internal.table_indexes\nWHERE\n\tindex_type = 'primary'\n\tAND descriptor_id = $1::REGCLASS\n       );\n\t" ["public.table3548"]: ERROR: error in argument for $1: relation "public.table3548" does not exist (SQLSTATE 42P01)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1983-07-28 23:42:56.000958':::TIMESTAMP,2202:::OID,''::STRING,'BOX(-0.8028798948299167 0.4866737668766704,0.25842982473176523 1.256152449899192)':::BOX2D,'{\"b\": [true, [{}, {\"c\": []}, true]], \"bar\": null, \"baz\": [[]], \"foo\": \"$Kk7U\u0026Sf,/\", \"|iHKGe\u0026Y|)i\": 0.21917494336233792}':::JSONB,90002:::OID)]: ERROR: duplicate key value violates unique constraint \"table3231_col3231_3240_col3231_3237_key\" (SQLSTATE 23505)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:15:59.206776",
          |  "ops": [
          |   "BEGIN",
          |   "INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1992-05-03 17:51:29.00024':::TIMESTAMP,869:::OID,e'\\x1bo|QL\u003e'::STRING,'BOX(-1.48848753568185 0.11413130096700558,-0.5829904756554781 1.2114179120739301)':::BOX2D,e'[{\"\\'-{\\\\\"#qb$\": [], \"c\": {\"Zo\u0026U\": {\"(;19~W5\": 0.3397086645906202, \"a\": []}, \"bar\": {\"{![Cg\": {}}}, \"foo\": {\"b\": {}, \"c\": {\"H)a7I\": []}}}, {\"Rf\\\\\\\\X\": false, \"}p\u003e\u0026I\\'AiYD,{\": {}}]':::JSONB,90004:::OID)",
          |   "INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1971-02-05 02:00:12.000346':::TIMESTAMP,1043:::OID,e'[rO\\x15iq\u003cW~'::STRING,'BOX(-0.9235027037328147 -1.2006760750506533,-0.8793129890189155 0.17191664750169722)':::BOX2D,'[[{}], [{\"b\": {}, \"bar\": {}, \"u/RSD.r``}\": [true]}, []], [{}], []]':::JSONB,17:::OID),('1974-08-28 05:54:34.000015':::TIMESTAMP,19:::OID,''::STRING,NULL,'1':::JSONB,1560:::OID)"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation\n OpGen log: \nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table3548\"]] :true\nStmts: \n[INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1992-05-03 17:51:29.00024':::TIMESTAMP,869:::OID,e'\\x1bo|QL\u003e'::STRING,'BOX(-1.48848753568185 0.11413130096700558,-0.5829904756554781 1.2114179120739301)':::BOX2D,e'[{\"\\'-{\\\\\"#qb$\": [], \"c\": {\"Zo\u0026U\": {\"(;19~W5\": 0.3397086645906202, \"a\": []}, \"bar\": {\"{![Cg\": {}}}, \"foo\": {\"b\": {}, \"c\": {\"H)a7I\": []}}}, {\"Rf\\\\\\\\X\": false, \"}p\u003e\u0026I\\'AiYD,{\": {}}]':::JSONB,90004:::OID) INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1971-02-05 02:00:12.000346':::TIMESTAMP,1043:::OID,e'[rO\\x15iq\u003cW~'::STRING,'BOX(-0.9235027037328147 -1.2006760750506533,-0.8793129890189155 0.17191664750169722)':::BOX2D,'[[{}], [{\"b\": {}, \"bar\": {}, \"u/RSD.r``}\": [true]}, []], [{}], []]':::JSONB,17:::OID),('1974-08-28 05:54:34.000015':::TIMESTAMP,19:::OID,''::STRING,NULL,'1':::JSONB,1560:::OID)]\n: scanStringArray \"\\nSELECT array_agg(index_name)\\n  FROM (\\nSELECT\\n\\tindex_name\\nFROM\\n\\tcrdb_internal.table_indexes\\nWHERE\\n\\tindex_type = 'primary'\\n\\tAND descriptor_id = $1::REGCLASS\\n       );\\n\\t\" [\"public.table3548\"]: ERROR: error in argument for $1: relation \"public.table3548\" does not exist (SQLSTATE 42P01)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ e4cafeb8b1d586d091fb98e3e570650d7eeea294:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_125223.608565153_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_125223.608565153_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  FLOAT4 NOT NULL, col139_141 TIME NOT NULL, col139_142 NAME NULL, col139_143 JSONB NOT NULL, col139_144 INET NOT NULL, col139_145 INET NULL, col139_146 STRING NOT NULL AS (lower(CAST(col139_144 AS STRING))) VIRTUAL, INVERTED INDEX (col139_141, col139_142 gin_trgm_ops ASC) PARTITION BY LIST (col139_141) (PARTITION table139_part_0 VALUES IN (('24:00:00':::TIME,), ('23:37:14.196281':::TIME,), ('05:25:13.281703':::TIME,), ('01:53:05.700101':::TIME,), ('19:55:22.660942':::TIME,), ('13:09:51.902929':::TIME,), ('07:35:05.868837':::TIME,), ('19:11:32.353996':::TIME,)), PARTITION table139_part_1 VALUES IN (('17:55:44.07421':::TIME,), ('20:48:27.589646':::TIME,), ('07:08:53.715879':::TIME,), ('11:21:19.692067':::TIME,), ('13:59:06.021694':::TIME,), ('10:02:12.241381':::TIME,), ('18:14:46.021724':::TIME,), ('12:29:38.802148':::TIME,)), PARTITION table139_part_2 VALUES IN (('14:42:29.996344':::TIME,), ('11:23:05.141123':::TIME,), ('01:48:59.930289':::TIME,), ('18:27:24.136821':::TIME,), ('00:00:00':::TIME,), ('14:56:32.653137':::TIME,), ('00:08:26.334906':::TIME,), ('18:30:00.665433':::TIME,)), PARTITION table139_part_3 VALUES IN (('05:32:15.193743':::TIME,), ('03:57:30.432515':::TIME,), ('13:11:47.918144':::TIME,), ('08:25:41.942095':::TIME,), ('04:09:21.342126':::TIME,), ('23:12:19.158671':::TIME,), ('17:29:23.834196':::TIME,), ('10:52:17.605332':::TIME,)), PARTITION table139_part_4 VALUES IN (('11:17:36.747781':::TIME,), ('07:47:06.089525':::TIME,), ('10:55:02.190203':::TIME,), ('10:48:58.225295':::TIME,), ('12:45:42.754913':::TIME,), ('10:54:59.160654':::TIME,), ('11:49:36.699482':::TIME,)), PARTITION table139_part_5 VALUES IN (('21:49:23.451647':::TIME,), ('15:17:22.036342':::TIME,), ('23:57:56.831033':::TIME,), ('17:45:29.588652':::TIME,), ('07:19:57.297439':::TIME,), ('12:35:18.123245':::TIME,)), PARTITION table139_part_6 VALUES IN (('13:01:37.934688':::TIME,), ('11:13:07.842972':::TIME,), ('07:34:37.631871':::TIME,), ('15:16:40.572605':::TIME,), ('22:14:53.010854':::TIME,), ('15:14:52.795521':::TIME,), ('01:04:03.747686':::TIME,)), PARTITION table139_part_7 VALUES IN (('05:05:05.689702':::TIME,), ('15:28:25.631971':::TIME,), ('17:48:21.603086':::TIME,), ('14:14:47.692422':::TIME,), ('04:37:12.672538':::TIME,), ('17:32:54.435286':::TIME,)), PARTITION table139_part_8 VALUES IN (('00:07:54.421109':::TIME,), ('22:23:01.532495':::TIME,), ('03:54:00.602109':::TIME,), ('00:36:29.80034':::TIME,), ('22:32:02.953054':::TIME,), ('06:38:22.466239':::TIME,)), PARTITION table139_part_9 VALUES IN (('03:13:02.860947':::TIME,), ('15:23:33.758746':::TIME,), ('03:34:32.950878':::TIME,), ('08:20:43.90328':::TIME,), ('07:28:35.873363':::TIME,), ('12:49:27.320904':::TIME,), ('10:45:10.306046':::TIME,))), UNIQUE (col139_144) STORING (col139_140, col139_141, col139_142), UNIQUE (lower(CAST(col139_144 AS STRING)) DESC, lower(CAST(col139_143 AS STRING)), lower(CAST(col139_145 AS STRING)) ASC), UNIQUE (col139_142, col139_141 DESC, col139_140 DESC, col139_144 DESC), FAMILY (col139_144), FAMILY (col139_140, col139_141, col139_143, col139_142), FAMILY (col139_145)) INSERT INTO public.table139 (col139_140,col139_141,col139_142,col139_143,col139_144,col139_145) VALUES ((-0.2963637709617615):::FLOAT8,'12:44:43.837899':::TIME,e'\'6"'::NAME:::NAME,'[{"#FI": [[[]], []], "a": [["}|-#+s2rC", [], [], true], false]}, [], {"5RK": [0.0421417206374064]}]':::JSONB,'997d:8848:7d08:79d5:5a5a:f1f4:b2f5:2b1e/86':::INET,NULL),((-0.13457858562469482):::FLOAT8,'19:55:35.483005':::TIME,e'\''::NAME,e'{"9:zLDXA": {"(kh_/)Z\\"d": {}, "QvUGlWg~o@": {"OM$J91": {}}, "^,(g!!Wl": [[{"bar": [[]]}], true], "b": [null], "s8JVv8": [], "|}YJ[I-": {}}, "r!t^y,": []}':::JSONB,'0.0.0.0/0':::INET,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET),((-1.0784741640090942):::FLOAT8,'00:00:00':::TIME,'c6EE'::NAME:::NAME,e'[[[[{"qQW!": {}}], {"H1\'@0z6z|P": [null, [], false, {}]}, [[]]], {}, null], []]':::JSONB,'195.61.126.28/5':::INET,NULL)]: ERROR: internal error: unexpected error from the vectorized engine: interface conversion: tree.Datum is *tree.DOidWrapper, not *tree.DString (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | , col139_142 NAME NULL, col139_143 JSONB NOT NULL, col139_144 INET NOT NULL, col139_145 INET NULL, col139_146 STRING NOT NULL AS (lower(CAST(col139_144 AS STRING))) VIRTUAL, INVERTED INDEX (col139_141, col139_142 gin_trgm_ops ASC) PARTITION BY LIST (col139_141) (PARTITION table139_part_0 VALUES IN (('24:00:00':::TIME,), ('23:37:14.196281':::TIME,), ('05:25:13.281703':::TIME,), ('01:53:05.700101':::TIME,), ('19:55:22.660942':::TIME,), ('13:09:51.902929':::TIME,), ('07:35:05.868837':::TIME,), ('19:11:32.353996':::TIME,)), PARTITION table139_part_1 VALUES IN (('17:55:44.07421':::TIME,), ('20:48:27.589646':::TIME,), ('07:08:53.715879':::TIME,), ('11:21:19.692067':::TIME,), ('13:59:06.021694':::TIME,), ('10:02:12.241381':::TIME,), ('18:14:46.021724':::TIME,), ('12:29:38.802148':::TIME,)), PARTITION table139_part_2 VALUES IN (('14:42:29.996344':::TIME,), ('11:23:05.141123':::TIME,), ('01:48:59.930289':::TIME,), ('18:27:24.136821':::TIME,), ('00:00:00':::TIME,), ('14:56:32.653137':::TIME,), ('00:08:26.334906':::TIME,), ('18:30:00.665433':::TIME,)), PARTITION table139_part_3 VALUES IN (('05:32:15.193743':::TIME,), ('03:57:30.432515':::TIME,), ('13:11:47.918144':::TIME,), ('08:25:41.942095':::TIME,), ('04:09:21.342126':::TIME,), ('23:12:19.158671':::TIME,), ('17:29:23.834196':::TIME,), ('10:52:17.605332':::TIME,)), PARTITION table139_part_4 VALUES IN (('11:17:36.747781':::TIME,), ('07:47:06.089525':::TIME,), ('10:55:02.190203':::TIME,), ('10:48:58.225295':::TIME,), ('12:45:42.754913':::TIME,), ('10:54:59.160654':::TIME,), ('11:49:36.699482':::TIME,)), PARTITION table139_part_5 VALUES IN (('21:49:23.451647':::TIME,), ('15:17:22.036342':::TIME,), ('23:57:56.831033':::TIME,), ('17:45:29.588652':::TIME,), ('07:19:57.297439':::TIME,), ('12:35:18.123245':::TIME,)), PARTITION table139_part_6 VALUES IN (('13:01:37.934688':::TIME,), ('11:13:07.842972':::TIME,), ('07:34:37.631871':::TIME,), ('15:16:40.572605':::TIME,), ('22:14:53.010854':::TIME,), ('15:14:52.795521':::TIME,), ('01:04:03.747686':::TIME,)), PARTITION table139_part_7 VALUES IN (('05:05:05.689702':::TIME,), ('15:28:25.631971':::TIME,), ('17:48:21.603086':::TIME,), ('14:14:47.692422':::TIME,), ('04:37:12.672538':::TIME,), ('17:32:54.435286':::TIME,)), PARTITION table139_part_8 VALUES IN (('00:07:54.421109':::TIME,), ('22:23:01.532495':::TIME,), ('03:54:00.602109':::TIME,), ('00:36:29.80034':::TIME,), ('22:32:02.953054':::TIME,), ('06:38:22.466239':::TIME,)), PARTITION table139_part_9 VALUES IN (('03:13:02.860947':::TIME,), ('15:23:33.758746':::TIME,), ('03:34:32.950878':::TIME,), ('08:20:43.90328':::TIME,), ('07:28:35.873363':::TIME,), ('12:49:27.320904':::TIME,), ('10:45:10.306046':::TIME,))), UNIQUE (col139_144) STORING (col139_140, col139_141, col139_142), UNIQUE (lower(CAST(col139_144 AS STRING)) DESC, lower(CAST(col139_143 AS STRING)), lower(CAST(col139_145 AS STRING)) ASC), UNIQUE (col139_142, col139_141 DESC, col139_140 DESC, col139_144 DESC), FAMILY (col139_144), FAMILY (col139_140, col139_141, col139_143, col139_142), FAMILY (col139_145)) INSERT INTO public.table139 (col139_140,col139_141,col139_142,col139_143,col139_144,col139_145) VALUES ((-0.2963637709617615):::FLOAT8,'12:44:43.837899':::TIME,e'\\'6\"'::NAME:::NAME,'[{\"#FI\": [[[]], []], \"a\": [[\"}|-#+s2rC\", [], [], true], false]}, [], {\"5RK\": [0.0421417206374064]}]':::JSONB,'997d:8848:7d08:79d5:5a5a:f1f4:b2f5:2b1e/86':::INET,NULL),((-0.13457858562469482):::FLOAT8,'19:55:35.483005':::TIME,e'\\''::NAME,e'{\"9:zLDXA\": {\"(kh_/)Z\\\\\"d\": {}, \"QvUGlWg~o@\": {\"OM$J91\": {}}, \"^,(g!!Wl\": [[{\"bar\": [[]]}], true], \"b\": [null], \"s8JVv8\": [], \"|}YJ[I-\": {}}, \"r!t^y,\": []}':::JSONB,'0.0.0.0/0':::INET,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET),((-1.0784741640090942):::FLOAT8,'00:00:00':::TIME,'c6EE'::NAME:::NAME,e'[[[[{\"qQW!\": {}}], {\"H1\\'@0z6z|P\": [null, [], false, {}]}, [[]]], {}, null], []]':::JSONB,'195.61.126.28/5':::INET,NULL)]: ERROR: internal error: unexpected error from the vectorized engine: interface conversion: tree.Datum is *tree.DOidWrapper, not *tree.DString (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 3dcf715f37595a73e8504b5331493f75207c6524:

          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_131753.284990293_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  sql:ALTER DATABASE schemachange ADD REGION "us-west1" time:396.443µs]
          | I220719 13:28:00.325512 234 workload/pgx_helpers.go:79  [-] 1302  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema2555.table3841" does not exist (SQLSTATE 42P01) pid:1657702 sql:CREATE TABLE schema3198.table3860 AS SELECT "IrrelevantColumnName", schema1303.table2587.col2587_2591, schema1303.table2587.col2587_2593, schema2809.table3147.col3147_3148, schema2809.table3147.col3147_3151, schema2809.table3147.col3147_3150, schema2809.table3147.col3147_3149, schema2809.table3147.col3147_3152, schema2809.table3147.col3147_3154 FROM schema2555.table3841, schema1303.table2587, schema2809.table3147 time:176.743834ms]
          | I220719 13:28:00.831970 227 workload/pgx_helpers.go:79  [-] 1303  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot add region "us-west1" to database schemachange (SQLSTATE 42P12) pid:1509596 sql:ALTER DATABASE schemachange ADD REGION "us-west1" time:177.165183ms]
          | I220719 13:28:02.092115 240 workload/pgx_helpers.go:79  [-] 1304  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3657.table3865" does not exist (SQLSTATE 42P01) pid:1679658 sql:INSERT INTO schema3657.table3865 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:808.447µs]
          | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305  pgx logger [error]: Query logParams=map[args:[] err:ERROR: referenced schema ID 1005: descriptor not found (SQLSTATE XXUUU) pid:1686650 sql:
          | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +  SELECT schema_name, table_name
          | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +    FROM [SHOW TABLES]
          | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +   WHERE table_name ~ 'table[0-9]+'
          | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +ORDER BY random()
          | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +   LIMIT 1;]
          | Error: ***UNEXPECTED ERROR; Failed to generate a random operation
          |  OpGen log: 
          |
          | Stmts: 
          | [CREATE TABLE schema2809.table3826 (col3826_3828 INT2 NOT NULL, col3826_3829 UUID, col3826_3830 REGROLE NULL, UNIQUE (col3826_3829, col3826_3830 ASC), INDEX (col3826_3828, col3826_3830) STORING (col3826_3829), UNIQUE (col3826_3829 ASC, col3826_3830 DESC, col3826_3828), INDEX (col3826_3830 ASC, col3826_3829, col3826_3828), INDEX (col3826_3829) PARTITION BY LIST (col3826_3829) (PARTITION table3826_part_0 VALUES IN (('00000000-0000-0000-0000-000000000000':::UUID,), ('15ad563c-19b9-40f2-90c8-e351cf96e0be':::UUID,), ('14e6a3a4-5c56-4211-a541-906b593b481a':::UUID,)), PARTITION table3826_part_1 VALUES IN (('382b745a-3db0-43aa-b015-7ab7c9ad1679':::UUID,), ('4c0cf83b-7c9f-4400-9457-4bb8636895df':::UUID,), ('e1ee0b87-de5a-4ad6-8716-9369d6ffb0bb':::UUID,)), PARTITION table3826_part_2 VALUES IN (('0d749848-c234-436b-86a0-7eff610b9c55':::UUID,), ('3bab0819-a490-40a6-9ea7-cf7635a8f30b':::UUID,), ('3061acf1-fbb2-44b9-b203-7352a335cb44':::UUID,)), PARTITION table3826_part_3 VALUES IN (('247beb63-fedb-4a82-9d9e-d86f27db3d6d':::UUID,), ('1d67cfa0-0db0-4e2e-8768-c19b725dab5f':::UUID,), ('3ab948c5-b41a-4071-a893-6ccec7aabb96':::UUID,)))) CREATE SCHEMA schema3842 AUTHORIZATION root SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema2809.table3147 (col3147_3148,col3147_3149,col3147_3150,col3147_3151,col3147_3152,col3147_3153,col3147_3154,col3147_3155) VALUES ((-7630.165712410323711):::DECIMAL,B'101100011011101010100111111',17:::OID,e'\x1fkg'::STRING,21:::OID,NULL,1560:::OID,B'011111111111111'),(2743276165876.467938:::DECIMAL,B'010011000111110110001110011',1083:::OID,e'^)x\x01\x1e"_D\x0b'::STRING,4089:::OID,'33 years 11 mons 539 days 20:35:09.254163':::INTERVAL,24:::OID,B'110001101011000'),(3661649782431.620875:::DECIMAL,B'111001110100001101001011111',1560:::OID,e'\x1d\x1b\x16hYb)nQ'::STRING,3802:::OID,'17 years 1 mon 987 days 19:05:02.435918':::INTERVAL,20:::OID,B'110011101000101')]
          | : error getting random table name: ERROR: referenced schema ID 1005: descriptor not found (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 0 REGROLE NULL, UNIQUE (col3826_3829, col3826_3830 ASC), INDEX (col3826_3828, col3826_3830) STORING (col3826_3829), UNIQUE (col3826_3829 ASC, col3826_3830 DESC, col3826_3828), INDEX (col3826_3830 ASC, col3826_3829, col3826_3828), INDEX (col3826_3829) PARTITION BY LIST (col3826_3829) (PARTITION table3826_part_0 VALUES IN (('00000000-0000-0000-0000-000000000000':::UUID,), ('15ad563c-19b9-40f2-90c8-e351cf96e0be':::UUID,), ('14e6a3a4-5c56-4211-a541-906b593b481a':::UUID,)), PARTITION table3826_part_1 VALUES IN (('382b745a-3db0-43aa-b015-7ab7c9ad1679':::UUID,), ('4c0cf83b-7c9f-4400-9457-4bb8636895df':::UUID,), ('e1ee0b87-de5a-4ad6-8716-9369d6ffb0bb':::UUID,)), PARTITION table3826_part_2 VALUES IN (('0d749848-c234-436b-86a0-7eff610b9c55':::UUID,), ('3bab0819-a490-40a6-9ea7-cf7635a8f30b':::UUID,), ('3061acf1-fbb2-44b9-b203-7352a335cb44':::UUID,)), PARTITION table3826_part_3 VALUES IN (('247beb63-fedb-4a82-9d9e-d86f27db3d6d':::UUID,), ('1d67cfa0-0db0-4e2e-8768-c19b725dab5f':::UUID,), ('3ab948c5-b41a-4071-a893-6ccec7aabb96':::UUID,))))",
          |   "CREATE SCHEMA schema3842 AUTHORIZATION root",
          |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
          |   "INSERT INTO schema2809.table3147 (col3147_3148,col3147_3149,col3147_3150,col3147_3151,col3147_3152,col3147_3153,col3147_3154,col3147_3155) VALUES ((-7630.165712410323711):::DECIMAL,B'101100011011101010100111111',17:::OID,e'\\x1fkg'::STRING,21:::OID,NULL,1560:::OID,B'011111111111111'),(2743276165876.467938:::DECIMAL,B'010011000111110110001110011',1083:::OID,e'^)x\\x01\\x1e\"_D\\x0b'::STRING,4089:::OID,'33 years 11 mons 539 days 20:35:09.254163':::INTERVAL,24:::OID,B'110001101011000'),(3661649782431.620875:::DECIMAL,B'111001110100001101001011111',1560:::OID,e'\\x1d\\x1b\\x16hYb)nQ'::STRING,3802:::OID,'17 years 1 mon 987 days 19:05:02.435918':::INTERVAL,20:::OID,B'110011101000101')"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation\n OpGen log: \n\nStmts: \n[CREATE TABLE schema2809.table3826 (col3826_3828 INT2 NOT NULL, col3826_3829 UUID, col3826_3830 REGROLE NULL, UNIQUE (col3826_3829, col3826_3830 ASC), INDEX (col3826_3828, col3826_3830) STORING (col3826_3829), UNIQUE (col3826_3829 ASC, col3826_3830 DESC, col3826_3828), INDEX (col3826_3830 ASC, col3826_3829, col3826_3828), INDEX (col3826_3829) PARTITION BY LIST (col3826_3829) (PARTITION table3826_part_0 VALUES IN (('00000000-0000-0000-0000-000000000000':::UUID,), ('15ad563c-19b9-40f2-90c8-e351cf96e0be':::UUID,), ('14e6a3a4-5c56-4211-a541-906b593b481a':::UUID,)), PARTITION table3826_part_1 VALUES IN (('382b745a-3db0-43aa-b015-7ab7c9ad1679':::UUID,), ('4c0cf83b-7c9f-4400-9457-4bb8636895df':::UUID,), ('e1ee0b87-de5a-4ad6-8716-9369d6ffb0bb':::UUID,)), PARTITION table3826_part_2 VALUES IN (('0d749848-c234-436b-86a0-7eff610b9c55':::UUID,), ('3bab0819-a490-40a6-9ea7-cf7635a8f30b':::UUID,), ('3061acf1-fbb2-44b9-b203-7352a335cb44':::UUID,)), PARTITION table3826_part_3 VALUES IN (('247beb63-fedb-4a82-9d9e-d86f27db3d6d':::UUID,), ('1d67cfa0-0db0-4e2e-8768-c19b725dab5f':::UUID,), ('3ab948c5-b41a-4071-a893-6ccec7aabb96':::UUID,)))) CREATE SCHEMA schema3842 AUTHORIZATION root SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema2809.table3147 (col3147_3148,col3147_3149,col3147_3150,col3147_3151,col3147_3152,col3147_3153,col3147_3154,col3147_3155) VALUES ((-7630.165712410323711):::DECIMAL,B'101100011011101010100111111',17:::OID,e'\\x1fkg'::STRING,21:::OID,NULL,1560:::OID,B'011111111111111'),(2743276165876.467938:::DECIMAL,B'010011000111110110001110011',1083:::OID,e'^)x\\x01\\x1e\"_D\\x0b'::STRING,4089:::OID,'33 years 11 mons 539 days 20:35:09.254163':::INTERVAL,24:::OID,B'110001101011000'),(3661649782431.620875:::DECIMAL,B'111001110100001101001011111',1560:::OID,e'\\x1d\\x1b\\x16hYb)nQ'::STRING,3802:::OID,'17 years 1 mon 987 days 19:05:02.435918':::INTERVAL,20:::OID,B'110011101000101')]\n: error getting random table name: ERROR: referenced schema ID 1005: descriptor not found (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 457d724622e4fa2e62d6f4e7926509dbc7d18511:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_115959.445950842_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_115959.445950842_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 1EE8488E7CC0BF441188F216910B5ED41501052B7C158EF416069B9A3755FD541F8B4BD46385BD3C14E167C21C80EF94108ADAF3D3520D841B0501F5D565902C2E44BC82A5A71FF41C0B2A926209EAEC132F8561E97A3E0C1AB30170D417C00C272CA35469D82F041010300008001000000060000007CD1FEA2B623D8C17CCE777AD4BA01C2E8CA69A1C80FFF41CCEDEE0FAEEAFB41F2546B070870F7C13051BE4A269FFD414EECA0C8802E00425C9192B2732EF3C114559B00B0E1FA4130F6EEC27441DB41E0A21237C207B3C1DE7C332A27CAF041D40923CF10E7E3C16C238D2EA407E0C17BA5A6A30835F6C17CD1FEA2B623D8C17CCE777AD4BA01C2E8CA69A1C80FFF41':::GEOMETRY,']'::CHAR,'01050000C00500000001020000C0040000001EAFD3AD664BE5C1DF04DB3B6E41FCC183614F7E491AFBC1B06D15ACFCCAFD41207C125BF462E6419A8646180038E5C1881F0585EEE4F24140F83D335783BC41526445E19885F64182074FAF8D62E0C1105D6DCB5008FF418074B65C47D9C641CCE32C047281014296EC346984CBF441300ABC85A0C0EBC14CE443C87C55FD4101020000C002000000E4E1507EC01FFC4100E38D05020897C170C151FE8539E84130229D56ED58F0410C2DE37B0438DFC114CC6898E5D3F141F4D8F2A9CF41F7C1E81C4076BCEAF04101020000C003000000D0DD0C6903B6DFC1C8A7A10F8C3DEC410FF8DAA4AB7801C28E8EDCC6C04EF7C1B6BB83E0C0AF00C2EE403724F47AF141C0D132317184A9C1B299A13D96CCE2C1A60A30C22F8902C21CA8D8AB989EF0418C780E2704ABEB412BE4A30C3242FDC101020000C0070000005E5976DB9666F94122C81BD336D3FDC190F5D39FF686E7C1284F940E9DF2FF413841BD0D42A6014234E890460A98F0C1997C9AC5A286F2C1D615E23792B0014268CCAA3D911302420467B9E9DCB3E8C148279AD3560A02426684B24EFC1CF3C16607189E07A3F5413887435CD2F6E0413CCA793F4103FC41AC14D286DE1EFC41606513ABA04FF2418CB268AA0C08ED415EFB9EFF4D2DE0C1AAD5F38A7B05FCC13C0AC420B5B2ED41C48C155AE437E941F6EA77CD72F70142582AC06FC277ED4138CD014EDCA9DD4110BC506F331DF941387F5B7CF390F74108B142FED8FC004201020000C003000000880F8481DA69E041C436EB8EA4D4E7C1B9D44DEA7234F2C1DC52FE9AD4A7F0C1A8FD463440D9D241FC75C8AF39B5EA4132AD2785D177F04134B24D33836FF5C1E028E205DC54CD41661E3292D4BFF041687D2FE84D9CDB41F0DD2682CA7BCBC1':::GEOMETRY,1184:::OID,ARRAY['645a8d2c-27da-491b-bc61-19e44dde64ed':::UUID,'07e8e918-16f7-42ab-88cb-e0dceb9ce790':::UUID,'677687e0-e219-4d1f-a970-cb23d7452754':::UUID],'eb31:3a52:9bc5:f3af:cd6a:af2a:da3d:f3cb/57':::INET,0.8386564254760742:::FLOAT8,2373136509439009929:::INT8,'[{"foo": [{"!_#,*.O<|": [0.4840091819807166]}], "xR@>": [[[]], []]}, false, false, {}, {}, {}]':::JSONB,B'101000000101') INSERT INTO schema5192.table5396 (col5396_5397,col5396_5398,col5396_5399) VALUES (0.8204111539383099:::FLOAT8,'1991-04-17 23:40:56.000009+00:00':::TIMESTAMPTZ,90002:::OID),(0.30766111449387734:::FLOAT8,'2028-12-30 00:45:15.00068+00:00':::TIMESTAMPTZ,3802:::OID) INSERT INTO schema5192.table5396 (col5396_5397,col5396_5398,col5396_5399) VALUES (0.6855257216696131:::FLOAT8,'2014-04-09 23:01:12.000986+00:00':::TIMESTAMPTZ,701:::OID) DROP SCHEMA "schema3298" CASCADE INSERT INTO schema3198.table5334 (col5334_5335,col5334_5336,col5334_5337,col5334_5338,col5334_5339,col5334_5340,col5334_5341,col5334_5342,col5334_5343,col5334_5344,col5334_5345,col5334_5346,col5334_5347,col5334_5348) VALUES ((-1.5266615288438692):::FLOAT8,'f7b40faa-f1bc-4ac3-adca-015e70c10362':::UUID,1082:::OID,'110.72.219.144/15':::INET,'806d:99c:c160:abd8:154f:db42:ac5d:b747/41':::INET,B'11',e'\\``Yw]uz\b'::STRING,B'000000001111100',1.2179748078821273:::FLOAT8,B'001001010011110110',2205:::OID,1186:::OID,'10:48:51.33529-13:32:00':::TIMETZ,2950:::OID),((-0.9290058183190448):::FLOAT8,'6ed4946b-5368-4920-b078-9ea5c8ab0e46':::UUID,90004:::OID,'236.60.11.151/24':::INET,'248.21.72.24':::INET,B'00',e'\x1fA-'::STRING,B'000110011001110',0.48114560760924063:::FLOAT8,B'011101110010011010',16:::OID,20:::OID,'21:35:46.893923-02:28:00':::TIMETZ,1700:::OID),((-1.0847846749268706):::FLOAT8,'519164f1-c696-4d04-bba6-e112d7bf2214':::UUID,2205:::OID,'0.0.0.0/0':::INET,'ad10:b60e:2905:f3ee:cbc0:c540:3ab5:3805/75':::INET,B'01',e'h\x1d'::STRING,B'011111111111111',0.7094286758149404:::FLOAT8,B'011111111111111111',90000:::OID,869:::OID,'09:43:23.326247-12:17:00':::TIMETZ,2950:::OID)]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 780976604469329921 could not be loaded: job with ID 780976604469329921 does not exist (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 0B5ED41501052B7C158EF416069B9A3755FD541F8B4BD46385BD3C14E167C21C80EF94108ADAF3D3520D841B0501F5D565902C2E44BC82A5A71FF41C0B2A926209EAEC132F8561E97A3E0C1AB30170D417C00C272CA35469D82F041010300008001000000060000007CD1FEA2B623D8C17CCE777AD4BA01C2E8CA69A1C80FFF41CCEDEE0FAEEAFB41F2546B070870F7C13051BE4A269FFD414EECA0C8802E00425C9192B2732EF3C114559B00B0E1FA4130F6EEC27441DB41E0A21237C207B3C1DE7C332A27CAF041D40923CF10E7E3C16C238D2EA407E0C17BA5A6A30835F6C17CD1FEA2B623D8C17CCE777AD4BA01C2E8CA69A1C80FFF41':::GEOMETRY,']'::CHAR,'01050000C00500000001020000C0040000001EAFD3AD664BE5C1DF04DB3B6E41FCC183614F7E491AFBC1B06D15ACFCCAFD41207C125BF462E6419A8646180038E5C1881F0585EEE4F24140F83D335783BC41526445E19885F64182074FAF8D62E0C1105D6DCB5008FF418074B65C47D9C641CCE32C047281014296EC346984CBF441300ABC85A0C0EBC14CE443C87C55FD4101020000C002000000E4E1507EC01FFC4100E38D05020897C170C151FE8539E84130229D56ED58F0410C2DE37B0438DFC114CC6898E5D3F141F4D8F2A9CF41F7C1E81C4076BCEAF04101020000C003000000D0DD0C6903B6DFC1C8A7A10F8C3DEC410FF8DAA4AB7801C28E8EDCC6C04EF7C1B6BB83E0C0AF00C2EE403724F47AF141C0D132317184A9C1B299A13D96CCE2C1A60A30C22F8902C21CA8D8AB989EF0418C780E2704ABEB412BE4A30C3242FDC101020000C0070000005E5976DB9666F94122C81BD336D3FDC190F5D39FF686E7C1284F940E9DF2FF413841BD0D42A6014234E890460A98F0C1997C9AC5A286F2C1D615E23792B0014268CCAA3D911302420467B9E9DCB3E8C148279AD3560A02426684B24EFC1CF3C16607189E07A3F5413887435CD2F6E0413CCA793F4103FC41AC14D286DE1EFC41606513ABA04FF2418CB268AA0C08ED415EFB9EFF4D2DE0C1AAD5F38A7B05FCC13C0AC420B5B2ED41C48C155AE437E941F6EA77CD72F70142582AC06FC277ED4138CD014EDCA9DD4110BC506F331DF941387F5B7CF390F74108B142FED8FC004201020000C003000000880F8481DA69E041C436EB8EA4D4E7C1B9D44DEA7234F2C1DC52FE9AD4A7F0C1A8FD463440D9D241FC75C8AF39B5EA4132AD2785D177F04134B24D33836FF5C1E028E205DC54CD41661E3292D4BFF041687D2FE84D9CDB41F0DD2682CA7BCBC1':::GEOMETRY,1184:::OID,ARRAY['645a8d2c-27da-491b-bc61-19e44dde64ed':::UUID,'07e8e918-16f7-42ab-88cb-e0dceb9ce790':::UUID,'677687e0-e219-4d1f-a970-cb23d7452754':::UUID],'eb31:3a52:9bc5:f3af:cd6a:af2a:da3d:f3cb/57':::INET,0.8386564254760742:::FLOAT8,2373136509439009929:::INT8,'[{\"foo\": [{\"!_#,*.O\u003c|\": [0.4840091819807166]}], \"xR@\u003e\": [[[]], []]}, false, false, {}, {}, {}]':::JSONB,B'101000000101') INSERT INTO schema5192.table5396 (col5396_5397,col5396_5398,col5396_5399) VALUES (0.8204111539383099:::FLOAT8,'1991-04-17 23:40:56.000009+00:00':::TIMESTAMPTZ,90002:::OID),(0.30766111449387734:::FLOAT8,'2028-12-30 00:45:15.00068+00:00':::TIMESTAMPTZ,3802:::OID) INSERT INTO schema5192.table5396 (col5396_5397,col5396_5398,col5396_5399) VALUES (0.6855257216696131:::FLOAT8,'2014-04-09 23:01:12.000986+00:00':::TIMESTAMPTZ,701:::OID) DROP SCHEMA \"schema3298\" CASCADE INSERT INTO schema3198.table5334 (col5334_5335,col5334_5336,col5334_5337,col5334_5338,col5334_5339,col5334_5340,col5334_5341,col5334_5342,col5334_5343,col5334_5344,col5334_5345,col5334_5346,col5334_5347,col5334_5348) VALUES ((-1.5266615288438692):::FLOAT8,'f7b40faa-f1bc-4ac3-adca-015e70c10362':::UUID,1082:::OID,'110.72.219.144/15':::INET,'806d:99c:c160:abd8:154f:db42:ac5d:b747/41':::INET,B'11',e'\\\\``Yw]uz\\b'::STRING,B'000000001111100',1.2179748078821273:::FLOAT8,B'001001010011110110',2205:::OID,1186:::OID,'10:48:51.33529-13:32:00':::TIMETZ,2950:::OID),((-0.9290058183190448):::FLOAT8,'6ed4946b-5368-4920-b078-9ea5c8ab0e46':::UUID,90004:::OID,'236.60.11.151/24':::INET,'248.21.72.24':::INET,B'00',e'\\x1fA-'::STRING,B'000110011001110',0.48114560760924063:::FLOAT8,B'011101110010011010',16:::OID,20:::OID,'21:35:46.893923-02:28:00':::TIMETZ,1700:::OID),((-1.0847846749268706):::FLOAT8,'519164f1-c696-4d04-bba6-e112d7bf2214':::UUID,2205:::OID,'0.0.0.0/0':::INET,'ad10:b60e:2905:f3ee:cbc0:c540:3ab5:3805/75':::INET,B'01',e'h\\x1d'::STRING,B'011111111111111',0.7094286758149404:::FLOAT8,B'011111111111111111',90000:::OID,869:::OID,'09:43:23.326247-12:17:00':::TIMETZ,2950:::OID)]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 780976604469329921 could not be loaded: job with ID 780976604469329921 does not exist (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ fafc3330130a016a881592d0337ec48d9ce43b61:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_132153.016794587_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_132153.016794587_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ALUES IN (('2027-01-11 12:50:48.000744+00:00':::TIMESTAMPTZ, '':::STRING), ('2014-04-17 16:10:45.000979+00:00':::TIMESTAMPTZ, e'G\x02d?\'':::STRING), ('2021-09-28 06:58:17.000619+00:00':::TIMESTAMPTZ, '':::STRING), ('1975-10-20 19:12:05.00035+00:00':::TIMESTAMPTZ, 'X':::STRING), (NULL, e'\x00':::STRING), ('1994-02-06 07:37:53.000065+00:00':::TIMESTAMPTZ, e'\U00002603':::STRING)), PARTITION table5446_part_4 VALUES IN (('2027-10-19 06:17:05.000432+00:00':::TIMESTAMPTZ, '':::STRING), ('1983-12-12 09:17:03.000777+00:00':::TIMESTAMPTZ, NULL), ('294276-12-31 23:59:59.999999+00:00':::TIMESTAMPTZ, e'Xr*\x04':::STRING), ('2015-09-23 03:37:28.000007+00:00':::TIMESTAMPTZ, '|:y,(Gq?&':::STRING), ('1976-09-15 16:29:33.000606+00:00':::TIMESTAMPTZ, e'q?\x15o':::STRING), ('1991-07-18 14:06:57.000163+00:00':::TIMESTAMPTZ, e'Ed\\':::STRING)), PARTITION table5446_part_5 VALUES IN (('1975-09-04 23:08:55.000393+00:00':::TIMESTAMPTZ, e'&\x0b\x03T':::STRING), ('1972-11-02 18:09:22.000426+00:00':::TIMESTAMPTZ, e'wP\x1e\x06U':::STRING), ('1975-11-14 21:19:05.00016+00:00':::TIMESTAMPTZ, e';\x06*\x1b~P29':::STRING), ('2009-03-29 09:52:49.000552+00:00':::TIMESTAMPTZ, e'\x12':::STRING), ('2013-07-06 22:04:57.000167+00:00':::TIMESTAMPTZ, e',it$m\x07\x15':::STRING), ('2000-10-14 11:32:57.000845+00:00':::TIMESTAMPTZ, 'gu':::STRING)), PARTITION table5446_part_6 VALUES IN (('0001-01-01 00:00:00+00:00':::TIMESTAMPTZ, 'ieJ':::STRING), ('1983-05-08 06:39:37.000944+00:00':::TIMESTAMPTZ, 'hcq':::STRING), (NULL, 'X':::STRING), ('1978-01-06 22:10:49.000055+00:00':::TIMESTAMPTZ, 'e':::STRING), ('2021-02-05 13:57:54.000635+00:00':::TIMESTAMPTZ, e'\x0e``m&ji':::STRING), (NULL, 'Tv8e':::STRING)), PARTITION table5446_part_7 VALUES IN ((NULL, e's}::F\x06':::STRING), ('2014-12-08 01:54:02.000899+00:00':::TIMESTAMPTZ, 'K?-"OTaU':::STRING), ('2012-03-19 22:56:10.000381+00:00':::TIMESTAMPTZ, NULL), ('-4713-11-24 00:00:00+00:00':::TIMESTAMPTZ, e'j\x16\x15\x11>1|':::STRING), ('1983-12-12 10:22:01.000002+00:00':::TIMESTAMPTZ, e'\x06ZbCP}t':::STRING), ('1971-03-23 12:07:13.000449+00:00':::TIMESTAMPTZ, e'\x02\te\x05X:]':::STRING))), UNIQUE (col5446_5447, col5446_5452, col5446_5463 ASC, (CASE WHEN col5446_5452 IS NULL THEN '':::STRING ELSE e'\x04z{D0M':::STRING END) ASC, col5446_5460 ASC, col5446_5448, col5446_5455, col5446_5459), UNIQUE (col5446_5458 DESC) STORING (col5446_5448, col5446_5453, col5446_5459), UNIQUE (col5446_5451 DESC, col5446_5459 DESC, col5446_5457 DESC, col5446_5460 ASC, col5446_5449 DESC, col5446_5462 ASC, col5446_5452 DESC) STORING (col5446_5450, col5446_5453, col5446_5456, col5446_5461), UNIQUE (col5446_5448, (col5446_5461 + 20847:::INT8) DESC, col5446_5461 DESC, (CASE WHEN col5446_5448 IS NULL THEN e'o\x07Zfw(_':::STRING ELSE e'\x17eQA\n(TM4':::STRING END), col5446_5453 DESC, (col5446_5447 + col5446_5460), (CASE WHEN col5446_5455 IS NULL THEN e'\'':::STRING ELSE e'\'':::STRING END) ASC, col5446_5451 DESC, lower(col5446_5456) DESC, col5446_5452, col5446_5454, col5446_5447 DESC, lower(col5446_5449) ASC, col5446_5457 DESC, col5446_5456 ASC, col5446_5449) STORING (col5446_5450, col5446_5455, col5446_5459, col5446_5460, col5446_5462), INDEX (col5446_5450 DESC, lower(CAST(col5446_5450 AS STRING)) DESC, col5446_5449 ASC, col5446_5457 ASC, col5446_5451 DESC, col5446_5454 ASC, col5446_5459 DESC, col5446_5456 ASC, (col5446_5460 + (-0.5515485427131867):::FLOAT8) DESC, (col5446_5453 + col5446_5461), col5446_5462), INDEX ((col5446_5459 + 5e-324:::FLOAT8) ASC, col5446_5451, (col5446_5461 + col5446_5453) ASC, col5446_5460, col5446_5453 ASC, col5446_5461 DESC, col5446_5459, col5446_5455, col5446_5454 DESC, col5446_5458 DESC, (col5446_5460 + col5446_5447) DESC, col5446_5463 ASC, col5446_5447 ASC, (col5446_5454 + (-9852209045120.41063):::DECIMAL), col5446_5450, lower(col5446_5449))) INSERT INTO public.table2287 (col2287_2288,col2287_2289,col2287_2290,col2287_2291) VALUES (26:::OID,'-10 years -1 mons -149 days -09:54:31.528373':::INTERVAL,true,'2032-11-03 15:08:54.000204+00:00':::TIMESTAMPTZ)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | PTZ, '':::STRING), ('2014-04-17 16:10:45.000979+00:00':::TIMESTAMPTZ, e'G\\x02d?\\'':::STRING), ('2021-09-28 06:58:17.000619+00:00':::TIMESTAMPTZ, '':::STRING), ('1975-10-20 19:12:05.00035+00:00':::TIMESTAMPTZ, 'X':::STRING), (NULL, e'\\x00':::STRING), ('1994-02-06 07:37:53.000065+00:00':::TIMESTAMPTZ, e'\\U00002603':::STRING)), PARTITION table5446_part_4 VALUES IN (('2027-10-19 06:17:05.000432+00:00':::TIMESTAMPTZ, '':::STRING), ('1983-12-12 09:17:03.000777+00:00':::TIMESTAMPTZ, NULL), ('294276-12-31 23:59:59.999999+00:00':::TIMESTAMPTZ, e'Xr*\\x04':::STRING), ('2015-09-23 03:37:28.000007+00:00':::TIMESTAMPTZ, '|:y,(Gq?\u0026':::STRING), ('1976-09-15 16:29:33.000606+00:00':::TIMESTAMPTZ, e'q?\\x15o':::STRING), ('1991-07-18 14:06:57.000163+00:00':::TIMESTAMPTZ, e'Ed\\\\':::STRING)), PARTITION table5446_part_5 VALUES IN (('1975-09-04 23:08:55.000393+00:00':::TIMESTAMPTZ, e'\u0026\\x0b\\x03T':::STRING), ('1972-11-02 18:09:22.000426+00:00':::TIMESTAMPTZ, e'wP\\x1e\\x06U':::STRING), ('1975-11-14 21:19:05.00016+00:00':::TIMESTAMPTZ, e';\\x06*\\x1b~P29':::STRING), ('2009-03-29 09:52:49.000552+00:00':::TIMESTAMPTZ, e'\\x12':::STRING), ('2013-07-06 22:04:57.000167+00:00':::TIMESTAMPTZ, e',it$m\\x07\\x15':::STRING), ('2000-10-14 11:32:57.000845+00:00':::TIMESTAMPTZ, 'gu':::STRING)), PARTITION table5446_part_6 VALUES IN (('0001-01-01 00:00:00+00:00':::TIMESTAMPTZ, 'ieJ':::STRING), ('1983-05-08 06:39:37.000944+00:00':::TIMESTAMPTZ, 'hcq':::STRING), (NULL, 'X':::STRING), ('1978-01-06 22:10:49.000055+00:00':::TIMESTAMPTZ, 'e':::STRING), ('2021-02-05 13:57:54.000635+00:00':::TIMESTAMPTZ, e'\\x0e``m\u0026ji':::STRING), (NULL, 'Tv8e':::STRING)), PARTITION table5446_part_7 VALUES IN ((NULL, e's}::F\\x06':::STRING), ('2014-12-08 01:54:02.000899+00:00':::TIMESTAMPTZ, 'K?-\"OTaU':::STRING), ('2012-03-19 22:56:10.000381+00:00':::TIMESTAMPTZ, NULL), ('-4713-11-24 00:00:00+00:00':::TIMESTAMPTZ, e'j\\x16\\x15\\x11\u003e1|':::STRING), ('1983-12-12 10:22:01.000002+00:00':::TIMESTAMPTZ, e'\\x06ZbCP}t':::STRING), ('1971-03-23 12:07:13.000449+00:00':::TIMESTAMPTZ, e'\\x02\\te\\x05X:]':::STRING))), UNIQUE (col5446_5447, col5446_5452, col5446_5463 ASC, (CASE WHEN col5446_5452 IS NULL THEN '':::STRING ELSE e'\\x04z{D0M':::STRING END) ASC, col5446_5460 ASC, col5446_5448, col5446_5455, col5446_5459), UNIQUE (col5446_5458 DESC) STORING (col5446_5448, col5446_5453, col5446_5459), UNIQUE (col5446_5451 DESC, col5446_5459 DESC, col5446_5457 DESC, col5446_5460 ASC, col5446_5449 DESC, col5446_5462 ASC, col5446_5452 DESC) STORING (col5446_5450, col5446_5453, col5446_5456, col5446_5461), UNIQUE (col5446_5448, (col5446_5461 + 20847:::INT8) DESC, col5446_5461 DESC, (CASE WHEN col5446_5448 IS NULL THEN e'o\\x07Zfw(_':::STRING ELSE e'\\x17eQA\\n(TM4':::STRING END), col5446_5453 DESC, (col5446_5447 + col5446_5460), (CASE WHEN col5446_5455 IS NULL THEN e'\\'':::STRING ELSE e'\\'':::STRING END) ASC, col5446_5451 DESC, lower(col5446_5456) DESC, col5446_5452, col5446_5454, col5446_5447 DESC, lower(col5446_5449) ASC, col5446_5457 DESC, col5446_5456 ASC, col5446_5449) STORING (col5446_5450, col5446_5455, col5446_5459, col5446_5460, col5446_5462), INDEX (col5446_5450 DESC, lower(CAST(col5446_5450 AS STRING)) DESC, col5446_5449 ASC, col5446_5457 ASC, col5446_5451 DESC, col5446_5454 ASC, col5446_5459 DESC, col5446_5456 ASC, (col5446_5460 + (-0.5515485427131867):::FLOAT8) DESC, (col5446_5453 + col5446_5461), col5446_5462), INDEX ((col5446_5459 + 5e-324:::FLOAT8) ASC, col5446_5451, (col5446_5461 + col5446_5453) ASC, col5446_5460, col5446_5453 ASC, col5446_5461 DESC, col5446_5459, col5446_5455, col5446_5454 DESC, col5446_5458 DESC, (col5446_5460 + col5446_5447) DESC, col5446_5463 ASC, col5446_5447 ASC, (col5446_5454 + (-9852209045120.41063):::DECIMAL), col5446_5450, lower(col5446_5449))) INSERT INTO public.table2287 (col2287_2288,col2287_2289,col2287_2290,col2287_2291) VALUES (26:::OID,'-10 years -1 mons -149 days -09:54:31.528373':::INTERVAL,true,'2032-11-03 15:08:54.000204+00:00':::TIMESTAMPTZ)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 7c95ccd0a9f0b0aca9cdb0ed813caf8d60b28732:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_130527.575685958_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_130527.575685958_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 3861257BFDEC188B8D86E4A86FB41C6AF289412BAE7C1002B65E312D5B2C198CB3418244FDA4198BC816C97C9DA411675869D1851F7411AC939EB1339E7C14A2584C1A4170042600DE8D9B819DEC1523A3E8A8BE9F0C18BDC21EE3D28FAC19E4738EF0EAEF4C10103000040010000000B000000D2A00A162C6BFAC1ECC569326AE7DBC1F4E8A1B93291FE4125C9096246F7F4C152A927F7C35BE7C160E0BE7668DED1C18C61AF8E6B6FF241BC5C712A023EFCC118AEFF37A0A7D6C13A997FD79671F141CECF051C3D54F9C100B282676E5BBF418CBFCE02650FF741466EF7845E64E7C124231618EA47FB4128035A1D97E90142186ABD91C41FD8412B342733E1FD00C2B85A00B2343DF74168A4D67D496AD6410C6524CE1E19FC41C0085B17612DEB41F07F136A3CD2C041CC634816563EFD41FE8026825A3EEEC18E007917E56DF64160191C4EA242D441E87C5CCCB8B0F9C15C7D6AF1FC0AF34114C33F7A6FD7F041D2A00A162C6BFAC1ECC569326AE7DBC1F4E8A1B93291FE41':::GEOMETRY,(-0.16251884400844574):::FLOAT8,e'\x14``'::STRING),(true,'1990-08-31 19:54:43.000701':::TIMESTAMP,'1982-11-03 06:03:59.000731+00:00':::TIMESTAMPTZ,'-2000-01-01 00:00:00+00:00':::TIMESTAMPTZ,false,e'\x1d'::CHAR,1043:::OID,2202:::OID,90000:::OID,20:::OID,B'10100111000111010011101011000','y?'::STRING,'BOX(-0.7538995678382455 0.1771061438774085,-0.06631651412197326 0.902130282323982)':::BOX2D,615143279860807163.9:::DECIMAL,'\x8033f0722fb27ca9':::BYTES,'010600004006000000010300004001000000080000009CA63734FF12FDC1D755F4DE9C61FAC124F14F86378DF8C138A5F6597E35E641977EEDE6EC37F0C148F3319A9C01DCC15C9F2E064C78E64184B13026A7B1D7C15058E966D5C8C1C1C0A8F41FA504B14148475F14F11B02428306235728B7F3C19A1B49B78823ECC15ABAAAFD3A83F941A481E3009D63F64150220DCDD33BFAC1901CF3EF1895DA41F0E7A6A88BB3E841011646BA5BA801C270ACDB07C091D741BC8160933FCCE6419CA63734FF12FDC1D755F4DE9C61FAC124F14F86378DF8C10103000040010000000D00000075D1847C2FB4FDC1B4F3D780C333ECC1808B4B9FFC1DDC412C5CA55DE6DDE4C12C0587A814C1E9C16C17DC937CA4F1415EF9F4D52430F3C1B8A9D9ABEB96F6C118596D8FA5B2D8C10086530CBA4FB5C15A6985AD2FD9EAC11E6AE24089B300C2FC0534E1052FE7C104576552DC3600C2FA6FF78F0453F6416CB797E2CEE40142DDFA7858E781F8C1C0CB29961A35EBC1F0AE51A6BB4B024220095DD5CD57BC41D510760EB3F0F6C1F061CC16199AFD41A2BEA7C89D44F04148190CCBA317014262AFE5A6B8C3004210B2C12F4E58FB41B2041159B1620242FA9C71155D91FA41BCA92103E61B00420313BA17ABA6F0C1D0849E859F01F141B21CDAC21B090242E11C6620153AFDC13C98A5082E95E6416EF4CAF7D566F841F096E6A1645FCAC175D1847C2FB4FDC1B4F3D780C333ECC1808B4B9FFC1DDC410103000040010000000900000046D87C2BF082F2C127CA46A104B1F4C1F3B224E85C14F1C1F20246182E0BE0C1E1F0DA3341A6F1C11077DDCC33A6C141606C794B45CDC3419A808EC66ED7F3C1404151593255AB414011CB912F87CC41FABA77154AE4F9C13469D92288FCEBC1C8F1B2E8AC14024290FDCE923774CF41603601AFD2CEF9C156A871CA186402421AF32868F889F24178B46BA3E49FE4C1542DFA44D5DDF541E4CCBF0C2C48E9418A410CCD639702C28C4EEDEA6B37DAC152A1E1CEA491F1418012B09ED555A54146D87C2BF082F2C127CA46A104B1F4C1F3B224E85C14F1C1010300004001000000050000000021234A245AB141D06840F80CEEEBC190738D1B4692CF4134EEC514B437F24100FFB955C4C48141606A5FEAC63FD7C1F8838C3D92EAECC158260F13BD7DF941D41127F3BB0EE441526D89229394F9C1487EB3AD97CBFC41164B30F680E6FCC10021234A245AB141D06840F80CEEEBC190738D1B4692CF410103000040010000000B000000F0DCCD40A68BF9C1B2920763B0EEE2C1FE7AAD99C374004218BA9DFD54C1D0C1A00B3D7964D1D0410CEBFA03DEB7F641DC739893D28AFEC1193BC4B2F2C1F4C1FC3FACF38608EF4146DB599FA42AE0C1AE9D5E86A56CEBC175D3CE260A9400C21C95EAA28518E8C1EB4A1855656A00C20456BF82C404EE414408BE49AA41F8413C80B455DC38F4414C4168E2D272FFC15CF0947DADB2FC415C19ADACE8550242CCFCFB994333D6C10C69CBDE7E2AF84100626F1A557500427430CC233FAFF4C1586DD91C4A2BDA4194B2FAC96FE6F34174649FF2E74FE941A01260001439E9C11E48F87B720301429216A28B5DFBEFC1F0DCCD40A68BF9C1B2920763B0EEE2C1FE7AAD99C3740042010300004001000000070000003EF17C724A88FCC180F26360B0289C414891D02F7B02DEC1A2F4051EA6E4F541C414BBA22AABF5C11CA194BA5461FBC1040DF0E05ACAE6418099CEA642FBA3C1785AED381619E641EC77A4CA6F0AFC4180228DA3E5DDB341F5AC45C88B4B02C24C55F04CC134FE416040989B69B4CD41028CF0A7F904F9C1B0067AE2B31201C27A8044BC7672F6411868CE31A5F6DC413EF17C724A88FCC180F26360B0289C414891D02F7B02DEC1':::GEOMETRY,1.725172996520996:::FLOAT8,e'(/\x16\fL'::STRING)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | FDEC188B8D86E4A86FB41C6AF289412BAE7C1002B65E312D5B2C198CB3418244FDA4198BC816C97C9DA411675869D1851F7411AC939EB1339E7C14A2584C1A4170042600DE8D9B819DEC1523A3E8A8BE9F0C18BDC21EE3D28FAC19E4738EF0EAEF4C10103000040010000000B000000D2A00A162C6BFAC1ECC569326AE7DBC1F4E8A1B93291FE4125C9096246F7F4C152A927F7C35BE7C160E0BE7668DED1C18C61AF8E6B6FF241BC5C712A023EFCC118AEFF37A0A7D6C13A997FD79671F141CECF051C3D54F9C100B282676E5BBF418CBFCE02650FF741466EF7845E64E7C124231618EA47FB4128035A1D97E90142186ABD91C41FD8412B342733E1FD00C2B85A00B2343DF74168A4D67D496AD6410C6524CE1E19FC41C0085B17612DEB41F07F136A3CD2C041CC634816563EFD41FE8026825A3EEEC18E007917E56DF64160191C4EA242D441E87C5CCCB8B0F9C15C7D6AF1FC0AF34114C33F7A6FD7F041D2A00A162C6BFAC1ECC569326AE7DBC1F4E8A1B93291FE41':::GEOMETRY,(-0.16251884400844574):::FLOAT8,e'\\x14``'::STRING),(true,'1990-08-31 19:54:43.000701':::TIMESTAMP,'1982-11-03 06:03:59.000731+00:00':::TIMESTAMPTZ,'-2000-01-01 00:00:00+00:00':::TIMESTAMPTZ,false,e'\\x1d'::CHAR,1043:::OID,2202:::OID,90000:::OID,20:::OID,B'10100111000111010011101011000','y?'::STRING,'BOX(-0.7538995678382455 0.1771061438774085,-0.06631651412197326 0.902130282323982)':::BOX2D,615143279860807163.9:::DECIMAL,'\\x8033f0722fb27ca9':::BYTES,'010600004006000000010300004001000000080000009CA63734FF12FDC1D755F4DE9C61FAC124F14F86378DF8C138A5F6597E35E641977EEDE6EC37F0C148F3319A9C01DCC15C9F2E064C78E64184B13026A7B1D7C15058E966D5C8C1C1C0A8F41FA504B14148475F14F11B02428306235728B7F3C19A1B49B78823ECC15ABAAAFD3A83F941A481E3009D63F64150220DCDD33BFAC1901CF3EF1895DA41F0E7A6A88BB3E841011646BA5BA801C270ACDB07C091D741BC8160933FCCE6419CA63734FF12FDC1D755F4DE9C61FAC124F14F86378DF8C10103000040010000000D00000075D1847C2FB4FDC1B4F3D780C333ECC1808B4B9FFC1DDC412C5CA55DE6DDE4C12C0587A814C1E9C16C17DC937CA4F1415EF9F4D52430F3C1B8A9D9ABEB96F6C118596D8FA5B2D8C10086530CBA4FB5C15A6985AD2FD9EAC11E6AE24089B300C2FC0534E1052FE7C104576552DC3600C2FA6FF78F0453F6416CB797E2CEE40142DDFA7858E781F8C1C0CB29961A35EBC1F0AE51A6BB4B024220095DD5CD57BC41D510760EB3F0F6C1F061CC16199AFD41A2BEA7C89D44F04148190CCBA317014262AFE5A6B8C3004210B2C12F4E58FB41B2041159B1620242FA9C71155D91FA41BCA92103E61B00420313BA17ABA6F0C1D0849E859F01F141B21CDAC21B090242E11C6620153AFDC13C98A5082E95E6416EF4CAF7D566F841F096E6A1645FCAC175D1847C2FB4FDC1B4F3D780C333ECC1808B4B9FFC1DDC410103000040010000000900000046D87C2BF082F2C127CA46A104B1F4C1F3B224E85C14F1C1F20246182E0BE0C1E1F0DA3341A6F1C11077DDCC33A6C141606C794B45CDC3419A808EC66ED7F3C1404151593255AB414011CB912F87CC41FABA77154AE4F9C13469D92288FCEBC1C8F1B2E8AC14024290FDCE923774CF41603601AFD2CEF9C156A871CA186402421AF32868F889F24178B46BA3E49FE4C1542DFA44D5DDF541E4CCBF0C2C48E9418A410CCD639702C28C4EEDEA6B37DAC152A1E1CEA491F1418012B09ED555A54146D87C2BF082F2C127CA46A104B1F4C1F3B224E85C14F1C1010300004001000000050000000021234A245AB141D06840F80CEEEBC190738D1B4692CF4134EEC514B437F24100FFB955C4C48141606A5FEAC63FD7C1F8838C3D92EAECC158260F13BD7DF941D41127F3BB0EE441526D89229394F9C1487EB3AD97CBFC41164B30F680E6FCC10021234A245AB141D06840F80CEEEBC190738D1B4692CF410103000040010000000B000000F0DCCD40A68BF9C1B2920763B0EEE2C1FE7AAD99C374004218BA9DFD54C1D0C1A00B3D7964D1D0410CEBFA03DEB7F641DC739893D28AFEC1193BC4B2F2C1F4C1FC3FACF38608EF4146DB599FA42AE0C1AE9D5E86A56CEBC175D3CE260A9400C21C95EAA28518E8C1EB4A1855656A00C20456BF82C404EE414408BE49AA41F8413C80B455DC38F4414C4168E2D272FFC15CF0947DADB2FC415C19ADACE8550242CCFCFB994333D6C10C69CBDE7E2AF84100626F1A557500427430CC233FAFF4C1586DD91C4A2BDA4194B2FAC96FE6F34174649FF2E74FE941A01260001439E9C11E48F87B720301429216A28B5DFBEFC1F0DCCD40A68BF9C1B2920763B0EEE2C1FE7AAD99C3740042010300004001000000070000003EF17C724A88FCC180F26360B0289C414891D02F7B02DEC1A2F4051EA6E4F541C414BBA22AABF5C11CA194BA5461FBC1040DF0E05ACAE6418099CEA642FBA3C1785AED381619E641EC77A4CA6F0AFC4180228DA3E5DDB341F5AC45C88B4B02C24C55F04CC134FE416040989B69B4CD41028CF0A7F904F9C1B0067AE2B31201C27A8044BC7672F6411868CE31A5F6DC413EF17C724A88FCC180F26360B0289C414891D02F7B02DEC1':::GEOMETRY,1.725172996520996:::FLOAT8,e'(/\\x16\\fL'::STRING)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 1129fbc650fe3a037b03aea1e5f1d8078618cb1c:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_134627.974910558_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_134627.974910558_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ff:ffff':::INET,'010600000001000000010300000001000000080000000070FAFD6A50D1C1DC805F837BEBE2C16675E53E2FD2F8C1AABC3EAB265101C2D4CA9FB2C6F3ED41EAD7DA574F46FFC14CABFAA3659DF541D490C6DE08A9E0C1949502A6E22BEB4172C2DDA7FBE6F241287112A95B28E54162FEB91D2655F34100F4A045B7AB9BC1F4134BA94450FD410070FAFD6A50D1C1DC805F837BEBE2C1':::GEOMETRY,'10:44:07.962312':::TIME)) AS t(col718_721,col718_722,col718_723,col718_724,col718_725,col718_726,col718_719,col718_720);" []] :[["w7j(/lp"]]
          | QUERY ["SELECT num_nulls(col718_726, col718_720, col718_727, col718_729, col718_724) > 0 FROM (VALUES('ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,'10:44:07.962312':::TIME,'w7j(/lp':::STRING,'1100111001101011000':::STRING,1042:::OID) ) AS T(col718_726,col718_720,col718_727,col718_729,col718_724)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM schema112.table718 WHERE (col718_726, col718_720, col718_727, col718_729, col718_724)= ( SELECT  (col718_726, col718_720, col718_727, col718_729, col718_724) FROM (VALUES( 'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,'10:44:07.962312':::TIME,'w7j(/lp':::STRING,'1100111001101011000':::STRING,1042:::OID) ) AS T(col718_726,col718_720,col718_727,col718_729,col718_724) ) )" []] :false
          | QUERY ["SELECT array[((col718_726, col718_720, col718_727, col718_729, col718_724))::STRING] FROM (VALUES('ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,'10:44:07.962312':::TIME,'w7j(/lp':::STRING,'1100111001101011000':::STRING,1042:::OID) ) AS T(col718_726,col718_720,col718_727,col718_729,col718_724)" []] :[["(ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff,10:44:07.962312,\"w7j(/lp\",1100111001101011000,1042)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema112.table718'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema112' AND\n\t\t\t\t\t\t\t\t    table_name='table718' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema112.table718', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema112'\n\t\t           AND table_name = 'table718'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema791.seq1169 INSERT INTO schema112.table718 (col718_719,col718_720,col718_721,col718_722,col718_723,col718_724,col718_725,col718_726) VALUES ('010600000001000000010300000001000000080000000070FAFD6A50D1C1DC805F837BEBE2C16675E53E2FD2F8C1AABC3EAB265101C2D4CA9FB2C6F3ED41EAD7DA574F46FFC14CABFAA3659DF541D490C6DE08A9E0C1949502A6E22BEB4172C2DDA7FBE6F241287112A95B28E54162FEB91D2655F34100F4A045B7AB9BC1F4134BA94450FD410070FAFD6A50D1C1DC805F837BEBE2C1':::GEOMETRY,'10:44:07.962312':::TIME,0.4319920165642957:::FLOAT8,B'1100111001101011000','-57 years -8 mons -658 days -16:17:00.682251':::INTERVAL,1042:::OID,'W7j(/lp'::STRING,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 574F46FFC14CABFAA3659DF541D490C6DE08A9E0C1949502A6E22BEB4172C2DDA7FBE6F241287112A95B28E54162FEB91D2655F34100F4A045B7AB9BC1F4134BA94450FD410070FAFD6A50D1C1DC805F837BEBE2C1':::GEOMETRY,'10:44:07.962312':::TIME)) AS t(col718_721,col718_722,col718_723,col718_724,col718_725,col718_726,col718_719,col718_720);\" []] :[[\"w7j(/lp\"]]\nQUERY [\"SELECT num_nulls(col718_726, col718_720, col718_727, col718_729, col718_724) \u003e 0 FROM (VALUES('ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,'10:44:07.962312':::TIME,'w7j(/lp':::STRING,'1100111001101011000':::STRING,1042:::OID) ) AS T(col718_726,col718_720,col718_727,col718_729,col718_724)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema112.table718 WHERE (col718_726, col718_720, col718_727, col718_729, col718_724)= ( SELECT  (col718_726, col718_720, col718_727, col718_729, col718_724) FROM (VALUES( 'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,'10:44:07.962312':::TIME,'w7j(/lp':::STRING,'1100111001101011000':::STRING,1042:::OID) ) AS T(col718_726,col718_720,col718_727,col718_729,col718_724) ) )\" []] :false\nQUERY [\"SELECT array[((col718_726, col718_720, col718_727, col718_729, col718_724))::STRING] FROM (VALUES('ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,'10:44:07.962312':::TIME,'w7j(/lp':::STRING,'1100111001101011000':::STRING,1042:::OID) ) AS T(col718_726,col718_720,col718_727,col718_729,col718_724)\" []] :[[\"(ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff,10:44:07.962312,\\\"w7j(/lp\\\",1100111001101011000,1042)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema112.table718'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema112' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table718' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema112.table718', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema112'\\n\\t\\t           AND table_name = 'table718'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema791.seq1169 INSERT INTO schema112.table718 (col718_719,col718_720,col718_721,col718_722,col718_723,col718_724,col718_725,col718_726) VALUES ('010600000001000000010300000001000000080000000070FAFD6A50D1C1DC805F837BEBE2C16675E53E2FD2F8C1AABC3EAB265101C2D4CA9FB2C6F3ED41EAD7DA574F46FFC14CABFAA3659DF541D490C6DE08A9E0C1949502A6E22BEB4172C2DDA7FBE6F241287112A95B28E54162FEB91D2655F34100F4A045B7AB9BC1F4134BA94450FD410070FAFD6A50D1C1DC805F837BEBE2C1':::GEOMETRY,'10:44:07.962312':::TIME,0.4319920165642957:::FLOAT8,B'1100111001101011000','-57 years -8 mons -658 days -16:17:00.682251':::INTERVAL,1042:::OID,'W7j(/lp'::STRING,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ cd113bf428e8868c38b2f2f295210f5c30d13a2d:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_140334.196375001_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_140334.196375001_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ious statements [INSERT INTO schema838.table1227 (col1227_1228,col1227_1229,col1227_1230,col1227_1231,col1227_1232,col1227_1233,col1227_1234,col1227_1235,col1227_1236,col1227_1237,col1227_1238,col1227_1239,col1227_1240,col1227_1241) VALUES ((-0.6244726594547083):::FLOAT8,(-0.33513265588858765):::FLOAT8,'-10 years -1 mons -963 days -05:36:45.651089':::INTERVAL,(-0.1309162974357605):::FLOAT8,'\x9ff57636fa':::BYTES,'01020000400A0000001DED04BA7DFE01C2D0B43532CD10E7C1367A16892E350242E1CCEE07D803F1C1BAE4A16D1A41FAC116618A8E074EF241981586DBBF7CE5C1E2EF3B8AC358FEC162907C1CA272FFC1604E3492F113BB41B327A54834C600C2B0D133CA9E48CE41605C7EF2087FC941A812905DB2BCF6C194C79E303552F741807D83DE2763F8418089C3FF968899C144F10E6A45D9F5C1645A1A31976CF241E45EE93902A8FB4137359F55F782F5C100E2439F8276D441ECB445335FCCF141F6063EA583B8F54140C28F59AD73CA414A1AEADD528300421029692086FAF0C1102A1579189EEDC1D4443C6B3237FC4144F64FF5CB9601C2':::GEOMETRY,'"bar"':::JSONB,'12 years 8 mons 422 days 23:39:29.694105':::INTERVAL,'W'::"char",4316980952942910951:::INT8,'#'::CHAR,(-0.09436736469957485):::FLOAT8,'BOX(0.47191703220180703 0.30870163019847774,0.6129597376893547 0.5240367951246152)':::BOX2D,'0104000060E610000002000000010100004068B809C34CA164408DF10F3E0F8254C0D87534478024E3C101010000409BFC3CEC8D1D51C014FBB9D7CCF639C0B88077D1C9CADF41':::GEOGRAPHY) INSERT INTO public.table1204 (col1204_1206,col1204_1207,col1204_1208,col1204_1209,col1204_1210,col1204_1211,col1204_1212,col1204_1213,col1204_1214,col1204_1215,col1204_1216,col1204_1217,col1204_1218,col1204_1219) VALUES ('2009-08-19 04:14:05.000968+00:00':::TIMESTAMPTZ,1.796717643737793:::FLOAT8,'01:17:50.838608':::TIME,'0106000000020000000103000000010000000B000000028CF0660FD101C218AF4132C87A00C2F431756E2B62D8C17CD2FA5A8A2EEAC1E0BCF56D29A6DBC12F6F48B0252AFAC1D01513C605B2E541E4C16B4C54B6FBC1B4AB5D77052DE641ECBE031B594DF5C19CA6C105F611FF41B8B087B8B9F0D0C18074ADE3BCE0AD41682DAE6EBC30D9414C9ABCDDB3D9DFC1944E4567EEA1FA41D8F739C2266DE3C140110E4731A8A9418E10B2B57871F5C1F0E6DB24398CD341028CF0660FD101C218AF4132C87A00C20103000000010000000B0000005CC2AC2F4A37FAC1F28734FF3713E5C1784E21FAB2C9F7C12214BE986DBDF5C136373D019718F941AB30AF1C3F7B02C244466615B216F0419DA9F19C8F09F1C1D4FEA4E8C635FC414221BC5574FAE3C1B45432E04309FC4108063CC550B2D0410EA2BBC96224024200E4F79DEF6BFA414060C9DF1E8DE541ACF3C1D3F2B3FE41C0563C7D4AB7B04140B266EFC4A1ED417C42E302031AF0C190E70AB07234E1415CC2AC2F4A37FAC1F28734FF3713E5C1':::GEOMETRY,20:::OID,0.16950203478336334:::FLOAT8,e'{"9$=@+_": "}/sILB5\\";um0", "b": ["baz"], "rif|;Pghykqe": []}':::JSONB,'07:39:16.11461':::TIME,2202:::OID,1082:::OID,'X'::"char",'03:01:55.252309+13:22:00':::TIMETZ,'2006-10-01 13:58:38.000882':::TIMESTAMP,'14:22:32.511635':::TIME),('2003-06-26 02:53:38.000967+00:00':::TIMESTAMPTZ,0.30741459131240845:::FLOAT8,'24:00:00':::TIME,'01060000000200000001030000000100000004000000000000000000444000000000000044400000000000003440000000000080464000000000008046400000000000003E4000000000000044400000000000004440010300000002000000060000000000000000003440000000000080414000000000000024400000000000003E40000000000000244000000000000024400000000000003E4000000000000014400000000000804640000000000000344000000000000034400000000000804140040000000000000000003E40000000000000344000000000000034400000000000002E40000000000000344000000000000039400000000000003E400000000000003440':::GEOMETRY,2950:::OID,0.34278255701065063:::FLOAT8,'null':::JSONB,'15:12:17.518276':::TIME,1560:::OID,17:::OID,'@'::"char",'03:20:55.427448+12:44:00':::TIMETZ,'1996-10-07 11:14:15.000317':::TIMESTAMP,NULL),('2013-05-24 23:41:34.000536+00:00':::TIMESTAMPTZ,1.0683718919754028:::FLOAT8,'18:06:09.945423':::TIME,'010500000000000000':::GEOMETRY,4089:::OID,(-0.3656197488307953):::FLOAT8,e'{"\'%an\\\\TU?u": {}, "b": {"ZU[$7~": [], "foobar": [[], [[]], [], {}]}, "bar": [{"4#hxcp$1": "ZBySh_"}, {}, {}]}':::JSONB,'07:45:50.65314':::TIME,1270:::OID,2205:::OID,e'\f'::"char",'09:54:29.301129+04:40:00':::TIMETZ,'-2000-01-01 00:00:00':::TIMESTAMP,'16:38:03.049822':::TIME)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 7 (col1227_1228,col1227_1229,col1227_1230,col1227_1231,col1227_1232,col1227_1233,col1227_1234,col1227_1235,col1227_1236,col1227_1237,col1227_1238,col1227_1239,col1227_1240,col1227_1241) VALUES ((-0.6244726594547083):::FLOAT8,(-0.33513265588858765):::FLOAT8,'-10 years -1 mons -963 days -05:36:45.651089':::INTERVAL,(-0.1309162974357605):::FLOAT8,'\\x9ff57636fa':::BYTES,'01020000400A0000001DED04BA7DFE01C2D0B43532CD10E7C1367A16892E350242E1CCEE07D803F1C1BAE4A16D1A41FAC116618A8E074EF241981586DBBF7CE5C1E2EF3B8AC358FEC162907C1CA272FFC1604E3492F113BB41B327A54834C600C2B0D133CA9E48CE41605C7EF2087FC941A812905DB2BCF6C194C79E303552F741807D83DE2763F8418089C3FF968899C144F10E6A45D9F5C1645A1A31976CF241E45EE93902A8FB4137359F55F782F5C100E2439F8276D441ECB445335FCCF141F6063EA583B8F54140C28F59AD73CA414A1AEADD528300421029692086FAF0C1102A1579189EEDC1D4443C6B3237FC4144F64FF5CB9601C2':::GEOMETRY,'\"bar\"':::JSONB,'12 years 8 mons 422 days 23:39:29.694105':::INTERVAL,'W'::\"char\",4316980952942910951:::INT8,'#'::CHAR,(-0.09436736469957485):::FLOAT8,'BOX(0.47191703220180703 0.30870163019847774,0.6129597376893547 0.5240367951246152)':::BOX2D,'0104000060E610000002000000010100004068B809C34CA164408DF10F3E0F8254C0D87534478024E3C101010000409BFC3CEC8D1D51C014FBB9D7CCF639C0B88077D1C9CADF41':::GEOGRAPHY) INSERT INTO public.table1204 (col1204_1206,col1204_1207,col1204_1208,col1204_1209,col1204_1210,col1204_1211,col1204_1212,col1204_1213,col1204_1214,col1204_1215,col1204_1216,col1204_1217,col1204_1218,col1204_1219) VALUES ('2009-08-19 04:14:05.000968+00:00':::TIMESTAMPTZ,1.796717643737793:::FLOAT8,'01:17:50.838608':::TIME,'0106000000020000000103000000010000000B000000028CF0660FD101C218AF4132C87A00C2F431756E2B62D8C17CD2FA5A8A2EEAC1E0BCF56D29A6DBC12F6F48B0252AFAC1D01513C605B2E541E4C16B4C54B6FBC1B4AB5D77052DE641ECBE031B594DF5C19CA6C105F611FF41B8B087B8B9F0D0C18074ADE3BCE0AD41682DAE6EBC30D9414C9ABCDDB3D9DFC1944E4567EEA1FA41D8F739C2266DE3C140110E4731A8A9418E10B2B57871F5C1F0E6DB24398CD341028CF0660FD101C218AF4132C87A00C20103000000010000000B0000005CC2AC2F4A37FAC1F28734FF3713E5C1784E21FAB2C9F7C12214BE986DBDF5C136373D019718F941AB30AF1C3F7B02C244466615B216F0419DA9F19C8F09F1C1D4FEA4E8C635FC414221BC5574FAE3C1B45432E04309FC4108063CC550B2D0410EA2BBC96224024200E4F79DEF6BFA414060C9DF1E8DE541ACF3C1D3F2B3FE41C0563C7D4AB7B04140B266EFC4A1ED417C42E302031AF0C190E70AB07234E1415CC2AC2F4A37FAC1F28734FF3713E5C1':::GEOMETRY,20:::OID,0.16950203478336334:::FLOAT8,e'{\"9$=@+_\": \"}/sILB5\\\\\";um0\", \"b\": [\"baz\"], \"rif|;Pghykqe\": []}':::JSONB,'07:39:16.11461':::TIME,2202:::OID,1082:::OID,'X'::\"char\",'03:01:55.252309+13:22:00':::TIMETZ,'2006-10-01 13:58:38.000882':::TIMESTAMP,'14:22:32.511635':::TIME),('2003-06-26 02:53:38.000967+00:00':::TIMESTAMPTZ,0.30741459131240845:::FLOAT8,'24:00:00':::TIME,'01060000000200000001030000000100000004000000000000000000444000000000000044400000000000003440000000000080464000000000008046400000000000003E4000000000000044400000000000004440010300000002000000060000000000000000003440000000000080414000000000000024400000000000003E40000000000000244000000000000024400000000000003E4000000000000014400000000000804640000000000000344000000000000034400000000000804140040000000000000000003E40000000000000344000000000000034400000000000002E40000000000000344000000000000039400000000000003E400000000000003440':::GEOMETRY,2950:::OID,0.34278255701065063:::FLOAT8,'null':::JSONB,'15:12:17.518276':::TIME,1560:::OID,17:::OID,'@'::\"char\",'03:20:55.427448+12:44:00':::TIMETZ,'1996-10-07 11:14:15.000317':::TIMESTAMP,NULL),('2013-05-24 23:41:34.000536+00:00':::TIMESTAMPTZ,1.0683718919754028:::FLOAT8,'18:06:09.945423':::TIME,'010500000000000000':::GEOMETRY,4089:::OID,(-0.3656197488307953):::FLOAT8,e'{\"\\'%an\\\\\\\\TU?u\": {}, \"b\": {\"ZU[$7~\": [], \"foobar\": [[], [[]], [], {}]}, \"bar\": [{\"4#hxcp$1\": \"ZBySh_\"}, {}, {}]}':::JSONB,'07:45:50.65314':::TIME,1270:::OID,2205:::OID,e'\\f'::\"char\",'09:54:29.301129+04:40:00':::TIMETZ,'-2000-01-01 00:00:00':::TIMESTAMP,'16:38:03.049822':::TIME)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 0352fbee5e6933e8bca28aae4b136bcd495ab227:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_201955.792640486_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_201955.792640486_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 8-08':::DATE,'BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)':::BOX2D) ) AS T(col1215_1223,col1215_1227,col1215_1228,col1215_1219,col1215_1229,col1215_1221,col1215_1224,col1215_1225)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM public.table1215 WHERE (col1215_1227, col1215_1228, col1215_1219, col1215_1229, col1215_1221, col1215_1224, col1215_1225, col1215_1223)= ( SELECT  (col1215_1227, col1215_1228, col1215_1219, col1215_1229, col1215_1221, col1215_1224, col1215_1225, col1215_1223) FROM (VALUES( 1082:::OID,'55.224.92.220/30':::INET,'18:02:53.30375+13:54:00':::STRING,2205:::OID,e'X]c\\x07[5k+=':::STRING,19:::OID,'1977-08-08':::DATE,'BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)':::BOX2D) ) AS T(col1215_1223,col1215_1227,col1215_1228,col1215_1219,col1215_1229,col1215_1221,col1215_1224,col1215_1225) ) )" []] :false
          | QUERY ["SELECT array[((col1215_1227, col1215_1228, col1215_1219, col1215_1229, col1215_1221, col1215_1224, col1215_1225, col1215_1223))::STRING] FROM (VALUES(1082:::OID,'55.224.92.220/30':::INET,'18:02:53.30375+13:54:00':::STRING,2205:::OID,e'X]c\\x07[5k+=':::STRING,19:::OID,'1977-08-08':::DATE,'BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)':::BOX2D) ) AS T(col1215_1223,col1215_1227,col1215_1228,col1215_1219,col1215_1229,col1215_1221,col1215_1224,col1215_1225)" []] :[["(55.224.92.220/30,18:02:53.30375+13:54:00,2205,X]c\a[5k+=,19,1977-08-08,\"BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)\",1082)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table1215'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table1215' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table1215', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table1215'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS public.seq1519 INSERT INTO public.table1215 (col1215_1216,col1215_1217,col1215_1218,col1215_1219,col1215_1220,col1215_1221,col1215_1222,col1215_1223,col1215_1224,col1215_1225,col1215_1226,col1215_1227) VALUES ('18:02:53.30375+13:54:00':::TIMETZ,ARRAY[B'00100101010000010100110000001100',B'01110111000101000000010111011001',B'10000111101100111011000001001011',B'11110010001100001001101000100010'],'2015-08-08 18:29:03.000373+00:00':::TIMESTAMPTZ,2205:::OID,B'010101001100011000001111',19:::OID,(-9.109498301556629271E+31):::DECIMAL,1082:::OID,'1977-08-08':::DATE,'BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)':::BOX2D,ARRAY[e'J&\x0bqat7':::STRING],'55.224.92.220/30':::INET)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 5_1228,col1215_1219,col1215_1229,col1215_1221,col1215_1224,col1215_1225)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM public.table1215 WHERE (col1215_1227, col1215_1228, col1215_1219, col1215_1229, col1215_1221, col1215_1224, col1215_1225, col1215_1223)= ( SELECT  (col1215_1227, col1215_1228, col1215_1219, col1215_1229, col1215_1221, col1215_1224, col1215_1225, col1215_1223) FROM (VALUES( 1082:::OID,'55.224.92.220/30':::INET,'18:02:53.30375+13:54:00':::STRING,2205:::OID,e'X]c\\\\x07[5k+=':::STRING,19:::OID,'1977-08-08':::DATE,'BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)':::BOX2D) ) AS T(col1215_1223,col1215_1227,col1215_1228,col1215_1219,col1215_1229,col1215_1221,col1215_1224,col1215_1225) ) )\" []] :false\nQUERY [\"SELECT array[((col1215_1227, col1215_1228, col1215_1219, col1215_1229, col1215_1221, col1215_1224, col1215_1225, col1215_1223))::STRING] FROM (VALUES(1082:::OID,'55.224.92.220/30':::INET,'18:02:53.30375+13:54:00':::STRING,2205:::OID,e'X]c\\\\x07[5k+=':::STRING,19:::OID,'1977-08-08':::DATE,'BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)':::BOX2D) ) AS T(col1215_1223,col1215_1227,col1215_1228,col1215_1219,col1215_1229,col1215_1221,col1215_1224,col1215_1225)\" []] :[[\"(55.224.92.220/30,18:02:53.30375+13:54:00,2205,X]c\\a[5k+=,19,1977-08-08,\\\"BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)\\\",1082)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table1215'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table1215' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table1215', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table1215'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS public.seq1519 INSERT INTO public.table1215 (col1215_1216,col1215_1217,col1215_1218,col1215_1219,col1215_1220,col1215_1221,col1215_1222,col1215_1223,col1215_1224,col1215_1225,col1215_1226,col1215_1227) VALUES ('18:02:53.30375+13:54:00':::TIMETZ,ARRAY[B'00100101010000010100110000001100',B'01110111000101000000010111011001',B'10000111101100111011000001001011',B'11110010001100001001101000100010'],'2015-08-08 18:29:03.000373+00:00':::TIMESTAMPTZ,2205:::OID,B'010101001100011000001111',19:::OID,(-9.109498301556629271E+31):::DECIMAL,1082:::OID,'1977-08-08':::DATE,'BOX(-1.9127286571846058 -0.8691400631787735,0.030533909965883232 1.2281994446407474)':::BOX2D,ARRAY[e'J\u0026\\x0bqat7':::STRING],'55.224.92.220/30':::INET)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ b83e83b7e54fabf6ebd3a42cff67f4b8c5b7b863:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133959.927621555_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_133959.927621555_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 34' AND\n\t\t\t\t\t\t\t\t    table_name='table891' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema834.table891', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema834'\n\t\t           AND table_name = 'table891'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() CREATE TABLE IF NOT EXISTS public.table2687 (col2687_2688 REGPROCEDURE, col2687_2689 REGPROC NULL, col2687_2690 UUID NOT NULL, col2687_2691 GEOGRAPHY NULL, col2687_2692 BIT(44) NULL, col2687_2693 VARCHAR NOT NULL, col2687_2694 TIMESTAMP NOT NULL, col2687_2695 STRING AS (CASE WHEN col2687_2688 IS NULL THEN ':}':::STRING ELSE e'\x04':::STRING END) VIRTUAL, col2687_2696 STRING AS (CASE WHEN col2687_2688 IS NULL THEN e'\f[5\x06\x15':::STRING ELSE e'\'':::STRING END) VIRTUAL, col2687_2697 STRING NULL AS (lower(CAST(col2687_2692 AS STRING))) STORED, col2687_2698 STRING NULL AS (lower(CAST(col2687_2691 AS STRING))) STORED, col2687_2699 STRING NOT NULL AS (lower(col2687_2693)) VIRTUAL, col2687_2700 STRING AS (CASE WHEN col2687_2688 IS NULL THEN e's\x14\x7f\x11\x17i[X\r':::STRING ELSE e'@\x1dr':::STRING END) VIRTUAL, UNIQUE (col2687_2693, col2687_2695, col2687_2698 DESC, col2687_2697 ASC), FAMILY (col2687_2694, col2687_2693, col2687_2691), FAMILY (col2687_2688, col2687_2692, col2687_2690, col2687_2689), FAMILY (col2687_2698), FAMILY (col2687_2697)) INSERT INTO schema834.table891 (col891_892,col891_893,col891_894,col891_895,col891_896,col891_897,col891_898,col891_899,col891_900,col891_901,col891_902,col891_903,col891_904,col891_905,col891_906,col891_907) VALUES ('93.148.38.110/17':::INET,e'y\t'::VARCHAR,'1987-12-09':::DATE,7602014641768752698:::INT8,(-1.1184642101467401):::FLOAT8,0:::INT8,B'0000000000000000000000000000000000000000000000000','1970-01-01':::DATE,false,B'010100000110000011110000100110',4096:::OID,1082:::OID,true,1186:::OID,18595:::INT8,(-62448210):::INT8) INSERT INTO schema834.table891 (col891_892,col891_893,col891_894,col891_895,col891_896,col891_897,col891_898,col891_899,col891_900,col891_901,col891_902,col891_903,col891_904,col891_905,col891_906,col891_907) VALUES ('109.182.212.14/13':::INET,'"'::VARCHAR,'1984-02-28':::DATE,2044499603715263107:::INT8,1.4792555694010043:::FLOAT8,(-14629):::INT8,B'0100010100100001010011100001001000011011100010100','1982-09-04':::DATE,false,NULL,1700:::OID,90004:::OID,true,2205:::OID,19428:::INT8,(-356155902):::INT8),('252.125.156.14/19':::INET,''::VARCHAR,'1981-05-16':::DATE,32767:::INT8,(-0.3798724731834566):::FLOAT8,(-29500):::INT8,B'1010110111011101001110010101001110111000011000100','1985-07-23':::DATE,true,B'111001010100101000100100000010',1266:::OID,2202:::OID,true,18:::OID,16351:::INT8,(-702564597):::INT8),('117.127.220.116/0':::INET,'s$<'::VARCHAR,'1977-08-20':::DATE,8779339269782111848:::INT8,0.3388516770457703:::FLOAT8,(-26475):::INT8,B'1101010010011010001110000010010101001001001100101','1985-10-18':::DATE,false,B'001010011110001110001000110001',2206:::OID,1184:::OID,false,1083:::OID,27020:::INT8,1943736625:::INT8)]: ERROR: relation "table2687" (701): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | nal.is_constraint_active('schema834.table891', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema834'\\n\\t\\t           AND table_name = 'table891'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() CREATE TABLE IF NOT EXISTS public.table2687 (col2687_2688 REGPROCEDURE, col2687_2689 REGPROC NULL, col2687_2690 UUID NOT NULL, col2687_2691 GEOGRAPHY NULL, col2687_2692 BIT(44) NULL, col2687_2693 VARCHAR NOT NULL, col2687_2694 TIMESTAMP NOT NULL, col2687_2695 STRING AS (CASE WHEN col2687_2688 IS NULL THEN ':}':::STRING ELSE e'\\x04':::STRING END) VIRTUAL, col2687_2696 STRING AS (CASE WHEN col2687_2688 IS NULL THEN e'\\f[5\\x06\\x15':::STRING ELSE e'\\'':::STRING END) VIRTUAL, col2687_2697 STRING NULL AS (lower(CAST(col2687_2692 AS STRING))) STORED, col2687_2698 STRING NULL AS (lower(CAST(col2687_2691 AS STRING))) STORED, col2687_2699 STRING NOT NULL AS (lower(col2687_2693)) VIRTUAL, col2687_2700 STRING AS (CASE WHEN col2687_2688 IS NULL THEN e's\\x14\\x7f\\x11\\x17i[X\\r':::STRING ELSE e'@\\x1dr':::STRING END) VIRTUAL, UNIQUE (col2687_2693, col2687_2695, col2687_2698 DESC, col2687_2697 ASC), FAMILY (col2687_2694, col2687_2693, col2687_2691), FAMILY (col2687_2688, col2687_2692, col2687_2690, col2687_2689), FAMILY (col2687_2698), FAMILY (col2687_2697)) INSERT INTO schema834.table891 (col891_892,col891_893,col891_894,col891_895,col891_896,col891_897,col891_898,col891_899,col891_900,col891_901,col891_902,col891_903,col891_904,col891_905,col891_906,col891_907) VALUES ('93.148.38.110/17':::INET,e'y\\t'::VARCHAR,'1987-12-09':::DATE,7602014641768752698:::INT8,(-1.1184642101467401):::FLOAT8,0:::INT8,B'0000000000000000000000000000000000000000000000000','1970-01-01':::DATE,false,B'010100000110000011110000100110',4096:::OID,1082:::OID,true,1186:::OID,18595:::INT8,(-62448210):::INT8) INSERT INTO schema834.table891 (col891_892,col891_893,col891_894,col891_895,col891_896,col891_897,col891_898,col891_899,col891_900,col891_901,col891_902,col891_903,col891_904,col891_905,col891_906,col891_907) VALUES ('109.182.212.14/13':::INET,'\"'::VARCHAR,'1984-02-28':::DATE,2044499603715263107:::INT8,1.4792555694010043:::FLOAT8,(-14629):::INT8,B'0100010100100001010011100001001000011011100010100','1982-09-04':::DATE,false,NULL,1700:::OID,90004:::OID,true,2205:::OID,19428:::INT8,(-356155902):::INT8),('252.125.156.14/19':::INET,''::VARCHAR,'1981-05-16':::DATE,32767:::INT8,(-0.3798724731834566):::FLOAT8,(-29500):::INT8,B'1010110111011101001110010101001110111000011000100','1985-07-23':::DATE,true,B'111001010100101000100100000010',1266:::OID,2202:::OID,true,18:::OID,16351:::INT8,(-702564597):::INT8),('117.127.220.116/0':::INET,'s$\u003c'::VARCHAR,'1977-08-20':::DATE,8779339269782111848:::INT8,0.3388516770457703:::FLOAT8,(-26475):::INT8,B'1101010010011010001110000010010101001001001100101','1985-10-18':::DATE,false,B'001010011110001110001000110001',2206:::OID,1184:::OID,false,1083:::OID,27020:::INT8,1943736625:::INT8)]: ERROR: relation \"table2687\" (701): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 748b89a3fbef294f4b0f930c9dbdf88294b3deeb:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133546.438743933_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_133546.438743933_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |           ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["public.table1362"]] :[]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table1362'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table1362' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table1362', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table1362'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [INSERT INTO public.table1909 (col1909_1910,col1909_1911,col1909_1912,col1909_1913,col1909_1914,col1909_1915,col1909_1916) VALUES ('2031-01-20 09:31:10.000805+00:00':::TIMESTAMPTZ,1146161460:::INT8,701:::OID,'X'::NAME:::NAME,e'@\x14Mq'::NAME:::NAME,'1982-04-17':::DATE,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET),('1982-11-21 17:10:09.000836+00:00':::TIMESTAMPTZ,1674598644:::INT8,90002:::OID,e'\x1b=ax\f-P4h'::NAME:::NAME,e'\x0e=a~t\x06[h<'::NAME:::NAME,'1979-05-31':::DATE,'205.188.206.20':::INET),('2000-05-20 10:39:32.000121+00:00':::TIMESTAMPTZ,(-2117337491):::INT8,1114:::OID,e'\tX<\\C\x03Y,'::NAME:::NAME,''::NAME:::NAME,'1996-11-01':::DATE,'d8fb:2bc:269e:3c99:5af5:5007:fec:ead7/127':::INET) INSERT INTO public.table1362 (col874_875,col864_871,col864_869,col864_870) VALUES (1043:::OID,701:::OID,e'[[[[{"iB^\\\\(\\"q6SOg": {}}]], [{"a})F]hV": {"c": []}}, [false]], {}], null, {}, true]':::JSONB,ARRAY[]:::BIT(8)[])]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | mputeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"public.table1362\"]] :[]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table1362'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table1362' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table1362', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table1362'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO public.table1909 (col1909_1910,col1909_1911,col1909_1912,col1909_1913,col1909_1914,col1909_1915,col1909_1916) VALUES ('2031-01-20 09:31:10.000805+00:00':::TIMESTAMPTZ,1146161460:::INT8,701:::OID,'X'::NAME:::NAME,e'@\\x14Mq'::NAME:::NAME,'1982-04-17':::DATE,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET),('1982-11-21 17:10:09.000836+00:00':::TIMESTAMPTZ,1674598644:::INT8,90002:::OID,e'\\x1b=ax\\f-P4h'::NAME:::NAME,e'\\x0e=a~t\\x06[h\u003c'::NAME:::NAME,'1979-05-31':::DATE,'205.188.206.20':::INET),('2000-05-20 10:39:32.000121+00:00':::TIMESTAMPTZ,(-2117337491):::INT8,1114:::OID,e'\\tX\u003c\\\\C\\x03Y,'::NAME:::NAME,''::NAME:::NAME,'1996-11-01':::DATE,'d8fb:2bc:269e:3c99:5af5:5007:fec:ead7/127':::INET) INSERT INTO public.table1362 (col874_875,col864_871,col864_869,col864_870) VALUES (1043:::OID,701:::OID,e'[[[[{\"iB^\\\\\\\\(\\\\\"q6SOg\": {}}]], [{\"a})F]hV\": {\"c\": []}}, [false]], {}], null, {}, true]':::JSONB,ARRAY[]:::BIT(8)[])]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 7326bbbf94610c40ae2d2fdf42c119dfea93a170:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133619.493287271_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_133619.493287271_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | :26:55.510518':::INTERVAL,NULL,'0001-01-01 00:00:00':::TIMESTAMP,false,'\\xd7988667a5c098dc8c':::BYTES,1043:::OID,1114:::OID)) AS t(col1457_1466,col1457_1467,col1457_1458,col1457_1460,col1457_1461,col1457_1462,col1457_1464,col1457_1465,col1457_1459,col1457_1463);" []] :[["n"]]
          | QUERY ["SELECT array[(lower(CAST(col1457_1464 AS STRING)))::STRING::STRING] AS c FROM ( VALUES(ARRAY[B'010010101101111010000000100011010010110110111'],'3e4b:859f:40f1:5822:6461:2c00:c05b:5968/74':::INET,'-20 years -11 mons -893 days -05:26:55.510518':::INTERVAL,NULL,'0001-01-01 00:00:00':::TIMESTAMP,false,'\\xd7988667a5c098dc8c':::BYTES,'010500008000000000':::GEOMETRY,1043:::OID,1114:::OID)) AS t(col1457_1467,col1457_1458,col1457_1460,col1457_1461,col1457_1462,col1457_1464,col1457_1465,col1457_1466,col1457_1459,col1457_1463);" []] :[["false"]]
          | QUERY ["SELECT array[(lower(CAST(col1457_1459 AS STRING)))::STRING::STRING] AS c FROM ( VALUES(1043:::OID,1114:::OID,'\\xd7988667a5c098dc8c':::BYTES,'010500008000000000':::GEOMETRY,ARRAY[B'010010101101111010000000100011010010110110111'],'3e4b:859f:40f1:5822:6461:2c00:c05b:5968/74':::INET,'-20 years -11 mons -893 days -05:26:55.510518':::INTERVAL,NULL,'0001-01-01 00:00:00':::TIMESTAMP,false)) AS t(col1457_1459,col1457_1463,col1457_1465,col1457_1466,col1457_1467,col1457_1458,col1457_1460,col1457_1461,col1457_1462,col1457_1464);" []] :[["1043"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema1422.table1457'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema1422' AND\n\t\t\t\t\t\t\t\t    table_name='table1457' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema1422.table1457', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema1422'\n\t\t           AND table_name = 'table1457'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [DROP TABLE IF EXISTS schema1422.table1699 CASCADE INSERT INTO schema1422.table1457 (col1457_1458,col1457_1459,col1457_1460,col1457_1461,col1457_1462,col1457_1463,col1457_1464,col1457_1465,col1457_1466,col1457_1467) VALUES ('9f31:3b8b:6322:9181:a637:ced3:6d9f:243a/94':::INET,26:::OID,'70 years 3 mons 276 days 07:33:33.798993':::INTERVAL,'null':::JSONB,'1997-04-30 20:50:37.000234':::TIMESTAMP,90000:::OID,false,NULL,'010200000000000000':::GEOMETRY,ARRAY[B'011111111111111111111111111111111111111111111',B'110001011110010111011011101100101001001010100',B'101011100010110110011000001001011000111110010',B'000001110010010011011011110001100000111001001']),('3e4b:859f:40f1:5822:6461:2c00:c05b:5968/74':::INET,1043:::OID,'-20 years -11 mons -893 days -05:26:55.510518':::INTERVAL,NULL,'0001-01-01 00:00:00':::TIMESTAMP,1114:::OID,false,'\xd7988667a5c098dc8c':::BYTES,'010500008000000000':::GEOMETRY,ARRAY[B'010010101101111010000000100011010010110110111'])]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 57_1466,col1457_1467,col1457_1458,col1457_1460,col1457_1461,col1457_1462,col1457_1464,col1457_1465,col1457_1459,col1457_1463);\" []] :[[\"n\"]]\nQUERY [\"SELECT array[(lower(CAST(col1457_1464 AS STRING)))::STRING::STRING] AS c FROM ( VALUES(ARRAY[B'010010101101111010000000100011010010110110111'],'3e4b:859f:40f1:5822:6461:2c00:c05b:5968/74':::INET,'-20 years -11 mons -893 days -05:26:55.510518':::INTERVAL,NULL,'0001-01-01 00:00:00':::TIMESTAMP,false,'\\\\xd7988667a5c098dc8c':::BYTES,'010500008000000000':::GEOMETRY,1043:::OID,1114:::OID)) AS t(col1457_1467,col1457_1458,col1457_1460,col1457_1461,col1457_1462,col1457_1464,col1457_1465,col1457_1466,col1457_1459,col1457_1463);\" []] :[[\"false\"]]\nQUERY [\"SELECT array[(lower(CAST(col1457_1459 AS STRING)))::STRING::STRING] AS c FROM ( VALUES(1043:::OID,1114:::OID,'\\\\xd7988667a5c098dc8c':::BYTES,'010500008000000000':::GEOMETRY,ARRAY[B'010010101101111010000000100011010010110110111'],'3e4b:859f:40f1:5822:6461:2c00:c05b:5968/74':::INET,'-20 years -11 mons -893 days -05:26:55.510518':::INTERVAL,NULL,'0001-01-01 00:00:00':::TIMESTAMP,false)) AS t(col1457_1459,col1457_1463,col1457_1465,col1457_1466,col1457_1467,col1457_1458,col1457_1460,col1457_1461,col1457_1462,col1457_1464);\" []] :[[\"1043\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema1422.table1457'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema1422' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table1457' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema1422.table1457', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema1422'\\n\\t\\t           AND table_name = 'table1457'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [DROP TABLE IF EXISTS schema1422.table1699 CASCADE INSERT INTO schema1422.table1457 (col1457_1458,col1457_1459,col1457_1460,col1457_1461,col1457_1462,col1457_1463,col1457_1464,col1457_1465,col1457_1466,col1457_1467) VALUES ('9f31:3b8b:6322:9181:a637:ced3:6d9f:243a/94':::INET,26:::OID,'70 years 3 mons 276 days 07:33:33.798993':::INTERVAL,'null':::JSONB,'1997-04-30 20:50:37.000234':::TIMESTAMP,90000:::OID,false,NULL,'010200000000000000':::GEOMETRY,ARRAY[B'011111111111111111111111111111111111111111111',B'110001011110010111011011101100101001001010100',B'101011100010110110011000001001011000111110010',B'000001110010010011011011110001100000111001001']),('3e4b:859f:40f1:5822:6461:2c00:c05b:5968/74':::INET,1043:::OID,'-20 years -11 mons -893 days -05:26:55.510518':::INTERVAL,NULL,'0001-01-01 00:00:00':::TIMESTAMP,1114:::OID,false,'\\xd7988667a5c098dc8c':::BYTES,'010500008000000000':::GEOMETRY,ARRAY[B'010010101101111010000000100011010010110110111'])]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 4d40b770f96e8248a937b614c5909289c38c777a:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_135631.781111956_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_135631.781111956_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 7BBAB30EDC1DF366C50B7ACF1C18037EFA09B08B94101020000C004000000D4CBE2596F58F6C19808FF58434BF1413144429A1F35FEC1E8C2AC49DECEEB414C9DBEAF17470042F0603EF023E2FD4194AF7DEBB2E4F4416286341BC866FDC1B6B6A630C348F14180F943306162FE414CE2E809E564DCC1F447F97A7D54EBC194E351543E8C01C2AAD84E62D7C00142C068D06A8E31A2C129697EDFDED4FDC101050000C00900000001020000C005000000BE78A6B887DB004228CEAB1AF77BECC1B0E7462D7CC9CD41821933094462F8416444090B95A0FD410056541102B8BB41DE523CAF238AF64156BDDC1E70B9ECC11828EC2B3E53DA41002C5C018726884168BB26AA8EF3D0C1908EF08E4CEEFB4114406FA990E0EB41CCA372C4C1CDE141006D2498FB84CD419815C141DC25D8C188EAFE6847AED64158A07792E66D0242C855EB8546B2D8C1A26A4692274001C201020000C0030000000C5F047646CCF6418B270267FD9BF0C163018627BD38F8C1E893948FA361FDC18AFC8B60B817F941D0435A632A8DE0C1CA434B4A31FBF8417088DEBAE955DB41A8D57DDE0FF3D8C120CCF2196C9BE041BEB3388997A7E2C1AAFEE6E29783E1C101020000C0060000005226DC1D82E9E3C1B444ADD35B80EFC1D0BD00A08887E341E6994D45BA71E7C14041ABAFEBA5EF417AE14AFCE4E8F8C14E2771D293660142A40AB4FD516B01C2E4F52AEB3766FF41C650816FDF63E0C194059C347A22F8414B061E502718FCC1E00CF16E1804BC41E8EA06620402EC41ECDAAA28452CF341B60EBBC9C343F6C184323F113B39E0C1A86EAB6F21F7FB41461747F376F0F941A480ED419F06F2C1C27538C6A545EFC1040CBDC96829F341B0F8A4299BE5C1C15CFA56AFE377EA4101020000C0050000005E005E91734FF041E0D4A70E6D42EB41003A3A5390C2FB4187E4C7E8AD1EF3C1EC2F5B2DC7FBFC41CA680B7B96C2F6418068281864E2EB41A0C01031A424F7C19CDE77A1E562F7C1A0964BD9BA760042B085257B8CD3CCC1B29EB46ECCC6EDC180DC80783F35EBC15CB8578160C5E64120C9619928F9DEC14C2BEDFA630EF84172A9400866ADFFC1603D2EEAF49BDA41E8579DA158BCE9419A63B6CA6D9AEEC101020000C0040000001479C87D21480242645A8D067221DCC1772784A9928902C2A2EEFEB7C2EEF34170A2BEBB1ABEC8C1585F73B1EC7FE341A08E2A90F6ADFB4102A3F9FAB665F1418836D66DFC6AD241B4CB145E1357FB4180C4679DC81CE641689FD7ED20C5FCC190E78C6D23E8C1C1621B20502E63F8410636D3973A25F141C06AA9EC7285BF4101020000C00600000040C000544AE8AFC126EABF67B031FEC1F00C34CB41FCF3C1207A15AF0153C8C1AE0CDECD4F7C01420213594E7AEBECC1AE03641444D3EBC104748FFD2416F5414C898C5E6D1CFF415CD49E0C53C2FB4141A43C1BB31FF0C18DB81916C46601C21419C8713D2EF541ECDCC25EBDA9FE410EE7DF750377F74130DA5A6CF647CC4140C745D3692EA9C128B7DF12B395F041F821428E00B70142902EFB3E812ACBC1801CEC3757C3C6C1767C7115C64EF741983FF52E7FBBD3C1569B52F0C53A024201020000C002000000DA4767F1FBD2F8416790DBD89049FFC1E0618456A558EC41342ECA1EE02CE84144BA3EAFD35CED41A237589A703E0142187F3F63BBECFBC1E8B3770195C9F54101020000C009000000A270F1DF4D88E7C1B83490BB13D1D741C6E7DFB8CE30E1C14066AD225BE0BE415360BC1DA9EEF2C18036C3F78391A141207286A5362BE541AA99532E46C1F341F04313FA1F7EE541503A173C92CCE94154A461732CF8FB4120E5A162BFFDCE412077EE6FECDDDE416A9FE7DC4E76FBC1B6833664BA8BFAC1CA1BFF4AC392FAC17EE965AC2F1DF5414033CE4AFCDCF6415CCDD78EC6A600C2B01A725EB5E7EE41284139E0BD8EFA41F4CDB77E26E6FB419E0C81FB4F22F4412004E9ECB71DCD4166808343D1AF0042EC1D5577CDE00042B3D57A2B013201C278C2F5FEFBDCF8C1CE2B3E3E432000426AE0C10E05670242D463952D974CFB4164C9C9FA0C7DF241F22A5A4B781AF841384D21993E4BFD417100B52F02D0FEC1B01F68B9CB32FC4101020000C004000000F08DCDD24AD5F441A0C53312953ECFC1D24C2B0D118801C2BAC6EF0066B401420842CA08B472D1C1D4BD27AFDDD0F741A815EC1B9DB8EF41EA06F220CDBBE0C187662E0EE0B4F5C11288DBFD0387F4416011EEEE5216CF4118715CBA670EEB418DE5999B3F5801C26CD7F7AA3A7AE64138213357CD1DD0C184BBDD3616610042':::GEOMETRY,(-1.2139384746551514):::FLOAT8,(-0.1151153952444875):::FLOAT8,701:::OID,e'FW8K\x10D3'::STRING,90002:::OID,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID,1560:::OID,'\x1b98b00028166fbfc1':::BYTES,1560:::OID,'X'::NAME,'\xeef2b081c1':::BYTES,e'\''::CHAR,'b1014627-4bff-4d5d-a6af-3266fbd7510d':::UUID),(NULL,'01010000C038A54D46AF2FE241B06065EE2C4EFF4150D8D01F95B3F6C1320427DB47F300C2':::GEOMETRY,(-0.9925894141197205):::FLOAT8,1.9782155121053844:::FLOAT8,16:::OID,e'zl\x19hy'::STRING,18:::OID,'ccc8b7b8-9c19-4356-84d4-9cc479ebc420':::UUID,90000:::OID,'\xd4':::BYTES,1043:::OID,'*SOl'::NAME:::NAME,'\xb71ca2d28aaeac7e':::BYTES,'0'::CHAR,'6c2e66ef-67ee-4494-970b-cc0cdb12451f':::UUID)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 1DF366C50B7ACF1C18037EFA09B08B94101020000C004000000D4CBE2596F58F6C19808FF58434BF1413144429A1F35FEC1E8C2AC49DECEEB414C9DBEAF17470042F0603EF023E2FD4194AF7DEBB2E4F4416286341BC866FDC1B6B6A630C348F14180F943306162FE414CE2E809E564DCC1F447F97A7D54EBC194E351543E8C01C2AAD84E62D7C00142C068D06A8E31A2C129697EDFDED4FDC101050000C00900000001020000C005000000BE78A6B887DB004228CEAB1AF77BECC1B0E7462D7CC9CD41821933094462F8416444090B95A0FD410056541102B8BB41DE523CAF238AF64156BDDC1E70B9ECC11828EC2B3E53DA41002C5C018726884168BB26AA8EF3D0C1908EF08E4CEEFB4114406FA990E0EB41CCA372C4C1CDE141006D2498FB84CD419815C141DC25D8C188EAFE6847AED64158A07792E66D0242C855EB8546B2D8C1A26A4692274001C201020000C0030000000C5F047646CCF6418B270267FD9BF0C163018627BD38F8C1E893948FA361FDC18AFC8B60B817F941D0435A632A8DE0C1CA434B4A31FBF8417088DEBAE955DB41A8D57DDE0FF3D8C120CCF2196C9BE041BEB3388997A7E2C1AAFEE6E29783E1C101020000C0060000005226DC1D82E9E3C1B444ADD35B80EFC1D0BD00A08887E341E6994D45BA71E7C14041ABAFEBA5EF417AE14AFCE4E8F8C14E2771D293660142A40AB4FD516B01C2E4F52AEB3766FF41C650816FDF63E0C194059C347A22F8414B061E502718FCC1E00CF16E1804BC41E8EA06620402EC41ECDAAA28452CF341B60EBBC9C343F6C184323F113B39E0C1A86EAB6F21F7FB41461747F376F0F941A480ED419F06F2C1C27538C6A545EFC1040CBDC96829F341B0F8A4299BE5C1C15CFA56AFE377EA4101020000C0050000005E005E91734FF041E0D4A70E6D42EB41003A3A5390C2FB4187E4C7E8AD1EF3C1EC2F5B2DC7FBFC41CA680B7B96C2F6418068281864E2EB41A0C01031A424F7C19CDE77A1E562F7C1A0964BD9BA760042B085257B8CD3CCC1B29EB46ECCC6EDC180DC80783F35EBC15CB8578160C5E64120C9619928F9DEC14C2BEDFA630EF84172A9400866ADFFC1603D2EEAF49BDA41E8579DA158BCE9419A63B6CA6D9AEEC101020000C0040000001479C87D21480242645A8D067221DCC1772784A9928902C2A2EEFEB7C2EEF34170A2BEBB1ABEC8C1585F73B1EC7FE341A08E2A90F6ADFB4102A3F9FAB665F1418836D66DFC6AD241B4CB145E1357FB4180C4679DC81CE641689FD7ED20C5FCC190E78C6D23E8C1C1621B20502E63F8410636D3973A25F141C06AA9EC7285BF4101020000C00600000040C000544AE8AFC126EABF67B031FEC1F00C34CB41FCF3C1207A15AF0153C8C1AE0CDECD4F7C01420213594E7AEBECC1AE03641444D3EBC104748FFD2416F5414C898C5E6D1CFF415CD49E0C53C2FB4141A43C1BB31FF0C18DB81916C46601C21419C8713D2EF541ECDCC25EBDA9FE410EE7DF750377F74130DA5A6CF647CC4140C745D3692EA9C128B7DF12B395F041F821428E00B70142902EFB3E812ACBC1801CEC3757C3C6C1767C7115C64EF741983FF52E7FBBD3C1569B52F0C53A024201020000C002000000DA4767F1FBD2F8416790DBD89049FFC1E0618456A558EC41342ECA1EE02CE84144BA3EAFD35CED41A237589A703E0142187F3F63BBECFBC1E8B3770195C9F54101020000C009000000A270F1DF4D88E7C1B83490BB13D1D741C6E7DFB8CE30E1C14066AD225BE0BE415360BC1DA9EEF2C18036C3F78391A141207286A5362BE541AA99532E46C1F341F04313FA1F7EE541503A173C92CCE94154A461732CF8FB4120E5A162BFFDCE412077EE6FECDDDE416A9FE7DC4E76FBC1B6833664BA8BFAC1CA1BFF4AC392FAC17EE965AC2F1DF5414033CE4AFCDCF6415CCDD78EC6A600C2B01A725EB5E7EE41284139E0BD8EFA41F4CDB77E26E6FB419E0C81FB4F22F4412004E9ECB71DCD4166808343D1AF0042EC1D5577CDE00042B3D57A2B013201C278C2F5FEFBDCF8C1CE2B3E3E432000426AE0C10E05670242D463952D974CFB4164C9C9FA0C7DF241F22A5A4B781AF841384D21993E4BFD417100B52F02D0FEC1B01F68B9CB32FC4101020000C004000000F08DCDD24AD5F441A0C53312953ECFC1D24C2B0D118801C2BAC6EF0066B401420842CA08B472D1C1D4BD27AFDDD0F741A815EC1B9DB8EF41EA06F220CDBBE0C187662E0EE0B4F5C11288DBFD0387F4416011EEEE5216CF4118715CBA670EEB418DE5999B3F5801C26CD7F7AA3A7AE64138213357CD1DD0C184BBDD3616610042':::GEOMETRY,(-1.2139384746551514):::FLOAT8,(-0.1151153952444875):::FLOAT8,701:::OID,e'FW8K\\x10D3'::STRING,90002:::OID,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID,1560:::OID,'\\x1b98b00028166fbfc1':::BYTES,1560:::OID,'X'::NAME,'\\xeef2b081c1':::BYTES,e'\\''::CHAR,'b1014627-4bff-4d5d-a6af-3266fbd7510d':::UUID),(NULL,'01010000C038A54D46AF2FE241B06065EE2C4EFF4150D8D01F95B3F6C1320427DB47F300C2':::GEOMETRY,(-0.9925894141197205):::FLOAT8,1.9782155121053844:::FLOAT8,16:::OID,e'zl\\x19hy'::STRING,18:::OID,'ccc8b7b8-9c19-4356-84d4-9cc479ebc420':::UUID,90000:::OID,'\\xd4':::BYTES,1043:::OID,'*SOl'::NAME:::NAME,'\\xb71ca2d28aaeac7e':::BYTES,'0'::CHAR,'6c2e66ef-67ee-4494-970b-cc0cdb12451f':::UUID)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ d687b14e513a49f94e1862c78a7685b78f4fbad9:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_135635.029281902_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_135635.029281902_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | \t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema915.table3617'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema915' AND\n\t\t\t\t\t\t\t\t    table_name='table3617' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema915.table3617', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema915'\n\t\t           AND table_name = 'table3617'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [CREATE TABLE IF NOT EXISTS schema915.table3617 (col3617_3618 FLOAT8 NOT NULL, col3617_3619 REGPROC NULL, col3617_3620 TIMETZ NOT NULL, col3617_3621 REGCLASS, col3617_3622 BIT(32) NOT NULL, col3617_3623 "char" NOT NULL, col3617_3624 TIME NOT NULL, col3617_3625 JSONB NOT NULL, col3617_3626 INTERVAL NULL, col3617_3627 FLOAT4 NOT NULL, col3617_3628 REGROLE, col3617_3629 "char" NULL, col3617_3630 REGROLE NULL, col3617_3631 REGCLASS NULL, col3617_3632 DATE NOT NULL, col3617_3633 NAME NOT NULL, col3617_3634 OID, col3617_3635 STRING NULL AS (CASE WHEN col3617_3631 IS NULL THEN e'\t\nAL)o<M+':::STRING ELSE NULL END) STORED, INDEX (col3617_3624 ASC, col3617_3623, col3617_3620 DESC, col3617_3626, col3617_3631 ASC, col3617_3633, col3617_3621 DESC, col3617_3628, col3617_3618), INDEX (col3617_3635 ASC, col3617_3628 DESC, col3617_3634 ASC, col3617_3630 ASC)) INSERT INTO schema915.table3617 (col3617_3618,col3617_3619,col3617_3620,col3617_3621,col3617_3622,col3617_3623,col3617_3624,col3617_3625,col3617_3626,col3617_3627,col3617_3628,col3617_3629,col3617_3630,col3617_3631,col3617_3632,col3617_3633,col3617_3634) VALUES ((-0.592258260110288):::FLOAT8,18:::OID,'17:36:19.203569-01:40:00':::TIMETZ,1700:::OID,B'01001101111100100010001001010111','p'::"char",'14:59:37.111732':::TIME,'{">&a": true, "b": true, "rzXgT)%": {"?{nEKYyzgPB": [[[]]], "c": {"c": null}, "iiS,CpNLbw": {"n|}uKh": [[]]}, "zU+KHndEj": {"#b7mFW1OlSo": true}}}':::JSONB,'-22 years -8 mons -150 days -17:23:50.367731':::INTERVAL,(-1.1088446378707886):::FLOAT8,1083:::OID,'|'::"char",23:::OID,1186:::OID,'1990-02-21':::DATE,e'\x03P\b3\x0e\'}d'::NAME:::NAME,1561:::OID),((-0.3472614547526389):::FLOAT8,4089:::OID,'24:00:00-15:59:00':::TIMETZ,4089:::OID,B'00010101011011000010101001001000','h'::"char",'09:31:38.284664':::TIME,'[[{"baz": "foo"}, null, {}], [[{"foobar": null}, {}], false, {",u(g~>:3@7C": []}], {"oxc^": []}]':::JSONB,'-56 years -10 mons -439 days -11:29:50.631042':::INTERVAL,(-0.5978038311004639):::FLOAT8,2206:::OID,'K'::"char",16:::OID,90000:::OID,'1990-02-16':::DATE,',*'::NAME:::NAME,1560:::OID),(0.49926461229851116:::FLOAT8,90004:::OID,'22:40:13.869607+04:26:00':::TIMETZ,23:::OID,B'11101110110111111001101010000111',e'\x0f'::"char",'06:50:41.646558':::TIME,'1':::JSONB,'-48 years -5 mons -278 days -04:28:38.84184':::INTERVAL,(-0.2627314329147339):::FLOAT8,20:::OID,','::"char",1083:::OID,2205:::OID,'1983-01-09':::DATE,''::NAME:::NAME,1083:::OID)]: ERROR: relation "table3617" (873): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | = 'schema915.table3617'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema915' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table3617' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema915.table3617', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema915'\\n\\t\\t           AND table_name = 'table3617'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [CREATE TABLE IF NOT EXISTS schema915.table3617 (col3617_3618 FLOAT8 NOT NULL, col3617_3619 REGPROC NULL, col3617_3620 TIMETZ NOT NULL, col3617_3621 REGCLASS, col3617_3622 BIT(32) NOT NULL, col3617_3623 \"char\" NOT NULL, col3617_3624 TIME NOT NULL, col3617_3625 JSONB NOT NULL, col3617_3626 INTERVAL NULL, col3617_3627 FLOAT4 NOT NULL, col3617_3628 REGROLE, col3617_3629 \"char\" NULL, col3617_3630 REGROLE NULL, col3617_3631 REGCLASS NULL, col3617_3632 DATE NOT NULL, col3617_3633 NAME NOT NULL, col3617_3634 OID, col3617_3635 STRING NULL AS (CASE WHEN col3617_3631 IS NULL THEN e'\\t\\nAL)o\u003cM+':::STRING ELSE NULL END) STORED, INDEX (col3617_3624 ASC, col3617_3623, col3617_3620 DESC, col3617_3626, col3617_3631 ASC, col3617_3633, col3617_3621 DESC, col3617_3628, col3617_3618), INDEX (col3617_3635 ASC, col3617_3628 DESC, col3617_3634 ASC, col3617_3630 ASC)) INSERT INTO schema915.table3617 (col3617_3618,col3617_3619,col3617_3620,col3617_3621,col3617_3622,col3617_3623,col3617_3624,col3617_3625,col3617_3626,col3617_3627,col3617_3628,col3617_3629,col3617_3630,col3617_3631,col3617_3632,col3617_3633,col3617_3634) VALUES ((-0.592258260110288):::FLOAT8,18:::OID,'17:36:19.203569-01:40:00':::TIMETZ,1700:::OID,B'01001101111100100010001001010111','p'::\"char\",'14:59:37.111732':::TIME,'{\"\u003e\u0026a\": true, \"b\": true, \"rzXgT)%\": {\"?{nEKYyzgPB\": [[[]]], \"c\": {\"c\": null}, \"iiS,CpNLbw\": {\"n|}uKh\": [[]]}, \"zU+KHndEj\": {\"#b7mFW1OlSo\": true}}}':::JSONB,'-22 years -8 mons -150 days -17:23:50.367731':::INTERVAL,(-1.1088446378707886):::FLOAT8,1083:::OID,'|'::\"char\",23:::OID,1186:::OID,'1990-02-21':::DATE,e'\\x03P\\b3\\x0e\\'}d'::NAME:::NAME,1561:::OID),((-0.3472614547526389):::FLOAT8,4089:::OID,'24:00:00-15:59:00':::TIMETZ,4089:::OID,B'00010101011011000010101001001000','h'::\"char\",'09:31:38.284664':::TIME,'[[{\"baz\": \"foo\"}, null, {}], [[{\"foobar\": null}, {}], false, {\",u(g~\u003e:3@7C\": []}], {\"oxc^\": []}]':::JSONB,'-56 years -10 mons -439 days -11:29:50.631042':::INTERVAL,(-0.5978038311004639):::FLOAT8,2206:::OID,'K'::\"char\",16:::OID,90000:::OID,'1990-02-16':::DATE,',*'::NAME:::NAME,1560:::OID),(0.49926461229851116:::FLOAT8,90004:::OID,'22:40:13.869607+04:26:00':::TIMETZ,23:::OID,B'11101110110111111001101010000111',e'\\x0f'::\"char\",'06:50:41.646558':::TIME,'1':::JSONB,'-48 years -5 mons -278 days -04:28:38.84184':::INTERVAL,(-0.2627314329147339):::FLOAT8,20:::OID,','::\"char\",1083:::OID,2205:::OID,'1983-01-09':::DATE,''::NAME:::NAME,1083:::OID)]: ERROR: relation \"table3617\" (873): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

fqazi commented 2 years ago

Currently observed failure are coming via: https://github.com/cockroachdb/cockroach/issues/85677

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ db95c2484a9f15c2de01466af0aae19c8ab57618:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_135157.569506369_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_135157.569506369_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | VALUES(1270:::OID,(-0.36704101994802585):::FLOAT8) ) AS T(col1026_1029,col1026_1037)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM schema624.table1026 WHERE (CASE WHEN col1026_1029 IS NULL THEN 'WQ527':::STRING ELSE e')?DSQ#s\\n':::STRING END, col1026_1029, col1026_1037)= ( SELECT  (CASE WHEN col1026_1029 IS NULL THEN 'WQ527':::STRING ELSE e')?DSQ#s\\n':::STRING END, col1026_1029, col1026_1037) FROM (VALUES( 1270:::OID,(-0.36704101994802585):::FLOAT8) ) AS T(col1026_1029,col1026_1037) ) )" []] :false
          | QUERY ["SELECT array[((CASE WHEN col1026_1029 IS NULL THEN 'WQ527':::STRING ELSE e')?DSQ#s\\n':::STRING END, col1026_1029, col1026_1037))::STRING] FROM (VALUES(1270:::OID,(-0.36704101994802585):::FLOAT8) ) AS T(col1026_1029,col1026_1037)" []] :[["(\")?DSQ#s\n\",1270,-0.36704101994802585)"]]
          | QUERY ["SELECT num_nulls(col1026_1035) > 0 FROM (VALUES(5999406383108772273:::INT8) ) AS T(col1026_1035)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM schema624.table1026 WHERE (col1026_1035)= ( SELECT  (col1026_1035) FROM (VALUES( 5999406383108772273:::INT8) ) AS T(col1026_1035) ) )" []] :false
          | QUERY ["SELECT array[((col1026_1035))::STRING] FROM (VALUES(5999406383108772273:::INT8) ) AS T(col1026_1035)" []] :[["5999406383108772273"]]
          | QUERY ["SELECT num_nulls(col1026_1031) > 0 FROM (VALUES((-0.18279770016670227):::FLOAT8) ) AS T(col1026_1031)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM schema624.table1026 WHERE (col1026_1031)= ( SELECT  (col1026_1031) FROM (VALUES( (-0.18279770016670227):::FLOAT8) ) AS T(col1026_1031) ) )" []] :false
          | QUERY ["SELECT array[((col1026_1031))::STRING] FROM (VALUES((-0.18279770016670227):::FLOAT8) ) AS T(col1026_1031)" []] :[["-0.18279770016670227"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema624.table1026'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema624' AND\n\t\t\t\t\t\t\t\t    table_name='table1026' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema624.table1026', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema624'\n\t\t           AND table_name = 'table1026'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema624.seq1067 INSERT INTO schema624.table1026 (col1026_1027,col1026_1028,col1026_1029,col1026_1030,col1026_1031,col1026_1032,col1026_1033,col1026_1034,col1026_1035,col1026_1036,col1026_1037,col1026_1038) VALUES (1700:::OID,'1970-12-06':::DATE,1270:::OID,1002:::OID,(-0.18279770016670227):::FLOAT8,2206:::OID,e'9c\x1cq-'::STRING,e'{"V80caU^GG": null, "baz": {"foobar": "``)W~&t", "zT\\\\cn": [[null, {"a": []}, []], 1.0719643222149688, {"c": null}, {}]}}':::JSONB,5999406383108772273:::INT8,90002:::OID,(-0.36704101994802585):::FLOAT8,(-1.094529151916504):::FLOAT8)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | L THEN 'WQ527':::STRING ELSE e')?DSQ#s\\\\n':::STRING END, col1026_1029, col1026_1037)= ( SELECT  (CASE WHEN col1026_1029 IS NULL THEN 'WQ527':::STRING ELSE e')?DSQ#s\\\\n':::STRING END, col1026_1029, col1026_1037) FROM (VALUES( 1270:::OID,(-0.36704101994802585):::FLOAT8) ) AS T(col1026_1029,col1026_1037) ) )\" []] :false\nQUERY [\"SELECT array[((CASE WHEN col1026_1029 IS NULL THEN 'WQ527':::STRING ELSE e')?DSQ#s\\\\n':::STRING END, col1026_1029, col1026_1037))::STRING] FROM (VALUES(1270:::OID,(-0.36704101994802585):::FLOAT8) ) AS T(col1026_1029,col1026_1037)\" []] :[[\"(\\\")?DSQ#s\\n\\\",1270,-0.36704101994802585)\"]]\nQUERY [\"SELECT num_nulls(col1026_1035) \u003e 0 FROM (VALUES(5999406383108772273:::INT8) ) AS T(col1026_1035)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema624.table1026 WHERE (col1026_1035)= ( SELECT  (col1026_1035) FROM (VALUES( 5999406383108772273:::INT8) ) AS T(col1026_1035) ) )\" []] :false\nQUERY [\"SELECT array[((col1026_1035))::STRING] FROM (VALUES(5999406383108772273:::INT8) ) AS T(col1026_1035)\" []] :[[\"5999406383108772273\"]]\nQUERY [\"SELECT num_nulls(col1026_1031) \u003e 0 FROM (VALUES((-0.18279770016670227):::FLOAT8) ) AS T(col1026_1031)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema624.table1026 WHERE (col1026_1031)= ( SELECT  (col1026_1031) FROM (VALUES( (-0.18279770016670227):::FLOAT8) ) AS T(col1026_1031) ) )\" []] :false\nQUERY [\"SELECT array[((col1026_1031))::STRING] FROM (VALUES((-0.18279770016670227):::FLOAT8) ) AS T(col1026_1031)\" []] :[[\"-0.18279770016670227\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema624.table1026'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema624' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table1026' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema624.table1026', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema624'\\n\\t\\t           AND table_name = 'table1026'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema624.seq1067 INSERT INTO schema624.table1026 (col1026_1027,col1026_1028,col1026_1029,col1026_1030,col1026_1031,col1026_1032,col1026_1033,col1026_1034,col1026_1035,col1026_1036,col1026_1037,col1026_1038) VALUES (1700:::OID,'1970-12-06':::DATE,1270:::OID,1002:::OID,(-0.18279770016670227):::FLOAT8,2206:::OID,e'9c\\x1cq-'::STRING,e'{\"V80caU^GG\": null, \"baz\": {\"foobar\": \"``)W~\u0026t\", \"zT\\\\\\\\cn\": [[null, {\"a\": []}, []], 1.0719643222149688, {\"c\": null}, {}]}}':::JSONB,5999406383108772273:::INT8,90002:::OID,(-0.36704101994802585):::FLOAT8,(-1.094529151916504):::FLOAT8)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ a7c91f06d8ee0fa2096bcd626f689009024947bb:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_140042.067553428_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_140042.067553428_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | :::OID,'45 years 7 mons 376 days 18:05:26.236078':::INTERVAL,'0105000020E6100000010000000102000000030000004067974D3FC70340EE7667E326ED48C0C42E4FD23422664070E660A45EE752C0904C12EB29375440006EC5576B934D40':::GEOGRAPHY,'01:52:08.446114-01:04:00':::TIMETZ)) AS t(col4007_4021,col4007_4009,col4007_4011,col4007_4012,col4007_4014,col4007_4016,col4007_4018,col4007_4010,col4007_4013,col4007_4015,col4007_4017,col4007_4019,col4007_4020);" []] :[["-0.3865167105051803"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table4007'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table4007' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table4007', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table4007'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [CREATE TABLE IF NOT EXISTS public.table4007 (col4007_4009 FLOAT8 NOT NULL, col4007_4010 FLOAT4 NOT NULL, col4007_4011 JSONB NOT NULL, col4007_4012 JSONB NOT NULL, col4007_4013 REGPROCEDURE, col4007_4014 JSONB, col4007_4015 REGCLASS, col4007_4016 INT8[] NULL, col4007_4017 INTERVAL NOT NULL, col4007_4018 REGROLE NULL, col4007_4019 GEOGRAPHY NOT NULL, col4007_4020 TIMETZ NOT NULL, col4007_4021 BIT(26) NOT NULL, col4007_4022 STRING NOT NULL AS (lower(CAST(col4007_4012 AS STRING))) STORED, col4007_4023 FLOAT8 NOT NULL AS (col4007_4010 + col4007_4009) VIRTUAL, col4007_4024 STRING NULL AS (CASE WHEN col4007_4016 IS NULL THEN e'\b%':::STRING ELSE e't\nV\x11':::STRING END) VIRTUAL, col4007_4025 FLOAT8 NOT NULL AS (col4007_4010 + col4007_4009) VIRTUAL, col4007_4026 STRING NOT NULL AS (lower(CAST(col4007_4021 AS STRING))) VIRTUAL, col4007_4027 FLOAT8 NOT NULL AS (col4007_4009 + (-0.5098495099668572):::FLOAT8) STORED, FAMILY (col4007_4012, col4007_4014, col4007_4009), FAMILY (col4007_4017, col4007_4019), FAMILY (col4007_4020, col4007_4022, col4007_4011, col4007_4016), FAMILY (col4007_4027), FAMILY (col4007_4013), FAMILY (col4007_4018), FAMILY (col4007_4021), FAMILY (col4007_4010, col4007_4015)) INSERT INTO public.table4007 (col4007_4009,col4007_4010,col4007_4011,col4007_4012,col4007_4013,col4007_4014,col4007_4015,col4007_4016,col4007_4017,col4007_4018,col4007_4019,col4007_4020,col4007_4021) VALUES (0.12333279946167686:::FLOAT8,(-0.8592272996902466):::FLOAT8,'[{"#:ZjB.sv``%LE": true}, false, "a"]':::JSONB,'1':::JSONB,25:::OID,'false':::JSONB,2205:::OID,ARRAY[740631440568000491:::INT8],'45 years 7 mons 376 days 18:05:26.236078':::INTERVAL,24:::OID,'0105000020E6100000010000000102000000030000004067974D3FC70340EE7667E326ED48C0C42E4FD23422664070E660A45EE752C0904C12EB29375440006EC5576B934D40':::GEOGRAPHY,'01:52:08.446114-01:04:00':::TIMETZ,B'00100110100000011101001110')]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 326ED48C0C42E4FD23422664070E660A45EE752C0904C12EB29375440006EC5576B934D40':::GEOGRAPHY,'01:52:08.446114-01:04:00':::TIMETZ)) AS t(col4007_4021,col4007_4009,col4007_4011,col4007_4012,col4007_4014,col4007_4016,col4007_4018,col4007_4010,col4007_4013,col4007_4015,col4007_4017,col4007_4019,col4007_4020);\" []] :[[\"-0.3865167105051803\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table4007'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4007' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table4007', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table4007'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [CREATE TABLE IF NOT EXISTS public.table4007 (col4007_4009 FLOAT8 NOT NULL, col4007_4010 FLOAT4 NOT NULL, col4007_4011 JSONB NOT NULL, col4007_4012 JSONB NOT NULL, col4007_4013 REGPROCEDURE, col4007_4014 JSONB, col4007_4015 REGCLASS, col4007_4016 INT8[] NULL, col4007_4017 INTERVAL NOT NULL, col4007_4018 REGROLE NULL, col4007_4019 GEOGRAPHY NOT NULL, col4007_4020 TIMETZ NOT NULL, col4007_4021 BIT(26) NOT NULL, col4007_4022 STRING NOT NULL AS (lower(CAST(col4007_4012 AS STRING))) STORED, col4007_4023 FLOAT8 NOT NULL AS (col4007_4010 + col4007_4009) VIRTUAL, col4007_4024 STRING NULL AS (CASE WHEN col4007_4016 IS NULL THEN e'\\b%':::STRING ELSE e't\\nV\\x11':::STRING END) VIRTUAL, col4007_4025 FLOAT8 NOT NULL AS (col4007_4010 + col4007_4009) VIRTUAL, col4007_4026 STRING NOT NULL AS (lower(CAST(col4007_4021 AS STRING))) VIRTUAL, col4007_4027 FLOAT8 NOT NULL AS (col4007_4009 + (-0.5098495099668572):::FLOAT8) STORED, FAMILY (col4007_4012, col4007_4014, col4007_4009), FAMILY (col4007_4017, col4007_4019), FAMILY (col4007_4020, col4007_4022, col4007_4011, col4007_4016), FAMILY (col4007_4027), FAMILY (col4007_4013), FAMILY (col4007_4018), FAMILY (col4007_4021), FAMILY (col4007_4010, col4007_4015)) INSERT INTO public.table4007 (col4007_4009,col4007_4010,col4007_4011,col4007_4012,col4007_4013,col4007_4014,col4007_4015,col4007_4016,col4007_4017,col4007_4018,col4007_4019,col4007_4020,col4007_4021) VALUES (0.12333279946167686:::FLOAT8,(-0.8592272996902466):::FLOAT8,'[{\"#:ZjB.sv``%LE\": true}, false, \"a\"]':::JSONB,'1':::JSONB,25:::OID,'false':::JSONB,2205:::OID,ARRAY[740631440568000491:::INT8],'45 years 7 mons 376 days 18:05:26.236078':::INTERVAL,24:::OID,'0105000020E6100000010000000102000000030000004067974D3FC70340EE7667E326ED48C0C42E4FD23422664070E660A45EE752C0904C12EB29375440006EC5576B934D40':::GEOGRAPHY,'01:52:08.446114-01:04:00':::TIMETZ,B'00100110100000011101001110')]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ ce8bfd4811e789bdd4cf31640722517a5cf46237:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_134546.076005004_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_134546.076005004_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | _2747);" []] :[[""]]
          | QUERY ["SELECT array[(lower(col2741_2743))::STRING::STRING] AS c FROM ( VALUES('-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,(-6754360917962809529):::INT8,1184:::OID,90004:::OID,'02:35:49.145328+02:27:00':::TIMETZ,''::VARCHAR,B'01101011')) AS t(col2741_2745,col2741_2746,col2741_2747,col2741_2748,col2741_2742,col2741_2743,col2741_2744);" []] :[[""]]
          | QUERY ["SELECT num_nulls(col2741_2752, col2741_2751, col2741_2744, col2741_2749, col2741_2745, col2741_2747) > 0 FROM (VALUES(B'01101011','02:35:49.145328+02:27:00':::STRING,'-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,1184:::OID,'':::STRING,'':::STRING) ) AS T(col2741_2744,col2741_2749,col2741_2745,col2741_2747,col2741_2752,col2741_2751)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM schema1918.table2741 WHERE (col2741_2752, col2741_2751, col2741_2744, col2741_2749, col2741_2745, col2741_2747)= ( SELECT  (col2741_2752, col2741_2751, col2741_2744, col2741_2749, col2741_2745, col2741_2747) FROM (VALUES( B'01101011','02:35:49.145328+02:27:00':::STRING,'-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,1184:::OID,'':::STRING,'':::STRING) ) AS T(col2741_2744,col2741_2749,col2741_2745,col2741_2747,col2741_2752,col2741_2751) ) )" []] :false
          | QUERY ["SELECT array[((col2741_2752, col2741_2751, col2741_2744, col2741_2749, col2741_2745, col2741_2747))::STRING] FROM (VALUES(B'01101011','02:35:49.145328+02:27:00':::STRING,'-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,1184:::OID,'':::STRING,'':::STRING) ) AS T(col2741_2744,col2741_2749,col2741_2745,col2741_2747,col2741_2752,col2741_2751)" []] :[["(\"\",\"\",01101011,02:35:49.145328+02:27:00,\"-75 years -1 mons -35 days -12:01:10.99099\",1184)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema1918.table2741'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema1918' AND\n\t\t\t\t\t\t\t\t    table_name='table2741' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema1918.table2741', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema1918'\n\t\t           AND table_name = 'table2741'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() DROP TABLE IF EXISTS schema1164.table2726 CASCADE INSERT INTO schema1918.table2741 (col2741_2742,col2741_2743,col2741_2744,col2741_2745,col2741_2746,col2741_2747,col2741_2748) VALUES ('02:32:46.51778+10:09:00':::TIMETZ,'DB'::VARCHAR,B'11101011','24 years 2 mons 975 days 02:31:13.363446':::INTERVAL,5357274576444009157:::INT8,2206:::OID,4096:::OID),('02:35:49.145328+02:27:00':::TIMETZ,''::VARCHAR,B'01101011','-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,(-6754360917962809529):::INT8,1184:::OID,90004:::OID)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ,(-6754360917962809529):::INT8,1184:::OID,90004:::OID,'02:35:49.145328+02:27:00':::TIMETZ,''::VARCHAR,B'01101011')) AS t(col2741_2745,col2741_2746,col2741_2747,col2741_2748,col2741_2742,col2741_2743,col2741_2744);\" []] :[[\"\"]]\nQUERY [\"SELECT num_nulls(col2741_2752, col2741_2751, col2741_2744, col2741_2749, col2741_2745, col2741_2747) \u003e 0 FROM (VALUES(B'01101011','02:35:49.145328+02:27:00':::STRING,'-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,1184:::OID,'':::STRING,'':::STRING) ) AS T(col2741_2744,col2741_2749,col2741_2745,col2741_2747,col2741_2752,col2741_2751)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema1918.table2741 WHERE (col2741_2752, col2741_2751, col2741_2744, col2741_2749, col2741_2745, col2741_2747)= ( SELECT  (col2741_2752, col2741_2751, col2741_2744, col2741_2749, col2741_2745, col2741_2747) FROM (VALUES( B'01101011','02:35:49.145328+02:27:00':::STRING,'-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,1184:::OID,'':::STRING,'':::STRING) ) AS T(col2741_2744,col2741_2749,col2741_2745,col2741_2747,col2741_2752,col2741_2751) ) )\" []] :false\nQUERY [\"SELECT array[((col2741_2752, col2741_2751, col2741_2744, col2741_2749, col2741_2745, col2741_2747))::STRING] FROM (VALUES(B'01101011','02:35:49.145328+02:27:00':::STRING,'-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,1184:::OID,'':::STRING,'':::STRING) ) AS T(col2741_2744,col2741_2749,col2741_2745,col2741_2747,col2741_2752,col2741_2751)\" []] :[[\"(\\\"\\\",\\\"\\\",01101011,02:35:49.145328+02:27:00,\\\"-75 years -1 mons -35 days -12:01:10.99099\\\",1184)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema1918.table2741'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema1918' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table2741' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema1918.table2741', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema1918'\\n\\t\\t           AND table_name = 'table2741'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() DROP TABLE IF EXISTS schema1164.table2726 CASCADE INSERT INTO schema1918.table2741 (col2741_2742,col2741_2743,col2741_2744,col2741_2745,col2741_2746,col2741_2747,col2741_2748) VALUES ('02:32:46.51778+10:09:00':::TIMETZ,'DB'::VARCHAR,B'11101011','24 years 2 mons 975 days 02:31:13.363446':::INTERVAL,5357274576444009157:::INT8,2206:::OID,4096:::OID),('02:35:49.145328+02:27:00':::TIMETZ,''::VARCHAR,B'01101011','-75 years -1 mons -35 days -12:01:10.99099':::INTERVAL,(-6754360917962809529):::INT8,1184:::OID,90004:::OID)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 9a0a233f4021f25c1766434bb6aee85536e32036:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133856.842784509_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_133856.842784509_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | '\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [INSERT INTO public.table71 (col15_33,col15_30,col15_36,col15_32,col15_31,col15_34) VALUES (2205:::OID,'01060000000700000001030000000100000007000000E43F3E0ACFE4F9C180A59022E57FC2C158EA66884D1BDDC19ED34D4B7ACA00C29CCEECD2A6560142A0CDE469F28700C2E0BD71A874AEFD4170742505621CE4C1DC11F6B1D863EAC1F41D7EBCA8920242925A795AB20EF9C198C801F9F604FC41E43F3E0ACFE4F9C180A59022E57FC2C10103000000010000000400000014A6BD0B75A8FAC1F96BDC1C1335F5C12AE89EA4E9F3F14164027AEFFC31FF41CE4CA2EBEC46F6C154ED25BF5E29004214A6BD0B75A8FAC1F96BDC1C1335F5C101030000000100000006000000CCA1393E3289F5C170FE7DFF898AC3C1BF5CC6A5A466F7C198DD28D24C88D8C156E7BEBE0CEDF14111E9236EF793F2C10002E511FD477541BC588322B336F441C01A119C7DBDB7C18C9E65445C400042CCA1393E3289F5C170FE7DFF898AC3C10103000000010000000A000000FE4460D71739FDC1700C4D08C85BC1413C4C5314538701C232CE9A8252B4F7C1C8E756F41EFFD841C011BF40854FBBC1CEFCFF2075170142600B4D2D0150BAC1563E5F82886301421CABFE730DB9F3419437B0A88E5D0142E84E8F22A8B500422298B43A742AF94154543649DFC7FA41A01259EB6FDFF741C8D3473B521FFD41E8C4040F40F8E6C184D19BCE0FE1FC41FE4460D71739FDC1700C4D08C85BC14101030000000100000008000000C4D86860CD65D6C10039F95830A2AC41F42D8D0DF389E24119A507C3DDDAF9C1A4D9AC6A9C8D00421CA8237364ECFBC13866C5B11A05E4412C8291764264014298E51F5104BCD241E243157DE8C1F041800F6D0CD989B3C1781EA1E5B11CE941C06B5EA74713CDC110F40B3D52E6ED41C4D86860CD65D6C10039F95830A2AC4101030000000100000009000000164E0228C3D8F2C1A0BD9A75EBE1CC41247AC0F50B44F04137F110AAA29F01C228F9F8CB9D53EF41A90330E3841AFDC1C4676A60A940014220904C576859B1413026E9780ACFE2418077E4CAF58C0242281DA22402F9D14178AA725C44E5FE41201C01C50975C8C1CE940AD00AADF84140A3CB5972BDD2C110A0014857CAE841164E0228C3D8F2C1A0BD9A75EBE1CC410103000000010000000B0000001FBD4AD8DF2AF7C19822BCF471B7E341BBDB037077EAF0C1803D66B725C7BC416016280DDD6ACBC10037BC8726D2874126FD51BED947E1C15A9CA11ADD84E3C100D96EFA73560242B0CE3E21DB39F0C13EA97AAB520AF6416EBCFB6EAD6FF341F40E18D1617AF24178BC82D98803F44148FA574DE5A5F041309E734CA270F64172B3767B6CBBF64194BC0E3AB5170142E82ABB5AB895E8416E66178AA78102421FBD4AD8DF2AF7C19822BCF471B7E341':::GEOMETRY,'sfk |'::STRING,'161.182.116.77/1':::INET,e'\f'::CHAR,23:::OID),(1560:::OID,'0101000080000000000000F87F000000000000F87F000000000000F87F':::GEOMETRY,e'@\x12&'::STRING,'11.189.154.228/17':::INET,e'\x16'::CHAR,1560:::OID),(21:::OID,'010500008004000000010200008003000000D82A54DF7419E241743454148969E141D4FDB1BDEF58F9C160DC8E193803EE41CA9BE983812DF24113690239F5CF00C2663DBC76DF61E9C1F0EA3FCF6F26F341960E4B145E6E024201020000800300000058BDFB41B8FEFF41327682C4FF36F1C1A08E7F83852CCDC16E7771942EE2F041509C7251B208E3C170F8765169CFFF41FA46F4B0A40CF841461556ACB0A6F0418651E00FC632FDC1010200008003000000FF5847499757F4C14EA0126D5299F4C1CBEE30082490F2C148522538A856D74182325197D95FFCC190FDEBCABCC8DAC128061EEA7848DE4102258BFABC11F2C156453AA52656004201020000800200000040CCA2A072F8A9C18022C257759A9441705B65B34B4AC241FC1C3D6FC0F5DDC1B02A9649CC83D4C1B62B9250C35D00C2':::GEOMETRY,e'\x114'::STRING,'c2e5:ec84:f11e:3fb:72f3:56ce:fdb5:abe9/35':::INET,'2'::CHAR,1082:::OID) INSERT INTO public.table71 (col15_33,col15_30,col15_36,col15_32,col15_31,col15_34) VALUES (701:::OID,'010300008000000000':::GEOMETRY,e'K7\x1dD'::STRING,'108.13.96.199/8':::INET,e'\x12'::CHAR,1083:::OID)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ion\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO public.table71 (col15_33,col15_30,col15_36,col15_32,col15_31,col15_34) VALUES (2205:::OID,'01060000000700000001030000000100000007000000E43F3E0ACFE4F9C180A59022E57FC2C158EA66884D1BDDC19ED34D4B7ACA00C29CCEECD2A6560142A0CDE469F28700C2E0BD71A874AEFD4170742505621CE4C1DC11F6B1D863EAC1F41D7EBCA8920242925A795AB20EF9C198C801F9F604FC41E43F3E0ACFE4F9C180A59022E57FC2C10103000000010000000400000014A6BD0B75A8FAC1F96BDC1C1335F5C12AE89EA4E9F3F14164027AEFFC31FF41CE4CA2EBEC46F6C154ED25BF5E29004214A6BD0B75A8FAC1F96BDC1C1335F5C101030000000100000006000000CCA1393E3289F5C170FE7DFF898AC3C1BF5CC6A5A466F7C198DD28D24C88D8C156E7BEBE0CEDF14111E9236EF793F2C10002E511FD477541BC588322B336F441C01A119C7DBDB7C18C9E65445C400042CCA1393E3289F5C170FE7DFF898AC3C10103000000010000000A000000FE4460D71739FDC1700C4D08C85BC1413C4C5314538701C232CE9A8252B4F7C1C8E756F41EFFD841C011BF40854FBBC1CEFCFF2075170142600B4D2D0150BAC1563E5F82886301421CABFE730DB9F3419437B0A88E5D0142E84E8F22A8B500422298B43A742AF94154543649DFC7FA41A01259EB6FDFF741C8D3473B521FFD41E8C4040F40F8E6C184D19BCE0FE1FC41FE4460D71739FDC1700C4D08C85BC14101030000000100000008000000C4D86860CD65D6C10039F95830A2AC41F42D8D0DF389E24119A507C3DDDAF9C1A4D9AC6A9C8D00421CA8237364ECFBC13866C5B11A05E4412C8291764264014298E51F5104BCD241E243157DE8C1F041800F6D0CD989B3C1781EA1E5B11CE941C06B5EA74713CDC110F40B3D52E6ED41C4D86860CD65D6C10039F95830A2AC4101030000000100000009000000164E0228C3D8F2C1A0BD9A75EBE1CC41247AC0F50B44F04137F110AAA29F01C228F9F8CB9D53EF41A90330E3841AFDC1C4676A60A940014220904C576859B1413026E9780ACFE2418077E4CAF58C0242281DA22402F9D14178AA725C44E5FE41201C01C50975C8C1CE940AD00AADF84140A3CB5972BDD2C110A0014857CAE841164E0228C3D8F2C1A0BD9A75EBE1CC410103000000010000000B0000001FBD4AD8DF2AF7C19822BCF471B7E341BBDB037077EAF0C1803D66B725C7BC416016280DDD6ACBC10037BC8726D2874126FD51BED947E1C15A9CA11ADD84E3C100D96EFA73560242B0CE3E21DB39F0C13EA97AAB520AF6416EBCFB6EAD6FF341F40E18D1617AF24178BC82D98803F44148FA574DE5A5F041309E734CA270F64172B3767B6CBBF64194BC0E3AB5170142E82ABB5AB895E8416E66178AA78102421FBD4AD8DF2AF7C19822BCF471B7E341':::GEOMETRY,'sfk |'::STRING,'161.182.116.77/1':::INET,e'\\f'::CHAR,23:::OID),(1560:::OID,'0101000080000000000000F87F000000000000F87F000000000000F87F':::GEOMETRY,e'@\\x12\u0026'::STRING,'11.189.154.228/17':::INET,e'\\x16'::CHAR,1560:::OID),(21:::OID,'010500008004000000010200008003000000D82A54DF7419E241743454148969E141D4FDB1BDEF58F9C160DC8E193803EE41CA9BE983812DF24113690239F5CF00C2663DBC76DF61E9C1F0EA3FCF6F26F341960E4B145E6E024201020000800300000058BDFB41B8FEFF41327682C4FF36F1C1A08E7F83852CCDC16E7771942EE2F041509C7251B208E3C170F8765169CFFF41FA46F4B0A40CF841461556ACB0A6F0418651E00FC632FDC1010200008003000000FF5847499757F4C14EA0126D5299F4C1CBEE30082490F2C148522538A856D74182325197D95FFCC190FDEBCABCC8DAC128061EEA7848DE4102258BFABC11F2C156453AA52656004201020000800200000040CCA2A072F8A9C18022C257759A9441705B65B34B4AC241FC1C3D6FC0F5DDC1B02A9649CC83D4C1B62B9250C35D00C2':::GEOMETRY,e'\\x114'::STRING,'c2e5:ec84:f11e:3fb:72f3:56ce:fdb5:abe9/35':::INET,'2'::CHAR,1082:::OID) INSERT INTO public.table71 (col15_33,col15_30,col15_36,col15_32,col15_31,col15_34) VALUES (701:::OID,'010300008000000000':::GEOMETRY,e'K7\\x1dD'::STRING,'108.13.96.199/8':::INET,e'\\x12'::CHAR,1083:::OID)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ f53c21f9fe0fe07ba32d2f28ac4bd2c4bc2ed50b:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133659.698033856_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_133659.698033856_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 1422037C30F74EABEC1B47CDFC14F31004201020000C004000000047C31C3A12DE341E9C0792AE6E1F0C1BE5D9DB5FFDDF44110CE3F0B10BCD24154C3399AE46EFB4122FE34A94A5FFBC16A827C04AE2AE3C1E70376043A6FFBC1AC171669CC74F3412CE4BFF108DAE841CC1715177EAAE3416345C35D8116F1C1CCA8BBEA2DF3E5C1986BA9B35C6ADC418211A90107CFE4C1C481DECB38A2FB4101020000C003000000C00B9D03E082EE41D04B3BB9883CC841C60BA4AD67860142E016D0444DBEFBC15CB8FBCA89D9004280B92CB0E44EDD41A894C9A740AFECC18057DC75E78491416C3A6BB6FC51E141341F9AF8F0D0FD4188054E577960D141C03B7E306A15F94101020000C0060000004CA63BD8BB2DE0415E68EB85AC23E4C12CC0FCF35FA3E0C1491E933B553601C2849AB3B56880F3419D5A8C5D42C2F3C1E4AF32324D5FFD410480A5DE657DEF4110FF73A2145700421EB79589F876F3C1266044D25912F4C1680CA939CBB1FFC1C80F223DF456F841788B7F5B7A92D641BE71EF7F46C10042B87DB12BB5BDFEC166FF30631044004264182AB1D03301421464173D76C8FE41A09EC00EA331C0C1F88E633209B5F9C1C0109A94E9B8C7414BCDEDD33445F1C1828056E13537F64101020000C002000000761AA09A1162F641E0F16A8368C0B741EF0D8800DB23F3C102EB06710DDEFFC1E07CBF99746EFC417049FA53072EEB41164AF5A4078A00C2C4F9998A25BCFA4101010000C0EC58A6FE6C74EFC178DAAAFAC50DFD41128962C47B3BF94180E97A4C5DD2EDC101050000C00200000001020000C0060000009B7162DF7496FAC1A6772DE6356EFDC11990480400E6F7C1C0F30E69ED7AB3C154B38E0E93F7E3C172CDCB13741CFCC13440FC17BD2AF3C19ED6437FE7B2FAC1C072F6D5B611AEC1A2AF6248EE1700C26CB57AF5D66DE0C1EACCC29AFDAEF0C180E55F95E7FF9241195A06C66DD5F5C11C35F0E1378201C29C0E7AA75FAEE041809E1D601004EE415E1DDA928DEFEDC10E8411911519F8C1BCB64091D3EBF3C190B2896E2AC4C54184728649BD0B0142083D8B47FB0CD441002F01967CE801C201020000C003000000981C2A178CE1FC41C127D7B05A7001C2342EDC28FC85D6C1DC440FBA2549E6C14C6C80026122FB41F4D3BE883FB7DBC1A8C05EDF51A6DC41548EA382CEBEEA418A73CB8CC41A00424090F33A69C5D741901F20284BB4FE411D98F81521D501C201060000C00100000001030000C0010000000700000010932249F0BEF0C1A87A8CE9D243F8C15B5114E37792F7C11275A39F3B6A01C2E66077E54DF0E9C1AED89A07839FFAC14A3F2B6B30530042E880B860D4A4E9410AC3CDC18B92F841B3C49510B568F8C10AB1157611B5F9C19233A08B5C240242822D73D80D74F44180E809303F09A5C1CEC803876A17FAC14C8D51A1C2C3EDC1F6D90444945CFA4194617185E0ABFB410ACE17F735E0EBC1E0EDF26B1C48F741927C596742AC01C250C9347AD419D2C1AEEA735B6B11F5C1D8E2653C9D2AF44110932249F0BEF0C1A87A8CE9D243F8C15B5114E37792F7C11275A39F3B6A01C201020000C002000000008E98E32554D941F8814A7CBB4AE141E0438255A91FB3C15869A51ACB8DF64148F134756C1BD541782734D434CEE541D2A630517970EDC1D4E6B252035EE9C1':::GEOMETRY,'2024-10-01 16:43:34.000523':::TIMESTAMP,3802:::OID,(-913133928):::INT8),('\x459696':::BYTES,0.4193844199180603:::FLOAT8,'010500008006000000010200008002000000A891CFEE569ED5C120DB03FE2261CF41FC03F70CD15CFB4108330D66AE3502C2B22D05F66CDCF441BE486DF9FD4EE5C1010200008003000000D03A08D0BD5EC7C14C7DB16F3F3FFD416F398E7C9443F2C19A88FCE1B704E6C170177B65505D024288E54898E32801C23C58443F4EBEF1C158763D242B20F141C3FE0F1C0863F5C10102000080090000009017E8EAE4C9C3C1DC16B1A7E61FD5C1AC201BDA452EFBC1DCE9B23441D3D8C16BFA035057FCF1C1568143F4EEF4F241483D5449F70ED141A4664EB2C2DEFBC10080F7604DDBF4410438B8A3C049E541A3CA27B8391EF4C128D6AE7D7C41FDC196226B46E347F3410710CC3A5230F6C1CEC7875002EFEFC1CA3B40DCC3F0F54194C7E1915FCEF6C102053B929516F14164B91A654104FE41107A2984E666F2C1BE728717EB23F8C1300537BA1980FB41AC3CFF7DC888F541D667219C58C8E2C100528CAC6836CA41F84A153B2D01F4414453D2098D82F2C10102000080060000006631AB5F6F43F141007ACC05C969E4C1704B7E07500ED141E082D445520EF54188749D75A0DED1C180ABE75289149541D43671A47016FD4150526FDE6CA9F141EC64D142FC64F941FA320B994B73F6415A10BC0CD19CF841C67F216AFD9300C211BF8C395C51F9C1E212E9858C6FF341F8A6A87FF252FF4192FAA0026E3FF0C1A0902217CD60BBC12EA0E5C6917DFEC1010200008003000000783292878C2EE4411155D9AEEAC6F8C1A841F240C127EF41BC0970AF9374FC411A37A31B5E03E3C1E90FECF89344F9C16038AD18814CC54124224CABEAF7DBC18040F02E8F8491C10102000080030000004C770C3E5B33FEC120F14089DCEDE1C15D1AD28D720E00C2ACCED327DF52E3C110F54E23BE37CB4198A13397CB03FAC1826B979F2668EEC1B0EDE83A1411D941F3DEFE45F53401C2':::GEOMETRY,'2006-03-21 06:18:56.000593':::TIMESTAMP,869:::OID,(-1713327734):::INT8)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 037C30F74EABEC1B47CDFC14F31004201020000C004000000047C31C3A12DE341E9C0792AE6E1F0C1BE5D9DB5FFDDF44110CE3F0B10BCD24154C3399AE46EFB4122FE34A94A5FFBC16A827C04AE2AE3C1E70376043A6FFBC1AC171669CC74F3412CE4BFF108DAE841CC1715177EAAE3416345C35D8116F1C1CCA8BBEA2DF3E5C1986BA9B35C6ADC418211A90107CFE4C1C481DECB38A2FB4101020000C003000000C00B9D03E082EE41D04B3BB9883CC841C60BA4AD67860142E016D0444DBEFBC15CB8FBCA89D9004280B92CB0E44EDD41A894C9A740AFECC18057DC75E78491416C3A6BB6FC51E141341F9AF8F0D0FD4188054E577960D141C03B7E306A15F94101020000C0060000004CA63BD8BB2DE0415E68EB85AC23E4C12CC0FCF35FA3E0C1491E933B553601C2849AB3B56880F3419D5A8C5D42C2F3C1E4AF32324D5FFD410480A5DE657DEF4110FF73A2145700421EB79589F876F3C1266044D25912F4C1680CA939CBB1FFC1C80F223DF456F841788B7F5B7A92D641BE71EF7F46C10042B87DB12BB5BDFEC166FF30631044004264182AB1D03301421464173D76C8FE41A09EC00EA331C0C1F88E633209B5F9C1C0109A94E9B8C7414BCDEDD33445F1C1828056E13537F64101020000C002000000761AA09A1162F641E0F16A8368C0B741EF0D8800DB23F3C102EB06710DDEFFC1E07CBF99746EFC417049FA53072EEB41164AF5A4078A00C2C4F9998A25BCFA4101010000C0EC58A6FE6C74EFC178DAAAFAC50DFD41128962C47B3BF94180E97A4C5DD2EDC101050000C00200000001020000C0060000009B7162DF7496FAC1A6772DE6356EFDC11990480400E6F7C1C0F30E69ED7AB3C154B38E0E93F7E3C172CDCB13741CFCC13440FC17BD2AF3C19ED6437FE7B2FAC1C072F6D5B611AEC1A2AF6248EE1700C26CB57AF5D66DE0C1EACCC29AFDAEF0C180E55F95E7FF9241195A06C66DD5F5C11C35F0E1378201C29C0E7AA75FAEE041809E1D601004EE415E1DDA928DEFEDC10E8411911519F8C1BCB64091D3EBF3C190B2896E2AC4C54184728649BD0B0142083D8B47FB0CD441002F01967CE801C201020000C003000000981C2A178CE1FC41C127D7B05A7001C2342EDC28FC85D6C1DC440FBA2549E6C14C6C80026122FB41F4D3BE883FB7DBC1A8C05EDF51A6DC41548EA382CEBEEA418A73CB8CC41A00424090F33A69C5D741901F20284BB4FE411D98F81521D501C201060000C00100000001030000C0010000000700000010932249F0BEF0C1A87A8CE9D243F8C15B5114E37792F7C11275A39F3B6A01C2E66077E54DF0E9C1AED89A07839FFAC14A3F2B6B30530042E880B860D4A4E9410AC3CDC18B92F841B3C49510B568F8C10AB1157611B5F9C19233A08B5C240242822D73D80D74F44180E809303F09A5C1CEC803876A17FAC14C8D51A1C2C3EDC1F6D90444945CFA4194617185E0ABFB410ACE17F735E0EBC1E0EDF26B1C48F741927C596742AC01C250C9347AD419D2C1AEEA735B6B11F5C1D8E2653C9D2AF44110932249F0BEF0C1A87A8CE9D243F8C15B5114E37792F7C11275A39F3B6A01C201020000C002000000008E98E32554D941F8814A7CBB4AE141E0438255A91FB3C15869A51ACB8DF64148F134756C1BD541782734D434CEE541D2A630517970EDC1D4E6B252035EE9C1':::GEOMETRY,'2024-10-01 16:43:34.000523':::TIMESTAMP,3802:::OID,(-913133928):::INT8),('\\x459696':::BYTES,0.4193844199180603:::FLOAT8,'010500008006000000010200008002000000A891CFEE569ED5C120DB03FE2261CF41FC03F70CD15CFB4108330D66AE3502C2B22D05F66CDCF441BE486DF9FD4EE5C1010200008003000000D03A08D0BD5EC7C14C7DB16F3F3FFD416F398E7C9443F2C19A88FCE1B704E6C170177B65505D024288E54898E32801C23C58443F4EBEF1C158763D242B20F141C3FE0F1C0863F5C10102000080090000009017E8EAE4C9C3C1DC16B1A7E61FD5C1AC201BDA452EFBC1DCE9B23441D3D8C16BFA035057FCF1C1568143F4EEF4F241483D5449F70ED141A4664EB2C2DEFBC10080F7604DDBF4410438B8A3C049E541A3CA27B8391EF4C128D6AE7D7C41FDC196226B46E347F3410710CC3A5230F6C1CEC7875002EFEFC1CA3B40DCC3F0F54194C7E1915FCEF6C102053B929516F14164B91A654104FE41107A2984E666F2C1BE728717EB23F8C1300537BA1980FB41AC3CFF7DC888F541D667219C58C8E2C100528CAC6836CA41F84A153B2D01F4414453D2098D82F2C10102000080060000006631AB5F6F43F141007ACC05C969E4C1704B7E07500ED141E082D445520EF54188749D75A0DED1C180ABE75289149541D43671A47016FD4150526FDE6CA9F141EC64D142FC64F941FA320B994B73F6415A10BC0CD19CF841C67F216AFD9300C211BF8C395C51F9C1E212E9858C6FF341F8A6A87FF252FF4192FAA0026E3FF0C1A0902217CD60BBC12EA0E5C6917DFEC1010200008003000000783292878C2EE4411155D9AEEAC6F8C1A841F240C127EF41BC0970AF9374FC411A37A31B5E03E3C1E90FECF89344F9C16038AD18814CC54124224CABEAF7DBC18040F02E8F8491C10102000080030000004C770C3E5B33FEC120F14089DCEDE1C15D1AD28D720E00C2ACCED327DF52E3C110F54E23BE37CB4198A13397CB03FAC1826B979F2668EEC1B0EDE83A1411D941F3DEFE45F53401C2':::GEOMETRY,'2006-03-21 06:18:56.000593':::TIMESTAMP,869:::OID,(-1713327734):::INT8)]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 43bfc3185cb362b14e0b8a331148f0b14542a4f2:

          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 115755 [runnable]:
          | runtime.gopark(0xc000f257b0?, 0x2?, 0xa7?, 0x45?, 0xc000f257a4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc000f25630 sp=0xc000f25610 pc=0x4644f6
          | runtime.selectgo(0xc000f257b0, 0xc000f257a0, 0xc0023789a0?, 0x0, 0xc0020ffec0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc000f25770 sp=0xc000f25630 pc=0x473fd2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc000f257e0 sp=0xc000f25770 pc=0xd0814c
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000f257e8 sp=0xc000f257e0 pc=0x4949e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 115796 [runnable]:
          | runtime.gopark(0xc000577fb0?, 0x2?, 0xf8?, 0x7e?, 0xc000577fa4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc000577e30 sp=0xc000577e10 pc=0x4644f6
          | runtime.selectgo(0xc000577fb0, 0xc000577fa0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc000577f70 sp=0xc000577e30 pc=0x473fd2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc000577fe0 sp=0xc000577f70 pc=0xd0814c
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000577fe8 sp=0xc000577fe0 pc=0x4949e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 110926 [runnable]:
          | runtime.gopark(0xc000571fb0?, 0x2?, 0xa7?, 0x45?, 0xc000571fa4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc000571e30 sp=0xc000571e10 pc=0x4644f6
          | runtime.selectgo(0xc000571fb0, 0xc000571fa0, 0xc0009c8fb0?, 0x0, 0xc000e03620?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc000571f70 sp=0xc000571e30 pc=0x473fd2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc000571fe0 sp=0xc000571f70 pc=0xd0814c
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000571fe8 sp=0xc000571fe0 pc=0x4949e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 3_816) ) )\" []] :false\nQUERY [\"SELECT array[((col803_807, col803_811, col803_810, col803_805, col803_813, col803_806, col803_804, col803_809, col803_808, col803_816))::STRING] FROM (VALUES(2950:::OID,e'\\\\U00002603'::VARCHAR,e'\\\\x10'::\\\"char\\\",700:::OID,'8889:36fd:d204:e5ae:df22:fedf:ecb9:d4ea/10':::INET,2206:::OID,'2027-10-29 06:17:59.00047+00:00':::TIMESTAMPTZ,'11:04:30.928401':::TIME,e'*\\\\x0fJ\\\\'KI=zX'::STRING,e'\\\\'':::STRING) ) AS T(col803_811,col803_810,col803_805,col803_813,col803_804,col803_809,col803_807,col803_806,col803_808,col803_816)\" []] :[[\"(\\\"2027-10-29 06:17:59.00047+00:00\\\",2950,☃,\\x10,700,11:04:30.928401,8889:36fd:d204:e5ae:df22:fedf:ecb9:d4ea/10,2206,*\\x0fJ'KI=zX,')\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table803'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table803' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table803', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table803'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [QUERY: CREATE SCHEMA schema1638 AUTHORIZATION root, Expected Errors: , Potential Errors:  QUERY: SELECT t0.col1307_1315 AS col0,t0.col1307_1318 AS col1,t1.col1307_1315 AS col2,t0.col1307_1320 AS col3,t0.col1307_1312 AS col4,t0.col1307_1308 AS col5,t1.col1307_1310 AS col6,t1.col1307_1311 AS col7,t1.col1307_1319 AS col8,t1.col1307_1312 AS col9,t1.col1307_1312 AS col10,t1.col1307_1313 AS col11,t0.col1307_1314 AS col12,t0.col1307_1311 AS col13,t1.col1307_1316 AS col14 FROM public.table1307 AS t0 ,public.table1307 AS t1  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO public.table803 (col803_804,col803_805,col803_806,col803_807,col803_808,col803_809,col803_810,col803_811,col803_812,col803_813,col803_814) VALUES ('120.141.174.26/17':::INET,e'\\U00002603'::\"char\",'06:33:00.367624':::TIME,'2005-08-06 20:47:39.000187+00:00':::TIMESTAMPTZ,e'a;n\\x13\\x1f\\tdC\\x1a'::STRING,90000:::OID,''::VARCHAR,21:::OID,19:::OID,701:::OID,NULL),('28e8:38e3:a84e:55b2:724c:1541:2854:ed38/124':::INET,'g'::\"char\",'24:00:00':::TIME,'2014-05-20 10:50:51.000547+00:00':::TIMESTAMPTZ,''::STRING,701:::OID,e'g0\\x0e\\x13Y\\x1dv'::VARCHAR,17:::OID,2950:::OID,701:::OID,(-1):::INT8),('8889:36fd:d204:e5ae:df22:fedf:ecb9:d4ea/10':::INET,e'\\x10'::\"char\",'11:04:30.928401':::TIME,'2027-10-29 06:17:59.00047+00:00':::TIMESTAMPTZ,e'*\\x0fJ\\'KI=zX'::STRING,2206:::OID,e'\\U00002603'::VARCHAR,2950:::OID,16:::OID,700:::OID,(-128):::INT8), Expected Errors: , Potential Errors: ]: ERROR: schema \"schema1638\" (452): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 773f7d4445ce3e0e806b7a182adba70a0f270f19:

          | runtime.gopark(0xc001328f78?, 0x3?, 0x0?, 0x30?, 0xc001328f4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc001328dc8 sp=0xc001328da8 pc=0x4644f6
          | runtime.selectgo(0xc001328f78, 0xc001328f44, 0xc001fea300?, 0x0, 0x25a1a60?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc001328f08 sp=0xc001328dc8 pc=0x473fd2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc0019ca2d0, {0x379f630, 0xc001fea300})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc001328fb8 sp=0xc001328f08 pc=0x18fdc8e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc001328fe0 sp=0xc001328fb8 pc=0x18fdf8e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc001328fe8 sp=0xc001328fe0 pc=0x4949e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 120306 [runnable]:
          | runtime.gopark(0xc0010df7b0?, 0x2?, 0xf8?, 0xf6?, 0xc0010df7a4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0010df630 sp=0xc0010df610 pc=0x4644f6
          | runtime.selectgo(0xc0010df7b0, 0xc0010df7a0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0010df770 sp=0xc0010df630 pc=0x473fd2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0010df7e0 sp=0xc0010df770 pc=0xd081ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0010df7e8 sp=0xc0010df7e0 pc=0x4949e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 119826 [runnable]:
          | runtime.gopark(0xc00132bf78?, 0x3?, 0x0?, 0x30?, 0xc00132bf4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00132bdc8 sp=0xc00132bda8 pc=0x4644f6
          | runtime.selectgo(0xc00132bf78, 0xc00132bf44, 0xc001fea300?, 0x0, 0x25a1a60?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00132bf08 sp=0xc00132bdc8 pc=0x473fd2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc0019cbc20, {0x379f630, 0xc001fea300})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc00132bfb8 sp=0xc00132bf08 pc=0x18fdc8e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc00132bfe0 sp=0xc00132bfb8 pc=0x18fdf8e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00132bfe8 sp=0xc00132bfe0 pc=0x4949e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 4A4F941BED4181D904A004234A8CCB08E6C0242B96340F9C45201C2B48354ED7A61F841198D0DD70DE6F9C138F82B1D9F50D5410103000000010000000A0000003814C04C02ABECC1D01FF3EB3F73DBC1A04336F71220D841C6BABC4E78DBF3C13CA6E6BE5766F4417BEBE8D4EB11FAC104C1DFB630ADE141BC504D0465A0E1C1BC6C2BAD729DE14108BE9476B3B5E241B0FD9265404BE84190B260B6916BF241A8B2F1EA7C15D2C114DF183B2A09E141985E0BA20DB9FCC1F8C246EEE08FE7417C6A6CB79094E7C1A0C3CF910AC8C6413814C04C02ABECC1D01FF3EB3F73DBC10103000000010000000600000068612E046ED2E5C11051657EB4EFF94136D7FD77A38EF2411CE39EEB11C0EF41D8ECCBEE6417EEC134C097C776B4FD41DB6831AC76D4F9C12C32192731E501429D73B389412C01C2CAF9B9F49DA7004268612E046ED2E5C11051657EB4EFF941010300000001000000090000004219DE64F0C8E3C15DFAC0469FD9F0C1260CB4B070F6E0C1384EF71123A9FFC1F235B173930CF641E723ACBCAEF001C2AC64FBAF7E2FF6416A52E4D23CC3ECC1B44EF0170E86E0418AB0E14E8AD6004258CD30C1CA54D2418865E809688EFD410454C344E4AED5C1789AD73411EAFE4118DA9870BBD6E7C198D978305825F9414219DE64F0C8E3C15DFAC0469FD9F0C10103000000010000000D0000001A7A94E7FAFEF5C19288508DD1DAE0C1A5785A0F584BF2C192AA12D85C57F8C184733D09C872E141391522514B2CF5C194DF6B3A674DF1418A43DA0BDF04FFC1B2D0EC642FA6F241CA324E013228FCC17C6F534A5A610242363E28687341F6413E9BC3DA3327F2412280D4A25364F141A633076D2BBDF041A407324CA0EBF1418871F46A705DEE41F2FFEA3D3A2CF241F404A8DE95EBE5413451DB659817FC41249ADE825487D6C12852473BAC22E941E4F930A4DFAFE9C1182CF8371F45ED411A7A94E7FAFEF5C19288508DD1DAE0C10103000000010000000A0000009BA0DC3BB64102C2570D1C51EA7CFCC1E5FCCE8FBE9AF0C1DDC5F3D9BA7A01C2BC949BE187D2E7C1E4781D5F6095FCC1C674A143770FF9417977DDC7CADAFAC1E874402F5233D941F0C2BC837A38C741CC95DF5DF026E5413888E3283EFDF54106884030EC67E0C1545B6306D122E641E87CC92629BCF0C144EDA2792832EF41F73FF5630ED401C2202ED48847EABC419BA0DC3BB64102C2570D1C51EA7CFCC10103000000010000000C0000005093ED966133E2C194F1D9BD7752D5C1528A835AF45AF0C104C4BE3DF943FCC1C0B4DE950A65AE41A0B7C491946BF0C1002F2980525687412890D8EA7100FBC1D4437C124789E941F88170F56772D74118E8DDAAE12AFB41446629E04B35EE4168B0FB05BDFB00424693DB62E5ECF4419C5148643AC1F641FE492FEC7810F041F8D01B955310E2415C3487D7EEA8FF41F0D925C796FDCCC1F03FA8962DB8FD411DDEDD39AE94F8C1FC264B842E32FE415093ED966133E2C194F1D9BD7752D5C101030000000100000004000000B054F8673D71F941650D65867CC5FBC13E257AB0852100422E39C841856BF441381530009AD6EC4100DDE4F4AEA2BC41B054F8673D71F941650D65867CC5FBC101030000000100000006000000185076DE73B0DE41B0DC20646835C9C120C3B1C8E7ACE14130EA326DEEEFCBC1249AAA66BE26F6414A24DFA9B0D1E2C1A4A681D4B61802420AF2BACEC1DCE7C1EC92489C90DCEFC1FC111B67246BF541185076DE73B0DE41B0DC20646835C9C1':::GEOMETRY,'1975-03-05 19:53:25.000041':::TIMESTAMP,21:::OID,'BOX(-0.033147670190147055 -0.8969164895660259,0.662099216791473 0.04112007442859014)':::BOX2D), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema163.table2295 (col2295_2296,col2295_2297,col2295_2298,col2295_2299,col2295_2300,col2295_2301,col2295_2302,col2295_2303,col2295_2304,col2295_2305) VALUES (196833641:::INT8,'[{\"{F|]\": false}, {\"baz\": {\"a\": {\"@W,{JtR.\": [], \"v``7#c*-JW+#\": false}}, \"rgR;\": [null, {}]}, 1.2893679982347823, []]':::JSONB,e'{\"\u0026O$Ttou@q;\": [], \"b\": {}, \"dZEU\u003e+@T\": {\"zfv7V_l.N\\\\\"^@\": []}}':::JSONB,90002:::OID,'-12 years -2 mons -171 days -12:17:05.695389':::INTERVAL,1042:::OID,'010400000000000000':::GEOMETRY,'2005-08-11 18:05:44.00039':::TIMESTAMP,1560:::OID,'BOX(-0.08184191738962293 -1.177141818253694,-0.017663281363286876 0.8658736915258685)':::BOX2D),((-45754491):::INT8,NULL,'{\"test\": \"json\"}':::JSONB,700:::OID,'71 years 1 mon 915 days 24:44:39.159594':::INTERVAL,2205:::OID,'01040000000500000001010000008A4E9F23B652F44110E2EEA61B25FC410101000000F4F5AF551965FC4118B5C8B0CD91F8C101010000007D503BC9305D01C27E576B1CE86FE7C101010000005404A2255981E74130A661715EA7DF410101000000A0098195345BD741BAE727D0A689F641':::GEOMETRY,'2018-10-15 23:15:03.000708':::TIMESTAMP,1561:::OID,'BOX(-1.2410489215297957 -0.6621837732277828,0.023749281329676963 2.3499112210123245)':::BOX2D), Expected Errors: , Potential Errors: ]: ERROR: internal error: queryOid: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 0dd438d3dc0b42543890455945a7a6b42811def1:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_144947.342921811_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_144947.342921811_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |                    (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema783.table1537"]] :[]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema783.table1537'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema783' AND\n\t\t\t\t\t\t\t\t    table_name='table1537' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema783.table1537', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema783'\n\t\t           AND table_name = 'table1537'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [QUERY: CREATE TABLE IF NOT EXISTS schema1652.table2258 (col2258_2259 REGTYPE, col2258_2260 REGCLASS NULL, col2258_2261 REGTYPE, col2258_2262 FLOAT8, col2258_2263 REGNAMESPACE NULL, col2258_2264 REGPROCEDURE NULL, col2258_2265 REGCLASS, col2258_2266 BOX2D NOT NULL, col2258_2267 JSONB NOT NULL, INDEX (col2258_2260 DESC) STORING (col2258_2259, col2258_2262, col2258_2263), FAMILY (col2258_2266, col2258_2259, col2258_2260), FAMILY (col2258_2263, col2258_2267), FAMILY (col2258_2261, col2258_2265), FAMILY (col2258_2264, col2258_2262)), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema783.table1537 (col1537_1538,col1537_1539,col1537_1540,col1537_1541,col1537_1542,col1537_1543,col1537_1544,col1537_1545,col1537_1546,col1537_1547,col1537_1548) VALUES (90002:::OID,(-5397441281.989083831):::DECIMAL,'1993-04-30 01:18:56.000975':::TIMESTAMP,'-Infinity':::DECIMAL,3802:::OID,'22:07:26.89967-01:29:00':::TIMETZ,1082:::OID,(-18956):::INT8,90000:::OID,e'\x110=>\x0e'::NAME:::NAME,''::VARCHAR), Expected Errors: , Potential Errors: ]: ERROR: relation "table2258" (598): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |                FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema783.table1537\"]] :[]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema783.table1537'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema783' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table1537' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema783.table1537', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema783'\\n\\t\\t           AND table_name = 'table1537'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [QUERY: CREATE TABLE IF NOT EXISTS schema1652.table2258 (col2258_2259 REGTYPE, col2258_2260 REGCLASS NULL, col2258_2261 REGTYPE, col2258_2262 FLOAT8, col2258_2263 REGNAMESPACE NULL, col2258_2264 REGPROCEDURE NULL, col2258_2265 REGCLASS, col2258_2266 BOX2D NOT NULL, col2258_2267 JSONB NOT NULL, INDEX (col2258_2260 DESC) STORING (col2258_2259, col2258_2262, col2258_2263), FAMILY (col2258_2266, col2258_2259, col2258_2260), FAMILY (col2258_2263, col2258_2267), FAMILY (col2258_2261, col2258_2265), FAMILY (col2258_2264, col2258_2262)), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema783.table1537 (col1537_1538,col1537_1539,col1537_1540,col1537_1541,col1537_1542,col1537_1543,col1537_1544,col1537_1545,col1537_1546,col1537_1547,col1537_1548) VALUES (90002:::OID,(-5397441281.989083831):::DECIMAL,'1993-04-30 01:18:56.000975':::TIMESTAMP,'-Infinity':::DECIMAL,3802:::OID,'22:07:26.89967-01:29:00':::TIMETZ,1082:::OID,(-18956):::INT8,90000:::OID,e'\\x110=\u003e\\x0e'::NAME:::NAME,''::VARCHAR), Expected Errors: , Potential Errors: ]: ERROR: relation \"table2258\" (598): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ f4042d47fa8062a612c38d4696eb6bee9cee7c21:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_141354.975317233_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_141354.975317233_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | \n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [QUERY: SELECT t0.col3161_3167 AS col0,t1.col3225_3226 AS col1 FROM schema1183.table3161 AS t0 ,schema1768.table3225 AS t1  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO public.table2446 (col2446_2447,col2446_2448,col2446_2449,col2446_2450,col2446_2451,col2446_2452,col2446_2453,col2446_2454,col2446_2455,col2446_2456,col2446_2457,col2446_2458,col2446_2459,col2446_2460) VALUES ('r'::"char",0.41360585443710884:::FLOAT8,e'{".Q\\"c4+W": 1.9568733329994854, "=y,z": {}, "foo": [{"foo": true}, [], {}], "wc9=~": {"A]wyCyMg": {"98sC&4YO^": "%]]ffYkLm", "c": true}}}':::JSONB,0.48026201050252054:::FLOAT8,'Bz'::VARCHAR,'3137bd1e-04fb-4b1a-9e00-a84b135cfc55':::UUID,1184:::OID,'2015-12-09 07:11:44.000622+00:00':::TIMESTAMPTZ,'BOX(-0.6884771216436342 -3.029306779832776,1.4167670423060694 -0.24407750579826068)':::BOX2D,'309c:3478:1d66:742d:741b:9ac5:d170:8e/12':::INET,'Q'::CHAR,e'\\'::CHAR,'02:28:22.900407-06:09:00':::TIMETZ,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID), Expected Errors: , Potential Errors:  QUERY: ALTER SEQUENCE schema2930.seq3409 RENAME TO schema2930.seq4278, Expected Errors: , Potential Errors:  QUERY: CREATE TABLE IF NOT EXISTS schema1183.table3161 (col3161_4280 OID NULL, col3161_4281 BOOL NOT NULL, col3161_4282 TIME NOT NULL, col3161_4283 GEOMETRY NOT NULL, col3161_4284 REGNAMESPACE NOT NULL, col3161_4285 VARCHAR[] NOT NULL, col3161_4286 CHAR NULL, col3161_4287 TIME NOT NULL, col3161_4288 STRING NULL AS (lower(col3161_4286)) VIRTUAL, col3161_4289 STRING NOT NULL AS (lower(CAST(col3161_4282 AS STRING))) STORED, col3161_4290 STRING NULL AS (lower(col3161_4286)) VIRTUAL, col3161_4291 STRING NOT NULL AS (lower(CAST(col3161_4283 AS STRING))) STORED, col3161_4292 STRING NOT NULL AS (lower(CAST(col3161_4287 AS STRING))) VIRTUAL, col3161_4293 STRING NULL AS (lower(CAST(col3161_4280 AS STRING))) STORED, col3161_4294 STRING NOT NULL AS (lower(CAST(col3161_4280 AS STRING))) VIRTUAL, INDEX (col3161_4288 ASC), FAMILY (col3161_4287), FAMILY (col3161_4280, col3161_4284), FAMILY (col3161_4289), FAMILY (col3161_4282, col3161_4286, col3161_4285), FAMILY (col3161_4281), FAMILY (col3161_4291, col3161_4293), FAMILY (col3161_4283)), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema1183.table3161 (col3161_3162,col3161_3163,col3161_3164,col3161_3165,col3161_3166,col3161_3167,col3161_3168,col3161_3169,col3161_3170,col3161_3171,col3161_3172) VALUES (1082:::OID,(-1.4857067069228798):::FLOAT8,3802:::OID,B'110111011000001011010000000101100001','BOX(-1.455390034810053 -1.1345515835703068,1.0752517555045316 0.6043571256313037)':::BOX2D,'[{"*gpq!": {}, "X%VTVK>xcN": {"bar": null}, "b": [{"foobar": true}, {}], "foo": 0.021596469527626616, "foobar": true}, true, [{}]]':::JSONB,(-1754358253):::INT8,'1.548718274059098':::JSONB,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,'1978-01-25':::DATE,90004:::OID),(90000:::OID,(-0.6673481304536667):::FLOAT8,23:::OID,B'000000000000000000000000000000000000','BOX(-0.1441023801393919 -0.5210741361570059,2.4086127271975566 0.7385558874914988)':::BOX2D,e'{"\'+9 ": [{" ),%S]": 0.2827681864364359, "foobar": false}, [], {}], "=>.3eqAut6": [], "R]huT@\\\\": [], "foobar": {}, "i&8>|vEA": [], "wWe%,v+xCvY": [[]]}':::JSONB,(-1136927873):::INT8,'[{"baz": [], "c": {"0F7a+cBU7d!.": {}, "bar": {"bar": {}}, "c": {"Ei8H3h": {}}}, "foo": 0.4795300322395375, "oQ}=": true}, {"foobar": [{"bar": {}}]}, ["c"]]':::JSONB,'66.48.143.55/28':::INET,'1980-05-13':::DATE,26:::OID),(1043:::OID,0.6555798392087997:::FLOAT8,1700:::OID,B'010010111101011001001001001111001100','BOX(-2.2707644085944327 -1.4325096177054282,-0.4994924960855296 -0.19614029643008546)':::BOX2D,'true':::JSONB,(-1641313872):::INT8,e'{"(o\\\\M0pyo}Ny": {}, "M!y%X": [], "bar": ["foobar"], "foobar": null, "z;{t?T(m": [[]]}':::JSONB,'155.56.83.46/5':::INET,'1990-02-27':::DATE,701:::OID), Expected Errors: , Potential Errors: ]: ERROR: relation "seq4278" (831): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 7 AS col0,t1.col3225_3226 AS col1 FROM schema1183.table3161 AS t0 ,schema1768.table3225 AS t1  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO public.table2446 (col2446_2447,col2446_2448,col2446_2449,col2446_2450,col2446_2451,col2446_2452,col2446_2453,col2446_2454,col2446_2455,col2446_2456,col2446_2457,col2446_2458,col2446_2459,col2446_2460) VALUES ('r'::\"char\",0.41360585443710884:::FLOAT8,e'{\".Q\\\\\"c4+W\": 1.9568733329994854, \"=y,z\": {}, \"foo\": [{\"foo\": true}, [], {}], \"wc9=~\": {\"A]wyCyMg\": {\"98sC\u00264YO^\": \"%]]ffYkLm\", \"c\": true}}}':::JSONB,0.48026201050252054:::FLOAT8,'Bz'::VARCHAR,'3137bd1e-04fb-4b1a-9e00-a84b135cfc55':::UUID,1184:::OID,'2015-12-09 07:11:44.000622+00:00':::TIMESTAMPTZ,'BOX(-0.6884771216436342 -3.029306779832776,1.4167670423060694 -0.24407750579826068)':::BOX2D,'309c:3478:1d66:742d:741b:9ac5:d170:8e/12':::INET,'Q'::CHAR,e'\\\\'::CHAR,'02:28:22.900407-06:09:00':::TIMETZ,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID), Expected Errors: , Potential Errors:  QUERY: ALTER SEQUENCE schema2930.seq3409 RENAME TO schema2930.seq4278, Expected Errors: , Potential Errors:  QUERY: CREATE TABLE IF NOT EXISTS schema1183.table3161 (col3161_4280 OID NULL, col3161_4281 BOOL NOT NULL, col3161_4282 TIME NOT NULL, col3161_4283 GEOMETRY NOT NULL, col3161_4284 REGNAMESPACE NOT NULL, col3161_4285 VARCHAR[] NOT NULL, col3161_4286 CHAR NULL, col3161_4287 TIME NOT NULL, col3161_4288 STRING NULL AS (lower(col3161_4286)) VIRTUAL, col3161_4289 STRING NOT NULL AS (lower(CAST(col3161_4282 AS STRING))) STORED, col3161_4290 STRING NULL AS (lower(col3161_4286)) VIRTUAL, col3161_4291 STRING NOT NULL AS (lower(CAST(col3161_4283 AS STRING))) STORED, col3161_4292 STRING NOT NULL AS (lower(CAST(col3161_4287 AS STRING))) VIRTUAL, col3161_4293 STRING NULL AS (lower(CAST(col3161_4280 AS STRING))) STORED, col3161_4294 STRING NOT NULL AS (lower(CAST(col3161_4280 AS STRING))) VIRTUAL, INDEX (col3161_4288 ASC), FAMILY (col3161_4287), FAMILY (col3161_4280, col3161_4284), FAMILY (col3161_4289), FAMILY (col3161_4282, col3161_4286, col3161_4285), FAMILY (col3161_4281), FAMILY (col3161_4291, col3161_4293), FAMILY (col3161_4283)), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema1183.table3161 (col3161_3162,col3161_3163,col3161_3164,col3161_3165,col3161_3166,col3161_3167,col3161_3168,col3161_3169,col3161_3170,col3161_3171,col3161_3172) VALUES (1082:::OID,(-1.4857067069228798):::FLOAT8,3802:::OID,B'110111011000001011010000000101100001','BOX(-1.455390034810053 -1.1345515835703068,1.0752517555045316 0.6043571256313037)':::BOX2D,'[{\"*gpq!\": {}, \"X%VTVK\u003excN\": {\"bar\": null}, \"b\": [{\"foobar\": true}, {}], \"foo\": 0.021596469527626616, \"foobar\": true}, true, [{}]]':::JSONB,(-1754358253):::INT8,'1.548718274059098':::JSONB,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,'1978-01-25':::DATE,90004:::OID),(90000:::OID,(-0.6673481304536667):::FLOAT8,23:::OID,B'000000000000000000000000000000000000','BOX(-0.1441023801393919 -0.5210741361570059,2.4086127271975566 0.7385558874914988)':::BOX2D,e'{\"\\'+9 \": [{\" ),%S]\": 0.2827681864364359, \"foobar\": false}, [], {}], \"=\u003e.3eqAut6\": [], \"R]huT@\\\\\\\\\": [], \"foobar\": {}, \"i\u00268\u003e|vEA\": [], \"wWe%,v+xCvY\": [[]]}':::JSONB,(-1136927873):::INT8,'[{\"baz\": [], \"c\": {\"0F7a+cBU7d!.\": {}, \"bar\": {\"bar\": {}}, \"c\": {\"Ei8H3h\": {}}}, \"foo\": 0.4795300322395375, \"oQ}=\": true}, {\"foobar\": [{\"bar\": {}}]}, [\"c\"]]':::JSONB,'66.48.143.55/28':::INET,'1980-05-13':::DATE,26:::OID),(1043:::OID,0.6555798392087997:::FLOAT8,1700:::OID,B'010010111101011001001001001111001100','BOX(-2.2707644085944327 -1.4325096177054282,-0.4994924960855296 -0.19614029643008546)':::BOX2D,'true':::JSONB,(-1641313872):::INT8,e'{\"(o\\\\\\\\M0pyo}Ny\": {}, \"M!y%X\": [], \"bar\": [\"foobar\"], \"foobar\": null, \"z;{t?T(m\": [[]]}':::JSONB,'155.56.83.46/5':::INET,'1990-02-27':::DATE,701:::OID), Expected Errors: , Potential Errors: ]: ERROR: relation \"seq4278\" (831): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ b173a16715e71e94115820374da1eb350b3b459d:

          | runtime.gopark(0xc00224dfb0?, 0x2?, 0x60?, 0xde?, 0xc00224dfa4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00224de30 sp=0xc00224de10 pc=0x465216
          | runtime.selectgo(0xc00224dfb0, 0xc00224dfa0, 0xc002dc85e0?, 0x0, 0xc000f3cae0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00224df70 sp=0xc00224de30 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc00224dfe0 sp=0xc00224df70 pc=0xd2d80c
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00224dfe8 sp=0xc00224dfe0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 395294 [runnable]:
          | runtime.gopark(0xc00144cf78?, 0x3?, 0x0?, 0x30?, 0xc00144cf4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00144cdc8 sp=0xc00144cda8 pc=0x465216
          | runtime.selectgo(0xc00144cf78, 0xc00144cf44, 0xc0013f4be8?, 0x0, 0x3f20a00?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00144cf08 sp=0xc00144cdc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc0023f6320, {0x549f958, 0xc000ae5f00})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc00144cfb8 sp=0xc00144cf08 pc=0x1a9274e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc00144cfe0 sp=0xc00144cfb8 pc=0x1a92a4e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00144cfe8 sp=0xc00144cfe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 393237 [runnable]:
          | runtime.gopark(0xc0000a9f78?, 0x3?, 0x0?, 0x30?, 0xc0000a9f4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0000a9dc8 sp=0xc0000a9da8 pc=0x465216
          | runtime.selectgo(0xc0000a9f78, 0xc0000a9f44, 0xc000ae5f00?, 0x0, 0x2e3028584f42272c?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0000a9f08 sp=0xc0000a9dc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002c54140, {0x549f958, 0xc000ae5f00})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0000a9fb8 sp=0xc0000a9f08 pc=0x1a9274e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0000a9fe0 sp=0xc0000a9fb8 pc=0x1a92a4e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000a9fe8 sp=0xc0000a9fe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 03348C012C77920D158F7C17465CDFFA7E5F0C1389FEEC638EB63C023EC248A394A53C02EF3380BBE1CE5C156F9874DD5F8F0C101010000E0E610000000E0B30277B15940119BC6F1E82B48C00E15CD022062F3C110C36501950BE9C1':::GEOGRAPHY,ARRAY['f3e49dfa-e282-4416-af25-aa79efadbecc':::UUID,'f6e0576e-2213-483c-8cec-b4326cdf846f':::UUID,'00000000-0000-0000-0000-000000000000':::UUID,'41f9c25c-b05e-4654-8df7-e934769d2f94':::UUID],'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET,1028:::OID,'01070000C00200000001020000C0070000007E941682C755F2C1B8969E36090601C2EA4ABF3EA5DCF5C1606FB4849C6BC0C1B0B6411CAF71E44100704337D6E2C741284062D878D2FF41605FE7BAE65CE041981DB7A3AC1FF24178BAD2EEA8AAE241228A6B93564C0042404926D2F3C4B3415A7862CDC215F041D83ECE58E8B2E741BDA2603EBF69FAC1686303A9EF3CE1C1182B8258E497DA41D2448C08DD7FF04183FCA1561859F4C100C820CB5828B8C110D09C85845FC6417A9EBFA4708F01429338A901488DFDC19A93EB329D88F0C1FF36A8910464F5C10CCBC915C62CE941B877986A78E4D241E832C5DEBC54DE4101030000C001000000080000005432CFF47F7001C24CF557AD463D02C2F6CF806A8118ECC1A8CCF75262D0D0416E7DA65F39B7FCC1DE04ED160BB301C288D9A3001425DDC1CA5F0CCAD2F8F2414045A31C5672ADC1CD5B3410AE8501C222523A964099F541E4082CB09571E2415C8F94909F93E141BC19F16DF29E00C26861C26E07D7F641221DC1DAD406E1C104542EB8A154E3C19044B1F8CFACC9C162D27C896EC4FBC1B8F7A60350E7F941B81091C629AEEAC14629CF754EE5EEC16C92AF7B224EEA41E2C32393CD7802428C6AC89C94D7F6C168C1C4D688EEE9418AE0424C11AAF54190E24AF330C6F2C15432CFF47F7001C24CF557AD463D02C2F6CF806A8118ECC1A8CCF75262D0D041':::GEOMETRY,'|OM8``o'::NAME:::NAME,NULL,4767439992144945501:::INT8,'010400000000000000':::GEOMETRY,(-0.4420646131038666):::FLOAT8,1505003948:::INT8,(-24203):::INT8,'05:51:06.282561':::TIME,'2029-09-18 14:39:30.000446+00:00':::TIMESTAMPTZ),('0105000020E6100000020000000102000000020000000C242AB3B54F60400B0693C137B442C088E25DC9815A6140F013A4E272B12DC0010200000004000000B2F7D8DD617D65409C23CC6D3B013AC05ADD5F1EF45562401CE4211378514140346EB96E98DD4D4000A525B9E2FB4A402435EF7E731A4F40A8CD2B0A2B535040':::GEOGRAPHY,ARRAY['94a16528-61ae-410b-adaa-1fe0939b1fbe':::UUID,'790cd9af-e6fb-4276-bacd-eb423b25fb90':::UUID],'76e2:62f6:f990:b2e:a1b6:9f47:2035:c8f0/103':::INET,1184:::OID,'0101000080524A1B067D8DF541C89E7A3E255CFEC17A4C40100535F441':::GEOMETRY,e'\\x07\\x7f'::NAME:::NAME,'\\x':::BYTES,(-3314324957254518985):::INT8,'01050000C00700000001020000C00200000088B38F5DB63CD1C1E0E2E63592F0F6416014F39ED042B1C1438430149B7801C2B7D2FA2D26E9FDC120279E55CEBFF64114AB2069A852E1414079101AACF7AE4101020000C002000000B8B8BC9BB40AF3C1D0A171A80C22C4C1DCB693A89515E6C1D8785FBCB069D3C110124D7F22A3FFC13A8A598CF8EE01C2041392FCFEF9F7413E26E31F974BE4C101020000C00200000012F270350D59E6C1C2C65DED2262FA4140E56F71F112E04150632C1D951202422A9C0FFA94FAF8C1C2DDDCA4DADEF241C69DBC0D6A6201422A1FAB9DCF0FFBC101020000C006000000F04B106BBECCE1C156C92F725603F4C180CE30CABEEBFB415A6999EBDFA3E3C1D055B463826ED4C1609DC7A7CE43FDC15C116F698BDFF4414C5B5D5EAA3AF9C1BC25454E5815D7C1C8093D688444EFC182B6DC21C0E1F641D8777A8D5D55EB41A048EF71B562C0C1FBD82E260025FAC1C6BBB01DA3DEF5411C4F8A18B1A4F34150A7BDB8A0A9DB41581B75FBC601FBC13CD506A25815F141088595296CD0E341A07F7CA46EA3D741EA1C746ACA15F54186AF80F5B4A7E4C120D3D75C0B8DC34101020000C002000000EE341A800075E8C14E41A6D3975EF341ACE2C544B073D5C1407197100D4FF541809B387C8F2701C2A05CE996311DE241FA97F97B3E28F2411E8403076EDDF34101020000C005000000C07263D601A4CFC181D4DDE174FDFBC1D294D39FA7AAF7414C6F1BF6DF94FA4140D994700F01BA4126B7B2F576F6ECC1C9614752F342F1C102FF549F0CCCF54118E5E4B9026CFD4174A11DE3A686F6C1C026E595B982A341F4FD89269936FDC1D0B32C802ACEEC4190A0A577C4B8D0C1A4436971EDC1F8C17085E6062084DE4138B78CA52291F941087DC7BE8FAEF6418A2E470F2CF9FBC1FA039C38CB67014201020000C00300000050C0BAAAF08AF9417AE2142FE2D6FFC1700BC471B0B3C04132117C92FA41F041E0B5749ECEAAE641802EA4F83277CC416087C333CE48CCC13018CCD0C2BBD6C1704C2300F8B4F941C81C6D4FFA0EF941E005290CB93EBDC18EA8030A9250E6C1':::GEOMETRY,(-0.63350510597229):::FLOAT8,1685280491:::INT8,(-8110):::INT8,'23:49:44.619477':::TIME,'1987-06-04 14:26:53.000748+00:00':::TIMESTAMPTZ), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 5c2c62ecf1bea60c807edc6b4da22d900ad4ae03:

          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc001d86fb8 sp=0xc001d86f08 pc=0x1a965ae
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc001d86fe0 sp=0xc001d86fb8 pc=0x1a968ae
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc001d86fe8 sp=0xc001d86fe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 20431 [runnable]:
          | runtime.gopark(0xc00188cfb0?, 0x2?, 0x0?, 0x0?, 0xc00188cfa4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00188ce30 sp=0xc00188ce10 pc=0x465216
          | runtime.selectgo(0xc00188cfb0, 0xc00188cfa0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00188cf70 sp=0xc00188ce30 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc00188cfe0 sp=0xc00188cf70 pc=0xd2deac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00188cfe8 sp=0xc00188cfe0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 87.256151500,1 within uncertainty interval ``t \u003c= (local=1660830987.631586405,0, global=1660830987.631586405,0)``; observed timestamps: [{1 1660830987.131586405,0} {2 1660830988.169682787,0} {3 1660830988.912203796,0}]: \"sql txn\" meta={id=f49287e4 key=/Table/116/1/\"\\x00\\x1c\\xf8\\x92ٮ\"/0 pri=0.00246000 epo=0 ts=1660830987.256151500,1 min=1660830987.131586405,0 seq=2} lock=true stat=PENDING rts=1660830987.131586405,0 wto=false gul=1660830987.631586405,0 (SQLSTATE 40001)"
          | }
          |    39.0s        0           13.0           13.4    201.3    469.8    604.0    604.0 opOk
          |    39.0s        0            3.0            2.1   2550.1   4160.7   4160.7   4160.7 txnCmtErr
          |    39.0s        0            1.0            1.3   4295.0   4295.0   4295.0   4295.0 txnOk
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |    39.0s        0            1.0            0.2     96.5     96.5     96.5     96.5 txnRbk
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:56:26.571474",
          |  "ops": [
          |   "BEGIN",
          |   "QUERY: INSERT INTO public.table51 (col51_52,col51_53,col51_54,col51_55,col51_56,col51_57,col51_58,col51_59,col51_60,col51_61) VALUES ('51cf:a880:5b6c:e9f4:e1e7:487b:bdb4:fb48/16':::INET,'C'::\"char\",'f'::CHAR,16:::OID,4.621268976567901209E+24:::DECIMAL,'2016-11-18 09:56:45.000313':::TIMESTAMP,(-1002501712):::INT8,'1973-02-22 14:40:30.000301+00:00':::TIMESTAMPTZ,17:::OID,true),('a930:f936:da41:a0f2:2155:fc56:efdb:d15e/79':::INET,'f'::\"char\",' '::CHAR,18:::OID,(-6.872022750759696440E+20):::DECIMAL,'1975-02-05 16:57:18.000271':::TIMESTAMP,447308527:::INT8,'2011-03-21 12:00:47.000443+00:00':::TIMESTAMPTZ,1266:::OID,false), Expected Errors: , Potential Errors: ",
          |   "QUERY: SELECT t1.col428_430 AS col0,t0.col51_53 AS col1 FROM public.table51 AS t0 ,schema262.table428 AS t1  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200",
          |   "QUERY: INSERT INTO schema262.table428 (col428_429,col428_430,col428_431) VALUES ('1981-11-20 17:10:24.000277+00:00':::TIMESTAMPTZ,'infinity':::DATE,23:::OID),('1976-08-29 08:11:02.000821+00:00':::TIMESTAMPTZ,'1978-05-01':::DATE,90004:::OID), Expected Errors: , Potential Errors: ",
          |   "QUERY: DROP SEQUENCE public.seq552, Expected Errors: , Potential Errors: "
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error.\n Dumping state before death:\nExpected errors: \nPotential errors: \nExpected commit errors: \nPotential commit errors: \n===========================\nExecuted queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT sequence_name\\n    FROM information_schema.sequences\\n   WHERE sequence_schema = $1\\n     AND sequence_name = $2\\n   )\" [\"public\" \"seq552\"]] :true\n\n===========================\nPrevious statements [QUERY: INSERT INTO public.table51 (col51_52,col51_53,col51_54,col51_55,col51_56,col51_57,col51_58,col51_59,col51_60,col51_61) VALUES ('51cf:a880:5b6c:e9f4:e1e7:487b:bdb4:fb48/16':::INET,'C'::\"char\",'f'::CHAR,16:::OID,4.621268976567901209E+24:::DECIMAL,'2016-11-18 09:56:45.000313':::TIMESTAMP,(-1002501712):::INT8,'1973-02-22 14:40:30.000301+00:00':::TIMESTAMPTZ,17:::OID,true),('a930:f936:da41:a0f2:2155:fc56:efdb:d15e/79':::INET,'f'::\"char\",' '::CHAR,18:::OID,(-6.872022750759696440E+20):::DECIMAL,'1975-02-05 16:57:18.000271':::TIMESTAMP,447308527:::INT8,'2011-03-21 12:00:47.000443+00:00':::TIMESTAMPTZ,1266:::OID,false), Expected Errors: , Potential Errors:  QUERY: SELECT t1.col428_430 AS col0,t0.col51_53 AS col1 FROM public.table51 AS t0 ,schema262.table428 AS t1  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO schema262.table428 (col428_429,col428_430,col428_431) VALUES ('1981-11-20 17:10:24.000277+00:00':::TIMESTAMPTZ,'infinity':::DATE,23:::OID),('1976-08-29 08:11:02.000821+00:00':::TIMESTAMPTZ,'1978-05-01':::DATE,90004:::OID), Expected Errors: , Potential Errors:  QUERY: DROP SEQUENCE public.seq552, Expected Errors: , Potential Errors: ]: ERROR: internal error: cannot use MemberOfWithAdminoption without a txn (SQLSTATE XX000)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ a0d8839aa6164af81a9ebb140147d3baf5321287:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_135936.470787343_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_135936.470787343_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [QUERY: CREATE VIEW schema4279.view4621 AS SELECT public.table3819.col3819_3820 FROM public.table3819, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema3806.table4023 (col4023_4024,col4023_4025,col4023_4026,col4023_4027,col4023_4028,col4023_4029,col4023_4030,col4023_4031,col4023_4032,col4023_4033,col4023_4034,col4023_4035,col4023_4036,col4023_4037,col4023_4038,col4023_4039) VALUES (e'{".[~^Z>@": {"&OgF": {}, "Q$u.3x7aP": [[false]]}, "\\\\DCR": {}, "]sE": false, "foobar": "@k.k"}':::JSONB,1042:::OID,'0106000020E6100000040000000103000000010000000600000054053FFDFD0F57C0243B76AA55B93040F8FEBAE31C3F5CC0806E09C16A62F13F6416EC6874DE5540586DA8BE17403EC0EA1D6CEF558564400438EE5B5A503040A63A941E762866C00AAA73048767564054053FFDFD0F57C0243B76AA55B9304001030000000100000008000000E0CFD1B0A55B15C0EC12DB330EE447C0407B463897DD2540D8FE570CC8774EC0C0B81941F0B052403E7FF3603C5654C01844CAA840A259404030918D73F70CC02C5C0145654044408A8DEE43F07A5640CF48E637DF9B5DC0C883606B7B102A407CC8DB9E902756C0808583E9BCC110C0E0CFD1B0A55B15C0EC12DB330EE447C001030000000100000004000000CADCEBA586FA52407847F712903E3E404EAFF6EED4BE614050685AC18E734340002071AFE9F6B13FCC6091C9A8E94440CADCEBA586FA52407847F712903E3E40010300000001000000070000001C673E15F5BC624095999192CC0D4BC082FF2EFB07425040A0E0EC28E66703C0E0F839D18A4E4F4004E0694DCA765440A06BC849CD4232C0FC3EE48710D943409F97535B56EB61C02C3A72CEAB4D4440D22CB31936C05EC0E8B671979D6935401C673E15F5BC624095999192CC0D4BC0':::GEOGRAPHY,19:::OID,'<}'::NAME:::NAME,'00:15:28.867825+01:24:00':::TIMETZ,'22e28527-dfa8-4f55-b7c6-cd68d7cb4ae1':::UUID,1.0:::FLOAT8,(-1472267120):::INT8,'\xac312686ee73':::BYTES,23:::OID,4096:::OID,e',}\x10\x03'::NAME:::NAME,'0101000040000000000000F87F000000000000F87F000000000000F87F':::GEOMETRY,''::NAME:::NAME,'77497147-cd0b-4e3d-927c-be77ca0ed4d0':::UUID),(e'{"&Z{uK<": false, "UO:": {"Um]": [], "b": false}, "a": true, "eZ,7me\\\\g\\\\Co": {}, "foobar": {}}':::JSONB,20:::OID,'01040000E0E61000000100000001010000C00092F67F43C2544098BABB18CCDE504034B6902B185EFD4125A97250739601C2':::GEOGRAPHY,17:::OID,'>``Y$N)'::NAME:::NAME,'02:55:31.380615-02:17:00':::TIMETZ,'3e9947aa-402a-49e9-b933-db8f1561669c':::UUID,1.9799622877237404:::FLOAT8,(-128):::INT8,'\x58':::BYTES,1042:::OID,4096:::OID,e'\x01'::NAME:::NAME,'0105000040070000000102000040030000006270D178D33AF6C14F92DFBBEA3DF4C1AA7D6E029A6202C26AB30498785302429763A213F08600C2B6AE94F99CF2E1C136E4AA39F0AAE6C17C3328993907E9410A139D5A9725F641010200004003000000B4EE6175D9E0DFC162F658B9096DFBC1D085D419092CCC412AFFB1D371C80142F473E63960E1D7C15AB1D27E36EB00420A6C15A62556EDC18C0304CC5106F7418CD884C47B37E641010200004003000000CA1A76873817E5C1708C77B66DE2C04120104287188FC0C168EC3415D2E1FBC1D461B92555A6E141CEF9BF09B0E8E2C1CE21A88612D2E4C1C090105CD198AFC11898BDF70468004201020000400500000014D2036A6F7C004254F842449E2CEE41C2AD269722B901C24C06FAAAC56EEF417E7DDB93BD40F1412AFC7D9EED3A024254FDB187F553F741588498800ABD004204F5AD9CD2E7E54186ADC56E1832F5418E6A815905230142DE6C0720FB5202C24C27C8A44919DBC1EAAAADD423C7F341E4C82009A961E0C10102000040020000008A0E5D9F090C0042408609376919A241BA21E5693E84FAC1F8B2923EB5A300C2404E8CC580EBA74143BDDF5BC3D8FEC1010200004003000000A02FC310B9D6D141807688761457CAC15E52BFD18F97F341F2885F99BCC3F041D230904FA56BE4C100D45B134A3F79C1E2293488F3EBF8413638726BFECFF5C140F184D0C7CAE6C10102000040040000000831BEB3C319F241CE3A86C7E78600C2427F256E3BCA01429EF8DB169DFFF741DA60D8E98C23F7C16A28BCF97F4DF041F897AFB43BBAD6416E13B82AB152F741DE98512331F1F0418CE927CE9D44F2C1B89FF6CE469AEF4150EEB72DDBC3C741':::GEOMETRY,e'\x15\n'::NAME:::NAME,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID), Expected Errors: , Potential Errors: ]: ERROR: relation "view4621" (1097): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ';\\n\" []] :[]\n===========================Previous statements [QUERY: CREATE VIEW schema4279.view4621 AS SELECT public.table3819.col3819_3820 FROM public.table3819, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema3806.table4023 (col4023_4024,col4023_4025,col4023_4026,col4023_4027,col4023_4028,col4023_4029,col4023_4030,col4023_4031,col4023_4032,col4023_4033,col4023_4034,col4023_4035,col4023_4036,col4023_4037,col4023_4038,col4023_4039) VALUES (e'{\".[~^Z\u003e@\": {\"\u0026OgF\": {}, \"Q$u.3x7aP\": [[false]]}, \"\\\\\\\\DCR\": {}, \"]sE\": false, \"foobar\": \"@k.k\"}':::JSONB,1042:::OID,'0106000020E6100000040000000103000000010000000600000054053FFDFD0F57C0243B76AA55B93040F8FEBAE31C3F5CC0806E09C16A62F13F6416EC6874DE5540586DA8BE17403EC0EA1D6CEF558564400438EE5B5A503040A63A941E762866C00AAA73048767564054053FFDFD0F57C0243B76AA55B9304001030000000100000008000000E0CFD1B0A55B15C0EC12DB330EE447C0407B463897DD2540D8FE570CC8774EC0C0B81941F0B052403E7FF3603C5654C01844CAA840A259404030918D73F70CC02C5C0145654044408A8DEE43F07A5640CF48E637DF9B5DC0C883606B7B102A407CC8DB9E902756C0808583E9BCC110C0E0CFD1B0A55B15C0EC12DB330EE447C001030000000100000004000000CADCEBA586FA52407847F712903E3E404EAFF6EED4BE614050685AC18E734340002071AFE9F6B13FCC6091C9A8E94440CADCEBA586FA52407847F712903E3E40010300000001000000070000001C673E15F5BC624095999192CC0D4BC082FF2EFB07425040A0E0EC28E66703C0E0F839D18A4E4F4004E0694DCA765440A06BC849CD4232C0FC3EE48710D943409F97535B56EB61C02C3A72CEAB4D4440D22CB31936C05EC0E8B671979D6935401C673E15F5BC624095999192CC0D4BC0':::GEOGRAPHY,19:::OID,'\u003c}'::NAME:::NAME,'00:15:28.867825+01:24:00':::TIMETZ,'22e28527-dfa8-4f55-b7c6-cd68d7cb4ae1':::UUID,1.0:::FLOAT8,(-1472267120):::INT8,'\\xac312686ee73':::BYTES,23:::OID,4096:::OID,e',}\\x10\\x03'::NAME:::NAME,'0101000040000000000000F87F000000000000F87F000000000000F87F':::GEOMETRY,''::NAME:::NAME,'77497147-cd0b-4e3d-927c-be77ca0ed4d0':::UUID),(e'{\"\u0026Z{uK\u003c\": false, \"UO:\": {\"Um]\": [], \"b\": false}, \"a\": true, \"eZ,7me\\\\\\\\g\\\\\\\\Co\": {}, \"foobar\": {}}':::JSONB,20:::OID,'01040000E0E61000000100000001010000C00092F67F43C2544098BABB18CCDE504034B6902B185EFD4125A97250739601C2':::GEOGRAPHY,17:::OID,'\u003e``Y$N)'::NAME:::NAME,'02:55:31.380615-02:17:00':::TIMETZ,'3e9947aa-402a-49e9-b933-db8f1561669c':::UUID,1.9799622877237404:::FLOAT8,(-128):::INT8,'\\x58':::BYTES,1042:::OID,4096:::OID,e'\\x01'::NAME:::NAME,'0105000040070000000102000040030000006270D178D33AF6C14F92DFBBEA3DF4C1AA7D6E029A6202C26AB30498785302429763A213F08600C2B6AE94F99CF2E1C136E4AA39F0AAE6C17C3328993907E9410A139D5A9725F641010200004003000000B4EE6175D9E0DFC162F658B9096DFBC1D085D419092CCC412AFFB1D371C80142F473E63960E1D7C15AB1D27E36EB00420A6C15A62556EDC18C0304CC5106F7418CD884C47B37E641010200004003000000CA1A76873817E5C1708C77B66DE2C04120104287188FC0C168EC3415D2E1FBC1D461B92555A6E141CEF9BF09B0E8E2C1CE21A88612D2E4C1C090105CD198AFC11898BDF70468004201020000400500000014D2036A6F7C004254F842449E2CEE41C2AD269722B901C24C06FAAAC56EEF417E7DDB93BD40F1412AFC7D9EED3A024254FDB187F553F741588498800ABD004204F5AD9CD2E7E54186ADC56E1832F5418E6A815905230142DE6C0720FB5202C24C27C8A44919DBC1EAAAADD423C7F341E4C82009A961E0C10102000040020000008A0E5D9F090C0042408609376919A241BA21E5693E84FAC1F8B2923EB5A300C2404E8CC580EBA74143BDDF5BC3D8FEC1010200004003000000A02FC310B9D6D141807688761457CAC15E52BFD18F97F341F2885F99BCC3F041D230904FA56BE4C100D45B134A3F79C1E2293488F3EBF8413638726BFECFF5C140F184D0C7CAE6C10102000040040000000831BEB3C319F241CE3A86C7E78600C2427F256E3BCA01429EF8DB169DFFF741DA60D8E98C23F7C16A28BCF97F4DF041F897AFB43BBAD6416E13B82AB152F741DE98512331F1F0418CE927CE9D44F2C1B89FF6CE469AEF4150EEB72DDBC3C741':::GEOMETRY,e'\\x15\\n'::NAME:::NAME,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID), Expected Errors: , Potential Errors: ]: ERROR: relation \"view4621\" (1097): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:11:41.623821",
          |  "ops": null,
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "WATCH DOG: failed to get session information: timeout: context canceled\n"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ cb55144cdec54d2a70f074ad64b4eca5e6c6891a:

          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:56:34.607045",
          |  "ops": [
          |   "BEGIN",
          |   "QUERY: ALTER TABLE schema1261.table2115 DROP CONSTRAINT IrrelevantConstraintName, Expected Errors: 42P01, Potential Errors: "
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error.\n Dumping state before death:\nExpected errors: 42P01\nPotential errors: \nExpected commit errors: \nPotential commit errors: \n===========================\nExecuted queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema1261\" \"table2115\"]] :false\n\n===========================\nPrevious statements [QUERY: ALTER TABLE schema1261.table2115 DROP CONSTRAINT IrrelevantConstraintName, Expected Errors: 42P01, Potential Errors: ]: ERROR: relation \"schema1261.table2115\" does not exist (SQLSTATE 42P01)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:56:34.650878",
          |  "ops": [
          |   "BEGIN",
          |   "QUERY: SELECT * FROM public.table2114 AS t0  FETCH FIRST 1 ROWS ONLY, Expected Errors: 42P01, Potential Errors: 53200"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error.\n Dumping state before death:\nExpected errors: 42P01\nPotential errors: 53200\nExpected commit errors: \nPotential commit errors: \n===========================\nExecuted queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table2114\"]] :false\n\n===========================\nPrevious statements [QUERY: SELECT * FROM public.table2114 AS t0  FETCH FIRST 1 ROWS ONLY, Expected Errors: 42P01, Potential Errors: 53200]: ERROR: relation \"public.table2114\" does not exist (SQLSTATE 42P01)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:56:32.835488",
          |  "ops": [
          |   "BEGIN",
          |   "QUERY: SELECT t0.col1179_1183 AS col0,t0.col1179_1182 AS col1 FROM schema1261.table1437 AS t0  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200",
          |   "QUERY: SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs(), Expected Errors: , Potential Errors: ",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": ""
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:56:40.953557",
          |  "ops": null,
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "WATCH DOG: failed to get session information: can't scan into dest[1]: cannot scan null into *string\n"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ aaf50e920ceff3c2863ab96b9e3614b8434b70a8:

          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc000fbd6b8 sp=0xc000fbd698 pc=0x465216
          | runtime.chanrecv(0xc0002b7380, 0x0, 0x1)
          |     GOROOT/src/runtime/chan.go:577 +0x56c fp=0xc000fbd748 sp=0xc000fbd6b8 pc=0x430b4c
          | runtime.chanrecv1(0xc000fbd7b0?, 0xc000fbd7a0?)
          |     GOROOT/src/runtime/chan.go:440 +0x18 fp=0xc000fbd770 sp=0xc000fbd748 pc=0x430578
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:51 +0x98 fp=0xc000fbd7e0 sp=0xc000fbd770 pc=0xd2e778
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000fbd7e8 sp=0xc000fbd7e0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          | panic: dumping stacks, we failed to terminate threads on time.
          |
          | goroutine 141360 [running]:
          | panic({0x38cfc00, 0x54a4830})
          |     GOROOT/src/runtime/panic.go:941 +0x397 fp=0xc0018a2708 sp=0xc0018a2648 pc=0x4622d7
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002772140, {0x54d92a8, 0xc000eaf280})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:96 +0x17b fp=0xc0018a27b8 sp=0xc0018a2708 pc=0x1aa25fb
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0018a27e0 sp=0xc0018a27b8 pc=0x1aa286e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0018a27e8 sp=0xc0018a27e0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ": {}}, \"c\": [true]}}':::JSONB,'1987-07-21 23:11:33.000112+00:00':::TIMESTAMPTZ,'15:24:09.378513':::TIME,e'+\\x03bmE\u003cg'::STRING,23:::OID,4096:::OID,true),(0.0:::FLOAT8,2206:::OID,'[[], null, [[{\"bar\": {}}, 0.5608818739028938]], {}]':::JSONB,'1970-07-03 04:16:44.000948+00:00':::TIMESTAMPTZ,'08:03:20.266354':::TIME,e'\\x00'::STRING,3802:::OID,90001:::OID,true),((-0.07988600432872772):::FLOAT8,1560:::OID,'null':::JSONB,'1998-03-04 13:54:49.000762+00:00':::TIMESTAMPTZ,'23:30:40.189601':::TIME,e'R\\x03wZ\\x18t'::STRING,90000:::OID,90000:::OID,true), Expected Errors: , Potential Errors: ",
          |   "QUERY: SELECT t0.col2871_2877 AS col0,t0.col2871_2873 AS col1,t0.col2871_2877 AS col2,t0.col2871_2877 AS col3,t0.col2871_2874 AS col4,t0.col2871_2875 AS col5 FROM public.table2871 AS t0  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table2871\"]] :true\n===========================Previous statements [QUERY: SELECT t1.col2775_2776 AS col0,t1.col2775_2782 AS col1,t1.col2775_2781 AS col2,t0.col3134_3147 AS col3,t1.col2775_2777 AS col4,t0.col3134_3141 AS col5,t0.col3134_3142 AS col6,t0.col3134_3144 AS col7 FROM public.table3134 AS t0 ,public.table2775 AS t1  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO public.table3134 (col3134_3139,col3134_3140,col3134_3141,col3134_3142,col3134_3143,col3134_3144,col3134_3145,col3134_3146,col3134_3147) VALUES ((-1.0805723667144775):::FLOAT8,1560:::OID,'[null, {\"b\": \"}f#\u0026Kw1\", \"bar\": {}}, [\"c\"], {}]':::JSONB,'2006-10-18 22:48:00.000906+00:00':::TIMESTAMPTZ,'05:05:12.897255':::TIME,e'Q\\x13\\x1a)\\x03A'::STRING,869:::OID,1700:::OID,false),(0.7764872312545776:::FLOAT8,4089:::OID,e'[[\"|*tdy0 ^\", null, [[]], []], {\"Qr/beE\\\\\\\\N4[c.\": 0.33585837229212223}]':::JSONB,'2026-08-08 20:24:11.000655+00:00':::TIMESTAMPTZ,'07:59:00.19395':::TIME,e'qH\\x1a'::STRING,2205:::OID,1002:::OID,false), Expected Errors: , Potential Errors:  QUERY: CREATE SEQUENCE IF NOT EXISTS public.seq3506, Expected Errors: , Potential Errors:  QUERY: INSERT INTO public.table3134 (col3134_3139,col3134_3140,col3134_3141,col3134_3142,col3134_3143,col3134_3144,col3134_3145,col3134_3146,col3134_3147) VALUES (0.6819256544113159:::FLOAT8,1186:::OID,e'{\"$AQTe^p\": \"a\", \",QaAMoV5(\\\\\"_\\\\\"\": {\"O/}a#!OE8^\": true}, \"Cm1NR3D\u003cE\": {}, \"bar\": [true], \"z;B\": {\"R#5\\\\\\\\\": {\"b\": {}}, \"c\": [true]}}':::JSONB,'1987-07-21 23:11:33.000112+00:00':::TIMESTAMPTZ,'15:24:09.378513':::TIME,e'+\\x03bmE\u003cg'::STRING,23:::OID,4096:::OID,true),(0.0:::FLOAT8,2206:::OID,'[[], null, [[{\"bar\": {}}, 0.5608818739028938]], {}]':::JSONB,'1970-07-03 04:16:44.000948+00:00':::TIMESTAMPTZ,'08:03:20.266354':::TIME,e'\\x00'::STRING,3802:::OID,90001:::OID,true),((-0.07988600432872772):::FLOAT8,1560:::OID,'null':::JSONB,'1998-03-04 13:54:49.000762+00:00':::TIMESTAMPTZ,'23:30:40.189601':::TIME,e'R\\x03wZ\\x18t'::STRING,90000:::OID,90000:::OID,true), Expected Errors: , Potential Errors:  QUERY: SELECT t0.col2871_2877 AS col0,t0.col2871_2873 AS col1,t0.col2871_2877 AS col2,t0.col2871_2877 AS col3,t0.col2871_2874 AS col4,t0.col2871_2875 AS col5 FROM public.table2871 AS t0  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200]: ERROR: relation \"seq3506\" (845): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:54:00.516975",
          |  "ops": null,
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "WATCH DOG: failed to get session information: can't scan into dest[1]: cannot scan null into *string\n"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 80c274877a917580af62be6eb0cd48c8c7ae9c08:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_134520.567843359_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1972
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:896
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_134520.567843359_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | (-10877):::INT8,1.2684699710001377:::FLOAT8,(-402293.1808395257648):::DECIMAL,1266:::OID,'14:26:06.770276-05:40:00':::TIMETZ,e'[{"c": {}}, {"foobar": []}, ["baz"], {"baz": [], "v>mW#i v": {"c": false, "foobar": []}}, {";``Mq#6G! ": {}}, "#:+;\\\\", []]':::JSONB,(-2.996240066590932290):::DECIMAL,'1988-03-05':::DATE,'01030000A0E6100000010000000A0000004FAEEDA4923A63C0182783C08F2234C0246A3EE18A02FA41DD67628523ED5DC0440EE0E0EA2E37C0B24DE25B75A6FA4197874654886264C01300DA208D6B51C0686B80807BD7FE41C86A4BFC7F436140AE97BF720ABA51C08E6C992809A3FDC138D5DEBFDE604740B056C6B41ECF10C03028B4D9214CD7C1B03007DE6D685740D0B9FE55066823401CE420D1D08EFE41CC2EB8694BD148C030659A42434A5640444C6043316101429BBEFF43B9D057C0C45A009E3A61524024DB9C3070A0FC41EAD04EC0595B5AC0087CF88F5C9A4940D81B68B917F6E3C14FAEEDA4923A63C0182783C08F2234C0246A3EE18A02FA41':::GEOGRAPHY,'1989-10-23':::DATE,'0103000020E610000000000000':::GEOGRAPHY),('01050000C00600000001020000C0040000008459EC4D7CF2FD414056F4C0BCDAC74116A596F9DBBDF7418AEED72FAA8801C2806575F51E7BDC41ACA01EC6AF89EF4100B79B2287C2F641CCC240A4AC18F141D005C38653C8FB412E0EDB01057B014256C74BB796CAEAC15460E77FFE6AF641F0C9432FE1E500C2DA8E3036CD810242448BF0807068EDC15ABDF3BF026EF9C101020000C0020000002E2AD37BBB700042E254954F5688F74124C55AF6C87DF241289F89BB577DE54110CE22CC9212F0C140937AA29079A4C1084B4C046A1300420EC1FA7F24F1E2C101020000C005000000D5C437F7F46F01C270237E09C715F6C1068DFECA39C7F7C14C928C92036FEE418EF9DF93C420E7C1D9CAD4A313F9F4C194C78D6081CBEEC1BC33341BBF53FE41982D2D3966CEE24181FA460BFB3901C27A9440382E1AE2C1C0402D980C07C6C128ADB3521585FC4154C9900F6870FBC138DFE11154CDFA41739378B83142F3C1B8C50E62EA8EFC41E8B79D07BA39E9C180B55A9BC196A241CE585521FF49F34101020000C003000000F491D759A845E641462C6A67FFA0FBC1F83A1B4B9517FEC1A4D9FD0903C6FD418412CCBC6FEDE841BE87D60DA788F9C1AC399AB5EE60E1410570465FC6A6FFC14EC119A77D8A0042BE0D49C28B11ECC120167B705D8DC6C14825CF6BAD5BFCC101020000C006000000DA0D00B11957F041B03E43769A80D1C1F809CB520C19E0418062FA14E8F5C541406FCCAC5C8DA9419C549FC03749E64120C1CA4051DDB341B2EEA0A71125E2C100BB10CB850CA1413CAF52B26ECAF641B7510A8771ACFCC180FB88921F409C415055756CCB26D2C1B09B31320151EB41E0CA224B1F4EB1C1002284C2C6D0B24182F4BD61CF5102C2501C026D8A6C0242C87369D11E7EF3C1307FA2443C7B0042870F103A915E02C2745D13A4A13E0242F209BBD2DD28ECC1B8C5549D8CA9FD4101020000C00B000000AC933FF61E59F6C1A437393EEDDADFC1E8B6212046C7EDC1F446DBE7607AF6419C7EDD5D07C301C2CD638B7D23BAF1C1B0578F081902C8C1D6E30A9646F4E3C1C41D35A47AE5E1C1FCB79061E82BE8C16C7A9E5A6481EBC184E4D54104DAF441E40ACE1DDF9EE541EAB8194A0FAEFBC180632AA7A8D1AD4182315731243DF3C1284CE735622E0042C0F0908D2951CFC11482DEA1915CEEC100D4881413C77641CC7A2208E47D0042A01E0037F5ACCEC102657C93352EFCC1667C536B7ACBF04108ADAAE8F9F2FA41D0B3B795E2D1D141681E3CBD7D18E8C10640A6E25EBAF6415C9691B0E94CF7416CA5F5C9A576F5417CB0E00CCC8BE1413C3DF06B703400424EA157F166BEF44142FBACF73D89F7410A6F7EECF6B9F841648514188FB40042C4C03E404376E9416C181B0826B4F0410821479BA419FB414A36EFD6F243F64180E39FA31E06B941F43B64146B62FB4158000DB5EB1C01C23067D3999008C8C1':::GEOMETRY,28589:::INT8,0.6268768194497791:::FLOAT8,5929460656268.042986:::DECIMAL,1001:::OID,'18:25:52.8747+05:36:00':::TIMETZ,e'{"6;|": null, "7GZ": {"+~%~SLQ|iQC1": {"ICn+F\\"$0": " 77]APx", "baz": null}}, "Oy}\'~ob]": {}, "bar": [], "s@An^c?fOw": 0.47427890319866634, "z\\\\%=u": 1.3940538084908451}':::JSONB,6.468408065451078184E+22:::DECIMAL,'1984-11-16':::DATE,'0102000020E6100000030000006ADA4A8FB3125FC0E21DFA1E25583CC0E02425A4990C4140192DD08C7C7F48C008B266AA1C7650402066A54E642C42C0':::GEOGRAPHY,'1972-05-11':::DATE,'0104000060E61000000700000001010000409CC537E347085940CABEF45DAF1452C0F45C018B3EB6F3C1010100004027F47FD3B44C66C0B25FDAA26C135640F3AAFB9C30DFF2C101010000405E9838265A9361408C4621FB0A3D37C0D5ACA0BAF682FEC101010000408885F0F554903F4070FC39076F2838401A0411C41C5201C20101000040383767969D2B3E40FCB3FCE5142A53408A2FC17BCB38E7C10101000040C43B0CB918F35E40D206612F45A1554088FD5BC70F73FBC10101000040BEE2415BFC4866409837B3F245274FC0A0E1F2626F82E8C1':::GEOGRAPHY), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 95257648):::DECIMAL,1266:::OID,'14:26:06.770276-05:40:00':::TIMETZ,e'[{\"c\": {}}, {\"foobar\": []}, [\"baz\"], {\"baz\": [], \"v\u003emW#i v\": {\"c\": false, \"foobar\": []}}, {\";``Mq#6G! \": {}}, \"#:+;\\\\\\\\\", []]':::JSONB,(-2.996240066590932290):::DECIMAL,'1988-03-05':::DATE,'01030000A0E6100000010000000A0000004FAEEDA4923A63C0182783C08F2234C0246A3EE18A02FA41DD67628523ED5DC0440EE0E0EA2E37C0B24DE25B75A6FA4197874654886264C01300DA208D6B51C0686B80807BD7FE41C86A4BFC7F436140AE97BF720ABA51C08E6C992809A3FDC138D5DEBFDE604740B056C6B41ECF10C03028B4D9214CD7C1B03007DE6D685740D0B9FE55066823401CE420D1D08EFE41CC2EB8694BD148C030659A42434A5640444C6043316101429BBEFF43B9D057C0C45A009E3A61524024DB9C3070A0FC41EAD04EC0595B5AC0087CF88F5C9A4940D81B68B917F6E3C14FAEEDA4923A63C0182783C08F2234C0246A3EE18A02FA41':::GEOGRAPHY,'1989-10-23':::DATE,'0103000020E610000000000000':::GEOGRAPHY),('01050000C00600000001020000C0040000008459EC4D7CF2FD414056F4C0BCDAC74116A596F9DBBDF7418AEED72FAA8801C2806575F51E7BDC41ACA01EC6AF89EF4100B79B2287C2F641CCC240A4AC18F141D005C38653C8FB412E0EDB01057B014256C74BB796CAEAC15460E77FFE6AF641F0C9432FE1E500C2DA8E3036CD810242448BF0807068EDC15ABDF3BF026EF9C101020000C0020000002E2AD37BBB700042E254954F5688F74124C55AF6C87DF241289F89BB577DE54110CE22CC9212F0C140937AA29079A4C1084B4C046A1300420EC1FA7F24F1E2C101020000C005000000D5C437F7F46F01C270237E09C715F6C1068DFECA39C7F7C14C928C92036FEE418EF9DF93C420E7C1D9CAD4A313F9F4C194C78D6081CBEEC1BC33341BBF53FE41982D2D3966CEE24181FA460BFB3901C27A9440382E1AE2C1C0402D980C07C6C128ADB3521585FC4154C9900F6870FBC138DFE11154CDFA41739378B83142F3C1B8C50E62EA8EFC41E8B79D07BA39E9C180B55A9BC196A241CE585521FF49F34101020000C003000000F491D759A845E641462C6A67FFA0FBC1F83A1B4B9517FEC1A4D9FD0903C6FD418412CCBC6FEDE841BE87D60DA788F9C1AC399AB5EE60E1410570465FC6A6FFC14EC119A77D8A0042BE0D49C28B11ECC120167B705D8DC6C14825CF6BAD5BFCC101020000C006000000DA0D00B11957F041B03E43769A80D1C1F809CB520C19E0418062FA14E8F5C541406FCCAC5C8DA9419C549FC03749E64120C1CA4051DDB341B2EEA0A71125E2C100BB10CB850CA1413CAF52B26ECAF641B7510A8771ACFCC180FB88921F409C415055756CCB26D2C1B09B31320151EB41E0CA224B1F4EB1C1002284C2C6D0B24182F4BD61CF5102C2501C026D8A6C0242C87369D11E7EF3C1307FA2443C7B0042870F103A915E02C2745D13A4A13E0242F209BBD2DD28ECC1B8C5549D8CA9FD4101020000C00B000000AC933FF61E59F6C1A437393EEDDADFC1E8B6212046C7EDC1F446DBE7607AF6419C7EDD5D07C301C2CD638B7D23BAF1C1B0578F081902C8C1D6E30A9646F4E3C1C41D35A47AE5E1C1FCB79061E82BE8C16C7A9E5A6481EBC184E4D54104DAF441E40ACE1DDF9EE541EAB8194A0FAEFBC180632AA7A8D1AD4182315731243DF3C1284CE735622E0042C0F0908D2951CFC11482DEA1915CEEC100D4881413C77641CC7A2208E47D0042A01E0037F5ACCEC102657C93352EFCC1667C536B7ACBF04108ADAAE8F9F2FA41D0B3B795E2D1D141681E3CBD7D18E8C10640A6E25EBAF6415C9691B0E94CF7416CA5F5C9A576F5417CB0E00CCC8BE1413C3DF06B703400424EA157F166BEF44142FBACF73D89F7410A6F7EECF6B9F841648514188FB40042C4C03E404376E9416C181B0826B4F0410821479BA419FB414A36EFD6F243F64180E39FA31E06B941F43B64146B62FB4158000DB5EB1C01C23067D3999008C8C1':::GEOMETRY,28589:::INT8,0.6268768194497791:::FLOAT8,5929460656268.042986:::DECIMAL,1001:::OID,'18:25:52.8747+05:36:00':::TIMETZ,e'{\"6;|\": null, \"7GZ\": {\"+~%~SLQ|iQC1\": {\"ICn+F\\\\\"$0\": \" 77]APx\", \"baz\": null}}, \"Oy}\\'~ob]\": {}, \"bar\": [], \"s@An^c?fOw\": 0.47427890319866634, \"z\\\\\\\\%=u\": 1.3940538084908451}':::JSONB,6.468408065451078184E+22:::DECIMAL,'1984-11-16':::DATE,'0102000020E6100000030000006ADA4A8FB3125FC0E21DFA1E25583CC0E02425A4990C4140192DD08C7C7F48C008B266AA1C7650402066A54E642C42C0':::GEOGRAPHY,'1972-05-11':::DATE,'0104000060E61000000700000001010000409CC537E347085940CABEF45DAF1452C0F45C018B3EB6F3C1010100004027F47FD3B44C66C0B25FDAA26C135640F3AAFB9C30DFF2C101010000405E9838265A9361408C4621FB0A3D37C0D5ACA0BAF682FEC101010000408885F0F554903F4070FC39076F2838401A0411C41C5201C20101000040383767969D2B3E40FCB3FCE5142A53408A2FC17BCB38E7C10101000040C43B0CB918F35E40D206612F45A1554088FD5BC70F73FBC10101000040BEE2415BFC4866409837B3F245274FC0A0E1F2626F82E8C1':::GEOGRAPHY), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 770ff3c545a51752490403da64d56fb397f49c5e:

          | runtime.gopark(0xc000f16f78?, 0x3?, 0x0?, 0x30?, 0xc000f16f4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc000f16dc8 sp=0xc000f16da8 pc=0x465216
          | runtime.selectgo(0xc000f16f78, 0xc000f16f44, 0xc000e41300?, 0x0, 0x3f65060?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc000f16f08 sp=0xc000f16dc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc001519310, {0x54f75f8, 0xc000e41300})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000f16fb8 sp=0xc000f16f08 pc=0x1aa614e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000f16fe0 sp=0xc000f16fb8 pc=0x1aa644e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000f16fe8 sp=0xc000f16fe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 398738 [runnable]:
          | runtime.gopark(0xc000f19f78?, 0x3?, 0x0?, 0x30?, 0xc000f19f4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc000f19dc8 sp=0xc000f19da8 pc=0x465216
          | runtime.selectgo(0xc000f19f78, 0xc000f19f44, 0xc000e41300?, 0x0, 0x3f65060?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc000f19f08 sp=0xc000f19dc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002f28230, {0x54f75f8, 0xc000e41300})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000f19fb8 sp=0xc000f19f08 pc=0x1aa614e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000f19fe0 sp=0xc000f19fb8 pc=0x1aa644e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000f19fe8 sp=0xc000f19fe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 399868 [runnable]:
          | runtime.gopark(0xc0010d47b0?, 0x2?, 0xc7?, 0x52?, 0xc0010d47a4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0010d4630 sp=0xc0010d4610 pc=0x465216
          | runtime.selectgo(0xc0010d47b0, 0xc0010d47a0, 0xc001b5ec40?, 0x0, 0xc00123d9e0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0010d4770 sp=0xc0010d4630 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0010d47e0 sp=0xc0010d4770 pc=0xd2e74c
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0010d47e8 sp=0xc0010d47e0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | '16:10:18.606199':::TIME,1700:::OID,2205:::OID)) AS t(col5054_5062,col5054_5065,col5054_5067,col5054_5056,col5054_5059,col5054_5060,col5054_5061,col5054_5066,col5054_5057,col5054_5058,col5054_5063,col5054_5064);\" []] :[[\"1.8677302310212767\"]]\nQUERY [\"SELECT array[(CASE WHEN col5054_5063 IS NULL THEN e'\\\\nc':::STRING ELSE NULL END)::STRING::STRING] AS c FROM ( VALUES(1700:::OID,2205:::OID,'2001-09-15 14:42:03.000816':::TIMESTAMP,1.3821410948367396:::FLOAT8,'16:10:18.606199':::TIME,(-4286092797334177307):::INT8,B'110100110011000','\\\\x':::BYTES,'55.192.182.72/17':::INET,'BOX(-0.7117974256635042 1.0600434656782092,2.086721901426354 1.062387683958085)':::BOX2D,(-0.343835946211582):::FLOAT8,0.48558913618453714:::FLOAT8)) AS t(col5054_5063,col5054_5064,col5054_5066,col5054_5057,col5054_5058,col5054_5060,col5054_5061,col5054_5062,col5054_5065,col5054_5067,col5054_5056,col5054_5059);\" []] :[[\"NULL\"]]\nQUERY [\"SELECT array[((col5054_5059 + col5054_5056) + col5054_5057)::FLOAT8::STRING] AS c FROM ( VALUES(0.48558913618453714:::FLOAT8,(-4286092797334177307):::INT8,B'110100110011000','\\\\x':::BYTES,'55.192.182.72/17':::INET,'BOX(-0.7117974256635042 1.0600434656782092,2.086721901426354 1.062387683958085)':::BOX2D,(-0.343835946211582):::FLOAT8,'16:10:18.606199':::TIME,1700:::OID,2205:::OID,'2001-09-15 14:42:03.000816':::TIMESTAMP,1.3821410948367396:::FLOAT8)) AS t(col5054_5059,col5054_5060,col5054_5061,col5054_5062,col5054_5065,col5054_5067,col5054_5056,col5054_5058,col5054_5063,col5054_5064,col5054_5066,col5054_5057);\" []] :[[\"1.5238942848096948\"]]\nQUERY [\"SELECT (((col5054_5074, col5054_5071, col5054_5068, col5054_5069, col5054_5059, col5054_5070))::STRING) IS NULL AS c FROM ( VALUES(1.3821410948367396:::FLOAT8,'16:10:18.606199':::TIME,1700:::OID,2205:::OID,'2001-09-15 14:42:03.000816':::TIMESTAMP,(-0.343835946211582):::FLOAT8,0.48558913618453714:::FLOAT8,(-4286092797334177307):::INT8,B'110100110011000','\\\\x':::BYTES,'55.192.182.72/17':::INET,'BOX(-0.7117974256635042 1.0600434656782092,2.086721901426354 1.062387683958085)':::BOX2D,'box(-0.7117974256635042 1.0600434656782092,2.086721901426354 1.062387683958085)':::STRING,'1.5238942848096948'::FLOAT8,'box(-0.7117974256635042 1.0600434656782092,2.086721901426354 1.062387683958085)':::STRING,'-5997129906165863374'::INT8,'1.8677302310212767'::FLOAT8,NULL,'1.5238942848096948'::FLOAT8)) AS t(col5054_5057,col5054_5058,col5054_5063,col5054_5064,col5054_5066,col5054_5056,col5054_5059,col5054_5060,col5054_5061,col5054_5062,col5054_5065,col5054_5067,col5054_5068,col5054_5069,col5054_5070,col5054_5071,col5054_5072,col5054_5073,col5054_5074);\" []] :false\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema3357.table5054 (col5054_5056,col5054_5057,col5054_5058,col5054_5059,col5054_5060,col5054_5061,col5054_5062,col5054_5063,col5054_5064,col5054_5065,col5054_5066,col5054_5067) VALUES (0.23803262181253587:::FLOAT8,(-0.4468306085109419):::FLOAT8,'07:01:45.647287':::TIME,0.9752059130835049:::FLOAT8,(-9223372036854775807):::INT8,B'001001011100011','\\x9de4c1a727a0c8':::BYTES,1083:::OID,90000:::OID,'44.207.224.159/7':::INET,'1986-07-27 15:30:18.000354':::TIMESTAMP,'BOX(-0.04043351191814204 0.07679549114655093,0.5533745864917653 0.23625116001236013)':::BOX2D),((-0.4573080470389305):::FLOAT8,(-0.006337058443789956):::FLOAT8,'16:46:33.164567':::TIME,0.32640360100083265:::FLOAT8,6895203054792436359:::INT8,B'110110101010111','\\x78ab24caaf':::BYTES,1184:::OID,2205:::OID,'17e8:39a1:799c:e7dd:f8d9:a356:f0f5:b5c2/102':::INET,'2020-09-27 10:39:24.000355':::TIMESTAMP,'BOX(-10 -10,10 10)':::BOX2D),((-0.343835946211582):::FLOAT8,1.3821410948367396:::FLOAT8,'16:10:18.606199':::TIME,0.48558913618453714:::FLOAT8,(-4286092797334177307):::INT8,B'110100110011000','\\x':::BYTES,1700:::OID,2205:::OID,'55.192.182.72/17':::INET,'2001-09-15 14:42:03.000816':::TIMESTAMP,'BOX(-0.7117974256635042 1.0600434656782092,2.086721901426354 1.062387683958085)':::BOX2D), Expected Errors: 22003,22P01,23502, Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ f59620ec646d1181d358d0dc41ab60815ecf59c9:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:897: output in run_142724.345168039_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1981
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:897
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1571
        Wraps: (2) output in run_142724.345168039_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 0987 sql:SELECT sum(num_retries) + sum(num_auto_retries) FROM crdb_internal.cluster_transactions WHERE id=$1]
          | I220828 14:39:32.873901 220144 workload/pgx_helpers.go:79  [-] 1866  pgx logger [error]: Query logParams=map[args:[170f8879fef0a36e0000000000000001] err:can't scan into dest[1]: cannot scan null into *string pid:1853638 sql:SELECT active_queries, kv_txn FROM crdb_internal.cluster_sessions WHERE session_id = $1]
          | I220828 14:39:32.918455 221 workload/pgx_helpers.go:79  [-] 1867  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: transaction committed but schema change aborted with error: (XXUUU): job 791763898982465537 could not be loaded: job with ID 791763898982465537 does not exist (SQLSTATE XXA00) pid:1109351 sql:commit time:1.311683227s]
          | Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:
          | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: ===========================Previous statements [QUERY: INSERT INTO schema1724.table2880 (col2089_2099,col2089_2097,col2089_2098) VALUES (NULL,1266:::OID,NULL),(NULL,2206:::OID,'1981-08-13 14:05:51.000334':::TIMESTAMP), Expected Errors: , Potential Errors:  QUERY: DROP SCHEMA "schema883" CASCADE, Expected Errors: , Potential Errors:  QUERY: SELECT t1.col3108_3111 AS col0,t2.col3108_3111 AS col1,t2.col3108_3112 AS col2,t2.col3108_3111 AS col3,t2.col3108_3112 AS col4 FROM schema1036.table3085 AS t0 ,schema2878.table3108 AS t1 ,schema2878.table3108 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO schema960.table3222 (col3222_3223,col3222_3224,col3222_3225,col3222_3226,col3222_3227,col3222_3228,col3222_3229,col3222_3230,col3222_3231,col3222_3232,col3222_3233,col3222_3234) VALUES (90004:::OID,869:::OID,'[{"baz": true}, false, [0.9247392964302076, {"+p)?ME#Igb``F": {}}]]':::JSONB,NULL,18:::OID,e'[[{}], [{"foo": [{"UP\\"<i3sF]": [], "\\\\)QRnj": "hX6``~$b{", "c": 2.1823215525789683}, "baz"], "lAL\\\\<,>{": "{iP%"}], [[], {}], false]':::JSONB,'e47a970c-e168-4337-a1f1-297363a38ccc':::UUID,'1984-01-19':::DATE,ARRAY[]:::BOX2D[],e'6\x01'::VARCHAR,(-10021):::INT8,3.4028234663852886e+38:::FLOAT8),(4096:::OID,90002:::OID,e'{"# %G?/ZU": {",)03p\\\\": {}}, "P+C": [{"\\"Hk-J": {}}, null, [true], 0.18354290333985984, null], "cy&rR": false, "r)J9B,6": []}':::JSONB,'1992-07-25':::DATE,1186:::OID,'[[[{}], {"baz": [], "foobar": [], "y>7sw~w": {}}, [{"a": [null]}, {}], []], {}, 0.6018873870111852]':::JSONB,'362ca0bc-871b-4058-96b2-b4dff1915ddc':::UUID,'1985-02-28':::DATE,ARRAY['BOX(-1.0181908860286981 0.18530395215814816,0.5254117218302146 1.034998708167156)':::BOX2D,'BOX(-0.28661207607902983 -0.5389349608185564,-0.015288337666203577 -0.40233266470478224)':::BOX2D,'BOX(-2.0543113996119904 0.7875088956272273,0.916865925285563 0.9893326932918927)':::BOX2D,'BOX(-1.2183186451741432 -0.7138806038744121,0.47261435420629283 -0.03558315795788117)':::BOX2D],'] oRWe,'::VARCHAR,(-22554):::INT8,(-0.2746864855289459):::FLOAT8),(90002:::OID,25:::OID,e'[{"I?uOviQg%)#2": "=5.n\\\\B", "V~u]r14e": {"MNHstWRZ": {"baz": "c", "foo": []}}}, [{"!q>Z": "_~\\\\( 8mU8c"}, []], {"<1[MSm.}": [], "a": []}]':::JSONB,'1994-12-17':::DATE,2205:::OID,e'[{"\'\'?hZZ@": [1.7963067695346675], "<!{$_30$U": [], "baz": null, "j%Xv>M": "b"}, {"mbk?": []}, [[]], false, {}]':::JSONB,'d1bfbad1-54cc-4e0e-b763-f4464294b9f3':::UUID,'1979-10-12':::DATE,ARRAY['BOX(-1.5615589742057119 -1.12256509236807,1.792201970494969 -0.5322828313219645)':::BOX2D,'BOX(0.31251457737123034 1.1537515644843743,1.2750033079224639 1.2541575071703341)':::BOX2D],e'\x18^(!k'::VARCHAR,(-32295):::INT8,0.6518183350563049:::FLOAT8), Expected Errors: , Potential Errors:  QUERY: SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs(), Expected Errors: , Potential Errors: ]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 791763898982465537 could not be loaded: job with ID 791763898982465537 does not exist (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ": []}, [[]], false, {}]':::JSONB,'d1bfbad1-54cc-4e0e-b763-f4464294b9f3':::UUID,'1979-10-12':::DATE,ARRAY['BOX(-1.5615589742057119 -1.12256509236807,1.792201970494969 -0.5322828313219645)':::BOX2D,'BOX(0.31251457737123034 1.1537515644843743,1.2750033079224639 1.2541575071703341)':::BOX2D],e'\\x18^(!k'::VARCHAR,(-32295):::INT8,0.6518183350563049:::FLOAT8), Expected Errors: , Potential Errors: ",
          |   "QUERY: SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs(), Expected Errors: , Potential Errors: ",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: ===========================Previous statements [QUERY: INSERT INTO schema1724.table2880 (col2089_2099,col2089_2097,col2089_2098) VALUES (NULL,1266:::OID,NULL),(NULL,2206:::OID,'1981-08-13 14:05:51.000334':::TIMESTAMP), Expected Errors: , Potential Errors:  QUERY: DROP SCHEMA \"schema883\" CASCADE, Expected Errors: , Potential Errors:  QUERY: SELECT t1.col3108_3111 AS col0,t2.col3108_3111 AS col1,t2.col3108_3112 AS col2,t2.col3108_3111 AS col3,t2.col3108_3112 AS col4 FROM schema1036.table3085 AS t0 ,schema2878.table3108 AS t1 ,schema2878.table3108 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO schema960.table3222 (col3222_3223,col3222_3224,col3222_3225,col3222_3226,col3222_3227,col3222_3228,col3222_3229,col3222_3230,col3222_3231,col3222_3232,col3222_3233,col3222_3234) VALUES (90004:::OID,869:::OID,'[{\"baz\": true}, false, [0.9247392964302076, {\"+p)?ME#Igb``F\": {}}]]':::JSONB,NULL,18:::OID,e'[[{}], [{\"foo\": [{\"UP\\\\\"\u003ci3sF]\": [], \"\\\\\\\\)QRnj\": \"hX6``~$b{\", \"c\": 2.1823215525789683}, \"baz\"], \"lAL\\\\\\\\\u003c,\u003e{\": \"{iP%\"}], [[], {}], false]':::JSONB,'e47a970c-e168-4337-a1f1-297363a38ccc':::UUID,'1984-01-19':::DATE,ARRAY[]:::BOX2D[],e'6\\x01'::VARCHAR,(-10021):::INT8,3.4028234663852886e+38:::FLOAT8),(4096:::OID,90002:::OID,e'{\"# %G?/ZU\": {\",)03p\\\\\\\\\": {}}, \"P+C\": [{\"\\\\\"Hk-J\": {}}, null, [true], 0.18354290333985984, null], \"cy\u0026rR\": false, \"r)J9B,6\": []}':::JSONB,'1992-07-25':::DATE,1186:::OID,'[[[{}], {\"baz\": [], \"foobar\": [], \"y\u003e7sw~w\": {}}, [{\"a\": [null]}, {}], []], {}, 0.6018873870111852]':::JSONB,'362ca0bc-871b-4058-96b2-b4dff1915ddc':::UUID,'1985-02-28':::DATE,ARRAY['BOX(-1.0181908860286981 0.18530395215814816,0.5254117218302146 1.034998708167156)':::BOX2D,'BOX(-0.28661207607902983 -0.5389349608185564,-0.015288337666203577 -0.40233266470478224)':::BOX2D,'BOX(-2.0543113996119904 0.7875088956272273,0.916865925285563 0.9893326932918927)':::BOX2D,'BOX(-1.2183186451741432 -0.7138806038744121,0.47261435420629283 -0.03558315795788117)':::BOX2D],'] oRWe,'::VARCHAR,(-22554):::INT8,(-0.2746864855289459):::FLOAT8),(90002:::OID,25:::OID,e'[{\"I?uOviQg%)#2\": \"=5.n\\\\\\\\B\", \"V~u]r14e\": {\"MNHstWRZ\": {\"baz\": \"c\", \"foo\": []}}}, [{\"!q\u003eZ\": \"_~\\\\\\\\( 8mU8c\"}, []], {\"\u003c1[MSm.}\": [], \"a\": []}]':::JSONB,'1994-12-17':::DATE,2205:::OID,e'[{\"\\'\\'?hZZ@\": [1.7963067695346675], \"\u003c!{$_30$U\": [], \"baz\": null, \"j%Xv\u003eM\": \"b\"}, {\"mbk?\": []}, [[]], false, {}]':::JSONB,'d1bfbad1-54cc-4e0e-b763-f4464294b9f3':::UUID,'1979-10-12':::DATE,ARRAY['BOX(-1.5615589742057119 -1.12256509236807,1.792201970494969 -0.5322828313219645)':::BOX2D,'BOX(0.31251457737123034 1.1537515644843743,1.2750033079224639 1.2541575071703341)':::BOX2D],e'\\x18^(!k'::VARCHAR,(-32295):::INT8,0.6518183350563049:::FLOAT8), Expected Errors: , Potential Errors:  QUERY: SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs(), Expected Errors: , Potential Errors: ]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 791763898982465537 could not be loaded: job with ID 791763898982465537 does not exist (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 3b16435371a43d603d193a1e2b480a23fba3f07a:

          | runtime.selectgo(0xc0000a4f78, 0xc0000a4f44, 0xc001094100?, 0x0, 0x3f65800?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0000a4f08 sp=0xc0000a4dc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002211310, {0x54f7c38, 0xc001094100})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0000a4fb8 sp=0xc0000a4f08 pc=0x1aa65ee
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0000a4fe0 sp=0xc0000a4fb8 pc=0x1aa68ee
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000a4fe8 sp=0xc0000a4fe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 373441 [runnable]:
          | runtime.gopark(0xc0011e4fb0?, 0x2?, 0xf8?, 0x4e?, 0xc0011e4fa4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0011e4e30 sp=0xc0011e4e10 pc=0x465216
          | runtime.selectgo(0xc0011e4fb0, 0xc0011e4fa0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0011e4f70 sp=0xc0011e4e30 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0011e4fe0 sp=0xc0011e4f70 pc=0xd2e74c
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0011e4fe8 sp=0xc0011e4fe0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | C2FB41AC8D24261F76014200D0BE0EBFEEFA4180F722105E4994C188D027A7B3BDD441F4A8FE813D21FF41347F0CD6AECAFD41C89173517689D0410CEBD8A5EE7EED412CAEF2950DBCE041404B28C95C63CDC100F452BE2FA5AB4100B275870AC5C1410157883371BC00C2F6B5572151A7F4C1B2B2E56777B0F1410103000080010000000C0000004AD7BFDA010702C2B4BBE96B2C13FDC1C856A9E0100AF34170A7003A1B67C8C1462933DC38A7FAC130B31632FEC4E0C1C00E558FD2B6A0C17D019DC7B4F5FEC1335FF0ED03CCF2C1E02CEC1F1657DE41F21A41BA869CFEC11096A4E3ABABFE41F884EC071159F141BE489885430AFEC15AA35056FA3FFDC1C073327DEEEFB3C108183A89131AEDC18CCCBA10B7FFE1419CEF902F00B9E241B0B41C113449EBC1FC51150F9673EB4140BFBCB4FAE5A34168967BDFCB80D5C1B0C02A1D7FF4FD417022DE672BC2D8C1E04CE577AA94BDC1340623497F8A02421EB5A658B085E9C1FAB598725368004200EB1FEFFF7597C176B6C3DCE70AF5C1CCEC3ED08FE3E6C1DA8EFF744BB0F1414AD7BFDA010702C2B4BBE96B2C13FDC1C856A9E0100AF3410103000080010000000B000000A0891162391BE2416063B49D442DFDC1B495ABC69E28FF41880006C12E69F7416A4B838C24EBF2C1C8E53C9D350CE34192FEBA47C2ED00424CAB10C1F183F7C1745959909F95E8C1B6654CBAD0EE0042E1DB376FA369F2C1814313B206E801C2641DCD5C0675FC4148B46C776E8EDFC1E82CE454A903ED413AA4F12AD10C004244355C500291F4418E776D0F2315F741A0D270C71CF4B541E00309D14704D341103AC981417FFB415025257439D2CE41A0E6F08671CFB741DB81210662D3FDC1E856A294D8C4EAC1C02C15DEA13BC74100BDF8291056CCC1B0DE94D07E4001C24812963F8030DEC1908E6C617136C441A0891162391BE2416063B49D442DFDC1B495ABC69E28FF410103000080010000000C000000658EAB1A2C6B02C28067A696F75C984192A8053799BCFDC1B2DF19E019C5F6C100ECFFFBDF6277C10A9E8C2D4F57F9C1AC243F067E14F2C1A505F7A05FF401C234497F970987004260C6F1F353F3C641024F3679E231E3C1B83364C2F3F2F6C1A8E17FD3B257FB41400197DCE86EE7C15D21D4C1026FF4C1267CD61903B6F041E05C7796EB4BC04190C2C33C4C04E141A41ECBC1AED1FC418438BE9B592EFD41D8429968EDBAD1C120C7450A6F06B9414434F7AED691E34140511F36E34BEA4188ED8C03715FDE41E8333F92D7A9F34121CE41E8EDD7F2C1887DF706DCD9DAC1965CEEB0DD58F041CE49CFF1450301429A31884CBCE601C27CEF44745F2AED41480D305026B0F7C1658EAB1A2C6B02C28067A696F75C984192A8053799BCFDC10103000080010000000C0000000059F4F350C5E8C15C3031495B92DEC17CA63C2E5D24FB4199A6E73BB851FFC192649573926701C210C015635AC6CE4148B94FE8CC2FF7C13EBCAB40A277FBC104778240A80E01C2AC636315E469F5415BCE779A9180FDC16C03BEAE819AECC178749AEC20BFEB412CF10208B1E8E3C170BF84BBCC50DE417A5E24923A3C0242A4D2E074D6F3E14190C04754ED01CCC180D5D994D881BFC196E67516218CF741F082202CE6D7CBC14A7AC1855776E3C130D199CE513AEC41165AD52F5B42FBC124E9B4AB8DFEF8C15CB2A656ACBDFF41949D31E97302E841DCADBFB0B5C0F1C100D9911A2A64D2419428C553086D00C21F0022FAFEC301C2082477482B06DF414038FDD56C70C2410059F4F350C5E8C15C3031495B92DEC17CA63C2E5D24FB4101030000800100000006000000464E3293B54000C20EC23643D889F6C10FBA02CB74ACF0C1D8F40969E5B2D4C13545FAF3BDD0F2C1AE4FF0649BE8F9C12622C22360640042A4D6BB577EEEF241CA76CC4597B8F7416496417DE058F541301BDF56F390F44128C06F222168F7415E43E8763E5EF6C1F091970286A2F341AC3E4951E85CFE41464E3293B54000C20EC23643D889F6C10FBA02CB74ACF0C10103000080010000000A000000BE1453EF7FC0E9C100156E0EEC42A2C1F4A65202C117FEC144C16AA198FBFEC15B787D4C2FBDF3C138FD8B9B063ADD413CECC8E8AE1CEBC104634FA43AABF0C1F5733E66EF9102C2D0386679CB47DF4129C9D5762A4AF4C16C322D7AF5DCFFC150442BE804E4F54150835E862554DCC1DC740101138B00C2BAE57AEC8863F941B053E9671B3BC2C19C28954C8691FE418049F4E3441EE141940D967EDB7EFD410075940DE7C9C8C1A8474F9C526800C20AE5492E1A7E02425434439437E4FC417218BE27F1DBFEC16A2FF629CAF3F0419498D8798E26FE41BE1453EF7FC0E9C100156E0EEC42A2C1F4A65202C117FEC1':::GEOMETRY,2202:::OID,1560:::OID), Expected Errors: , Potential Errors: ]: ERROR: relation \"seq3541\" (816): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:01:37.348142",
          |  "ops": null,
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "WATCH DOG: failed to get session information: can't scan into dest[1]: cannot scan null into *string\n"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:01:37.379094",
          |  "ops": null,
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "WATCH DOG: failed to get session information: can't scan into dest[1]: cannot scan null into *string\n"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ d35c174b71065264b8a3910df3f488d10741f788:

          | I220830 13:59:12.309875 228 workload/pgx_helpers.go:79  [-] 1740  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3256.table4424" does not exist (SQLSTATE 42P01) pid:1414074 sql:INSERT INTO schema3256.table4424 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:349.260748ms]
          | I220830 13:59:12.317142 278172 workload/pgx_helpers.go:79  [-] 1741  pgx logger [error]: Query logParams=map[args:[1710239df6eab8620000000000000001] err:can't scan into dest[1]: cannot scan null into *string pid:1880538 sql:SELECT active_queries, kv_txn FROM crdb_internal.cluster_sessions WHERE session_id = $1]
          | I220830 13:59:13.096869 229 workload/pgx_helpers.go:79  [-] 1742  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "public.table4426" does not exist (SQLSTATE 42P01) pid:1676029 sql:SELECT t0.col4238_4245 AS col0,t0.col4238_4245 AS col1,t0.col4238_4244 AS col2,t0.col4238_4240 AS col3,t0.col4238_4240 AS col4 FROM schema3523.table4238 AS t0 ,public.table4426 AS t1 ,schema4427.table4428 AS t2  FETCH FIRST 1 ROWS ONLY]
          | I220830 13:59:14.106330 216 workload/pgx_helpers.go:79  [-] 1743  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot add region "us-west1" to database schemachange (SQLSTATE 42P12) pid:2025457 sql:ALTER DATABASE schemachange ADD REGION "us-west1" time:366.088µs]
          | I220830 13:59:15.082626 224 workload/pgx_helpers.go:79  [-] 1744  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3523.table4433" does not exist (SQLSTATE 42P01) pid:1423779 sql:INSERT INTO schema3523.table4433 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:4.669346ms]
          | I220830 13:59:15.193300 277802 workload/pgx_helpers.go:79  [-] 1745  pgx logger [error]: Query logParams=map[args:[1710239df74abab00000000000000001] err:can't scan into dest[1]: cannot scan null into *string pid:1857984 sql:SELECT active_queries, kv_txn FROM crdb_internal.cluster_sessions WHERE session_id = $1]
          | I220830 13:59:16.048989 226 workload/pgx_helpers.go:79  [-] 1746  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "seq4408" (1042): expected matching namespace entry, found none (SQLSTATE XXUUU) pid:1631576 sql:commit time:262.266381ms]
          | Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:
          | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema3523" "table4238"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema3523" "table4238"]] :true
          | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema3523" "table4238"]] :true
          | ===========================Previous statements [QUERY: CREATE SEQUENCE IF NOT EXISTS schema3256.seq4408, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema3523.table4238 (col4238_4239,col4238_4240,col4238_4241,col4238_4242,col4238_4243) VALUES (90004:::OID,e'7.K\x07~'::NAME:::NAME,'-71 years -10 mons -436 days -08:35:40.778842':::INTERVAL,''::CHAR,701:::OID),(1700:::OID,e'\x1c*i'::NAME:::NAME,'-1 years -4 mons -550 days -10:38:42.211373':::INTERVAL,'x'::CHAR,18:::OID), Expected Errors: , Potential Errors:  QUERY: SELECT t1.col4238_4245 AS col0,t2.col4238_4239 AS col1,t2.col4238_4243 AS col2,t1.col4238_4242 AS col3,t0.col4238_4242 AS col4,t0.col4238_4243 AS col5 FROM schema3523.table4238 AS t0 ,schema3523.table4238 AS t1 ,schema3523.table4238 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200]: ERROR: relation "seq4408" (1042): expected matching namespace entry, found none (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 8} lock=true stat=PENDING rts=1661867950.072283708,0 wto=false gul=1661867950.572283708,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:59:10.217362",
          |  "ops": [
          |   "BEGIN",
          |   "QUERY: INSERT INTO schema3523.table4238 (col4238_4239,col4238_4240,col4238_4241,col4238_4242,col4238_4243) VALUES (1560:::OID,e'\\x17CkRq'::NAME:::NAME,'80 years 8 mons 378 days 01:58:44.972277':::INTERVAL,e'\\x02'::CHAR,1560:::OID),(700:::OID,e';[lh\\t\\x07L\\x1f'::NAME:::NAME,'-51 years -9 mons -511 days -02:54:43.912576':::INTERVAL,'7'::CHAR,4089:::OID),(1042:::OID,e'9\\x1a\\x18'::NAME:::NAME,'-5 years -7 mons -183 days -09:23:35.871636':::INTERVAL,e'\\t'::CHAR,701:::OID), Expected Errors: , Potential Errors: ",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/104/2/1): \"sql txn\" meta={id=1550ed73 key=/Table/1008/1/\"\"/80 years 8 mons 378 days 01:58:44.972277/\"\\x02\"/\"\\x17CkRq\"/1560/\"w\\x0f\\th\\rWS'{\"/0 pri=0.04391046 epo=0 ts=1661867952.632632671,2 min=1661867950.215964378,0 seq=12} lock=true stat=PENDING rts=1661867950.215964378,0 wto=false gul=1661867950.715964378,0 (SQLSTATE 40001)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "13:59:07.251598",
          |  "ops": [
          |   "BEGIN",
          |   "QUERY: CREATE SEQUENCE IF NOT EXISTS schema3256.seq4408, Expected Errors: , Potential Errors: ",
          |   "QUERY: INSERT INTO schema3523.table4238 (col4238_4239,col4238_4240,col4238_4241,col4238_4242,col4238_4243) VALUES (90004:::OID,e'7.K\\x07~'::NAME:::NAME,'-71 years -10 mons -436 days -08:35:40.778842':::INTERVAL,''::CHAR,701:::OID),(1700:::OID,e'\\x1c*i'::NAME:::NAME,'-1 years -4 mons -550 days -10:38:42.211373':::INTERVAL,'x'::CHAR,18:::OID), Expected Errors: , Potential Errors: ",
          |   "QUERY: SELECT t1.col4238_4245 AS col0,t2.col4238_4239 AS col1,t2.col4238_4243 AS col2,t1.col4238_4242 AS col3,t0.col4238_4242 AS col4,t0.col4238_4243 AS col5 FROM schema3523.table4238 AS t0 ,schema3523.table4238 AS t1 ,schema3523.table4238 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3523\" \"table4238\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3523\" \"table4238\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3523\" \"table4238\"]] :true\n===========================Previous statements [QUERY: CREATE SEQUENCE IF NOT EXISTS schema3256.seq4408, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema3523.table4238 (col4238_4239,col4238_4240,col4238_4241,col4238_4242,col4238_4243) VALUES (90004:::OID,e'7.K\\x07~'::NAME:::NAME,'-71 years -10 mons -436 days -08:35:40.778842':::INTERVAL,''::CHAR,701:::OID),(1700:::OID,e'\\x1c*i'::NAME:::NAME,'-1 years -4 mons -550 days -10:38:42.211373':::INTERVAL,'x'::CHAR,18:::OID), Expected Errors: , Potential Errors:  QUERY: SELECT t1.col4238_4245 AS col0,t2.col4238_4239 AS col1,t2.col4238_4243 AS col2,t1.col4238_4242 AS col3,t0.col4238_4242 AS col4,t0.col4238_4243 AS col5 FROM schema3523.table4238 AS t0 ,schema3523.table4238 AS t1 ,schema3523.table4238 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200]: ERROR: relation \"seq4408\" (1042): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ b316a5ed5fe7253d113174d9d95ddebf1143b4e4:

          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 368902 [runnable]:
          | runtime.gopark(0xc000f11f78?, 0x3?, 0x0?, 0x30?, 0xc000f11f4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc000f11dc8 sp=0xc000f11da8 pc=0x465216
          | runtime.selectgo(0xc000f11f78, 0xc000f11f44, 0xc0010a2b00?, 0x0, 0x3f71820?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc000f11f08 sp=0xc000f11dc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002bbcc30, {0x5506a98, 0xc0010a2b00})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000f11fb8 sp=0xc000f11f08 pc=0x1aab6ae
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000f11fe0 sp=0xc000f11fb8 pc=0x1aab9ae
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000f11fe8 sp=0xc000f11fe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 368989 [runnable]:
          | runtime.gopark(0xc00222ef78?, 0x3?, 0x0?, 0x30?, 0xc00222ef4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00222edc8 sp=0xc00222eda8 pc=0x465216
          | runtime.selectgo(0xc00222ef78, 0xc00222ef44, 0xc0010a2b00?, 0x0, 0x3f71820?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00222ef08 sp=0xc00222edc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002868960, {0x5506a98, 0xc0010a2b00})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc00222efb8 sp=0xc00222ef08 pc=0x1aab6ae
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc00222efe0 sp=0xc00222efb8 pc=0x1aab9ae
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00222efe8 sp=0xc00222efe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  (col4955_4961, col4955_4957, col4955_4956, col4955_4967, col4955_4958, col4955_4963, col4955_4968, col4955_4960, col4955_4966, col4955_4962, col4955_4965)= ( SELECT  (col4955_4961, col4955_4957, col4955_4956, col4955_4967, col4955_4958, col4955_4963, col4955_4968, col4955_4960, col4955_4966, col4955_4962, col4955_4965) FROM (VALUES( 'BOX(-1.6594455775631447 -0.2781458012779048,-1.4804816641738205 0.77814576626752)':::BOX2D,'24:00:00':::TIME,'SL':::STRING,'23:24:43.616041':::STRING,'23:24:43.616041':::STRING,e'\\\\x0f\\\\x1f\\\\x1b[^u\u003eg9':::STRING,1043:::OID,false,'24:00:00':::STRING,701:::OID,'23:24:43.616041':::TIME) ) AS T(col4955_4961,col4955_4956,col4955_4967,col4955_4968,col4955_4966,col4955_4965,col4955_4957,col4955_4958,col4955_4963,col4955_4960,col4955_4962) ) )\" []] :false\nQUERY [\"SELECT array[((col4955_4961, col4955_4957, col4955_4956, col4955_4967, col4955_4958, col4955_4963, col4955_4968, col4955_4960, col4955_4966, col4955_4962, col4955_4965))::STRING] FROM (VALUES('BOX(-1.6594455775631447 -0.2781458012779048,-1.4804816641738205 0.77814576626752)':::BOX2D,'24:00:00':::TIME,'SL':::STRING,'23:24:43.616041':::STRING,'23:24:43.616041':::STRING,e'\\\\x0f\\\\x1f\\\\x1b[^u\u003eg9':::STRING,1043:::OID,false,'24:00:00':::STRING,701:::OID,'23:24:43.616041':::TIME) ) AS T(col4955_4961,col4955_4956,col4955_4967,col4955_4968,col4955_4966,col4955_4965,col4955_4957,col4955_4958,col4955_4963,col4955_4960,col4955_4962)\" []] :[[\"(\\\"BOX(-1.6594455775631447 -0.2781458012779048,-1.4804816641738205 0.77814576626752)\\\",1043,24:00:00,SL,f,24:00:00,23:24:43.616041,701,23:24:43.616041,23:24:43.616041,\\x0f\\x1f\\x1b[^u\u003eg9)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema4574.table4955'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema4574' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4955' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema4574.table4955', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema4574'\\n\\t\\t           AND table_name = 'table4955'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema4574.table4955 (col4955_4956,col4955_4957,col4955_4958,col4955_4959,col4955_4960,col4955_4961,col4955_4962) VALUES ('24:00:00':::TIME,1043:::OID,false,1042:::OID,701:::OID,'BOX(-1.6594455775631447 -0.2781458012779048,-1.4804816641738205 0.77814576626752)':::BOX2D,'23:24:43.616041':::TIME), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:22:17.366748",
          |  "ops": null,
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "WATCH DOG: failed to get session information: timeout: context canceled\n"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!