search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

roachtest: schemachange/random-load failed #82133

Closed cockroach-teamcity closed 2 years ago

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 662cc5c3070e6d64d155a9cc9f33253ee5d99ee9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:884: output in run_124321.258348568_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1947
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:884
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) output in run_124321.258348568_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 09FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);" []] :false
          | QUERY ["SELECT (((col8964_8969, col8964_8968))::STRING) IS NULL AS c FROM ( VALUES('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);" []] :false
          | QUERY ["SELECT (((col8964_8969))::STRING) IS NULL AS c FROM ( VALUES(NULL,'0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY)) AS t(col8964_8969,col8964_8968);" []] :true
          | QUERY ["\n    WITH tab_json AS (\n                    SELECT crdb_internal.pb_to_json(\n                            'desc',\n                            descriptor\n                           )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema4785.table8964"]] :[["(col8964_8969)"] ["(CASE WHEN col8964_8969 IS NULL THEN e'\\'':::STRING ELSE e'\\x1e!#':::STRING END)"] ["(col8964_8969)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema4785.table8964'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema4785'\n\t\t           AND table_name = 'table8964'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          | ===========================Previous statements [INSERT INTO schema4785.table8964 (col8964_8968,col8964_8969) VALUES ('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)]: ERROR: duplicate key value violates unique constraint "table8964_expr_key" (SQLSTATE 23505)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 2C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);\" []] :false\nQUERY [\"SELECT (((col8964_8969))::STRING) IS NULL AS c FROM ( VALUES(NULL,'0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY)) AS t(col8964_8969,col8964_8968);\" []] :true\nQUERY [\"\\n    WITH tab_json AS (\\n                    SELECT crdb_internal.pb_to_json(\\n                            'desc',\\n                            descriptor\\n                           )-\u003e'table' AS t\\n                      FROM system.descriptor\\n                     WHERE id = $1::REGCLASS\\n                  ),\\n         columns_json AS (\\n                        SELECT json_array_elements(t-\u003e'columns') AS c FROM tab_json\\n                      ),\\n         columns AS (\\n                    SELECT (c-\u003e\u003e'id')::INT8 AS col_id,\\n                           IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema4785.table8964\"]] :[[\"(col8964_8969)\"] [\"(CASE WHEN col8964_8969 IS NULL THEN e'\\\\'':::STRING ELSE e'\\\\x1e!#':::STRING END)\"] [\"(col8964_8969)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema4785.table8964'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema4785'\\n\\t\\t           AND table_name = 'table8964'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO schema4785.table8964 (col8964_8968,col8964_8969) VALUES ('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)]: ERROR: duplicate key value violates unique constraint \"table8964_expr_key\" (SQLSTATE 23505)"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-16222

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ c080f7ac2f5be11aedd787dceba77cca7df18a16:

          | runtime.gopark(0xc001528f78?, 0x3?, 0x0?, 0x30?, 0xc001528f4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc001528dc8 sp=0xc001528da8 pc=0x465216
          | runtime.selectgo(0xc001528f78, 0xc001528f44, 0xc001372100?, 0x0, 0x3f763c0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc001528f08 sp=0xc001528dc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc0028e6000, {0x550cc68, 0xc001372100})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc001528fb8 sp=0xc001528f08 pc=0x1a90b2e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc001528fe0 sp=0xc001528fb8 pc=0x1a90e2e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc001528fe8 sp=0xc001528fe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 260035 [runnable]:
          | runtime.gopark(0xc0011767b0?, 0x2?, 0xf8?, 0x66?, 0xc0011767a4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc001176630 sp=0xc001176610 pc=0x465216
          | runtime.selectgo(0xc0011767b0, 0xc0011767a0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc001176770 sp=0xc001176630 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0011767e0 sp=0xc001176770 pc=0xd2f2ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0011767e8 sp=0xc0011767e0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 259134 [runnable]:
          | runtime.gopark(0xc00205af78?, 0x3?, 0x0?, 0x30?, 0xc00205af4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00205adc8 sp=0xc00205ada8 pc=0x465216
          | runtime.selectgo(0xc00205af78, 0xc00205af44, 0xc001372100?, 0x0, 0x3f763c0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00205af08 sp=0xc00205adc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002819130, {0x550cc68, 0xc001372100})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc00205afb8 sp=0xc00205af08 pc=0x1a90b2e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc00205afe0 sp=0xc00205afb8 pc=0x1a90e2e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00205afe8 sp=0xc00205afe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | T8,e'\\\\'::CHAR),(869:::OID,'01070000A0E61000000600000001010000A0E610000078232150B11234C0F40846488B144140B4B786E7B10CF5C101010000A0E610000078A1E4D644F54640497632D536AA54C0B85290D47233F9C101060000A0E61000000200000001030000800100000009000000E87C9176AC6649C0D431A1E53C103BC0ACB25448152EFF41E06FDB2115E01CC040F727751A4AF53F062DAFC50EB9F5C140E943BF19B209C00F730C260BF749C070EACC26FB54EB4160AAF26F3C432D40786A6174DFFE35C044F2796C331EFF41301665F8A7EF564019944CD6FBC842C07A5C042F7D01F84124D3A3BA81B14C40B8D002ACA3014740B8B566EA8115FCC1686ABCBDEEE747C0E23BD2140A395540A80AD0E1E825DC41079C97DA643651C0A225A244F11C56407DD452E9D889F6C1E87C9176AC6649C0D431A1E53C103BC0ACB25448152EFF410103000080010000000800000032B54540758C5AC0BC8BCD447AEE4C40705DDA0D0FC3CCC177DA1DD2218862C0F067E87706E347405C933FDF7480EBC1389068D2F3433CC0D4C3DA891F964740DC49EAE13C12FFC1E4F95021EEF765401E8BF560FE4D564088BAC64DC9CED641DCCB8FC089D25340EEB4998DEA085640D2802FFECB47F5C120695247D37743C042C04CFFE172544090F748EE7A1AC3C1B0F9EEEF39654DC0CECF9A995E4D56404E6259E9C580E2C132B54540758C5AC0BC8BCD447AEE4C40705DDA0D0FC3CCC101050000A0E610000004000000010200008007000000E1116249ED995AC06C2C37797B5032C00015EDDF87D699C1F76E041972A358C0CE5CE13136363EC06413DC8C2405E541604CB7A392004EC03AD7415222014FC0BB872925A58700C23808E938F74C32C014B86837B2A636C0A601C783961EF4C1A8BFCCF1C8634840D1FC8CFD89DD55C016578A60BD80F2412CB633566DF45E4078923C029FCF3FC0645C76D02431F2412406A366C3245A40B0FF74BD4AB81B40C0483398517FF8C1010200008004000000208E3AC53ABD4240B8B5BE9E19A246C0EC4C5CE61F5700428AFD6CD649C56340305365F1E388314044219536AFEEFA4106FEBDD2B2E762401055A4CDA2A7444028D312E394FCF1414644628AA14E58C08C14BF046C04444094C25304632E01C2010200008002000000DA8AEEA5CBC75AC0FFDAE3C3141E4AC026389877CC61F3415EF79E2E8344524094116ABB60E853C0ECA8BBCF18ACE7C101020000800300000025E39C8DD57665C004197F8F455953C06484CADEE01DE641F0691A7BDA7F20C00059C6BE6375FC3F5419AC8345E2F5414AD33D3732965FC06263001FCC2C5240A0DF93968E7AF1C101030000A0E6100000010000000A000000E3DF28A5FC7665C0F015734266443AC03F3F36716A2102C2465C5541DA884EC0A4162F8AEC6D48C05409AC3E3F6AF841E8E9215A24FE3FC03845AC1797842AC0A09F085B1D84D8411A68250188D86540ED33C3750EB047C0D88E40E0E6AFEAC1B6BED12651FC52408CBAD1032DA54E40E437746BE17AE84190FE32FA624139400401160F5A9E5540204178D89E80BFC1D4DF5274C93D41C0E08F073E009C524012763F64F3F4F34123C6619F20D758C0C08D1051D0E55440728435B62B34F141EE7C035E16A75AC0988F9333B2115540E88B50E5EC0ED441E3DF28A5FC7665C0F015734266443AC03F3F36716A2102C201040000A0E610000002000000010100008084F1B16835C454C084D45729903835406CA136EF0E46FD410101000080EA952F8D65805EC0C844526C897F4F4058E0B141710BE041':::GEOGRAPHY,'04:23:28.782631':::TIME,'cbfc86f3-c3d9-4723-b562-4740e29fabd5':::UUID,'0d847842-566c-41dd-8183-fd1f8bdcbc36':::UUID,2187169389959324096:::INT8,B'000001101110110101000101010101010',1186:::OID,'0104000020E6100000030000000101000000081F0D0D9A4333409EAD6E073D1C424001010000001EB6F32EDC2366404C1DD30AEDAF3CC001010000007C29A7CF0DBD5140A006077214E90240':::GEOGRAPHY,'2005-11-26 15:30:46.000852+00:00':::TIMESTAMPTZ,'24:00:00-15:59:00':::TIMETZ,'63 years 3 mons 675 days 03:44:01.766508':::INTERVAL,21:::OID,ARRAY['18:00:11.435092':::TIME,'19:06:05.940257':::TIME],0.41053306360871844:::FLOAT8,'0'::CHAR),(1184:::OID,'0102000060E61000000300000040433FACE35048C0A77DD090001644C03B610381F841F0C1E3225450D5F252C00A751814F4B051C01E6E80A482A4FCC18846503C5F055440E1DE1CF9F99554C034E16C6C6E4EF241':::GEOGRAPHY,'13:59:26.56293':::TIME,'7f76bf35-399d-46b5-abd8-477ef84b9e1e':::UUID,'678d6b1d-530b-478c-94cc-4cb105e306c4':::UUID,357741825856761355:::INT8,B'011000010100011001011100000111101',1184:::OID,'01040000A0E6100000020000000101000080EC6F4C74A73964C013563C96D9764BC0646D316B7B23F7C10101000080E4D9CB19049740C02066625300B30740306D7E8A1B42CBC1':::GEOGRAPHY,'2017-07-14 08:50:06.000775+00:00':::TIMESTAMPTZ,'08:37:00.070212+02:32:00':::TIMETZ,'6 years 10 mons 213 days 07:53:00.391277':::INTERVAL,19:::OID,ARRAY['00:13:45.26256':::TIME,'13:09:40.062221':::TIME],'+Inf':::FLOAT8,'3'::CHAR), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 4dcb32c0346e20a95847763f89b9b0796d9ed4dc:

          |
          | goroutine 449077 [runnable]:
          | runtime.gopark(0xc0000a7f78?, 0x3?, 0x0?, 0x30?, 0xc0000a7f4a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0000a7dc8 sp=0xc0000a7da8 pc=0x465216
          | runtime.selectgo(0xc0000a7f78, 0xc0000a7f44, 0xc000f02dc0?, 0x0, 0x3f769e0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0000a7f08 sp=0xc0000a7dc8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc003206190, {0x550d408, 0xc000f02dc0})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0000a7fb8 sp=0xc0000a7f08 pc=0x1a90b2e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0000a7fe0 sp=0xc0000a7fb8 pc=0x1a90e2e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000a7fe8 sp=0xc0000a7fe0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 449979 [runnable]:
          | runtime.gopark(0xc002dcdfb0?, 0x2?, 0xf8?, 0xde?, 0xc002dcdfa4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc002dcde30 sp=0xc002dcde10 pc=0x465216
          | runtime.selectgo(0xc002dcdfb0, 0xc002dcdfa0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc002dcdf70 sp=0xc002dcde30 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc002dcdfe0 sp=0xc002dcdf70 pc=0xd2f2ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc002dcdfe8 sp=0xc002dcdfe0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 450193 [runnable]:
          | runtime.gopark(0xc0027197b0?, 0x2?, 0x60?, 0x96?, 0xc0027197a4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc002719630 sp=0xc002719610 pc=0x465216
          | runtime.selectgo(0xc0027197b0, 0xc0027197a0, 0xc000880df0?, 0x0, 0xc00163f0e0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc002719770 sp=0xc002719630 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0027197e0 sp=0xc002719770 pc=0xd2f2ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0027197e8 sp=0xc0027197e0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | mons 849 days 09:07:15.24373':::INTERVAL,0.4116644884997049971:::DECIMAL,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID,'1 year 7 mons 925 days 00:15:40.084379':::INTERVAL) ) AS T(col5418_5429,col5418_5425,col5418_5433,col5418_5435,col5418_5430,col5418_5421,col5418_5422,col5418_5427,col5418_5432,col5418_5431,col5418_5423)\" []] :[[\"(\\\"1 year 7 mons 925 days 00:15:40.084379\\\",\\\"2003-07-02 22:06:29.000174+00:00\\\",1560,4096,~\\x16eF,\\\"62 years 11 mons 849 days 09:07:15.24373\\\",-0.5100394236031163782,9bb933f3-8aaa-4cae-b3cf-c09fd3240b84,-0.9217039121028213753,ffffffff-ffff-ffff-ffff-ffffffffffff,\\\"-73 years -7 mons -692 days -18:26:57.373198\\\")\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema2653.table5418'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema2653' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table5418' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema2653.table5418', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema2653'\\n\\t\\t           AND table_name = 'table5418'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema2653.table5418 (col5418_5419,col5418_5420,col5418_5421,col5418_5422,col5418_5423,col5418_5424,col5418_5425,col5418_5426,col5418_5427,col5418_5428,col5418_5429,col5418_5430,col5418_5431,col5418_5432,col5418_5433,col5418_5434,col5418_5435) VALUES ('[false, [0.0335919891811407], {}, []]':::JSONB,'294276-12-31 23:59:59.999999+00:00':::TIMESTAMPTZ,2202:::OID,e'B8=\\x03\\x0e'::NAME:::NAME,'-69 years -3 mons -400 days -04:12:29.915842':::INTERVAL,'eaef:b976:d581:34d1:3971:e4d2:9e65:f096/81':::INET,'57 years 8 mons 297 days 02:46:46.541525':::INTERVAL,1042:::OID,'19 years 6 mons 383 days 16:31:51.365574':::INTERVAL,1042:::OID,'9cbebe5e-c572-43cc-8786-843e3e6c541d':::UUID,23:::OID,'266b423e-de2d-421e-aa6a-5d0dcc2c4569':::UUID,(-3.636366667979596381E+27):::DECIMAL,6053688165.121697478:::DECIMAL,700:::OID,'-2000-01-01 00:00:00+00:00':::TIMESTAMPTZ),(e'[[true, {\"LK##HTIVF\": {\"PbYyY\u003csM\": 1.1330497642449417}}, {\"9W[M\\'\": {\"foobar\": []}, \"}8Uz;X[K\": null}, \"N_YsP\", true, [], {}], false]':::JSONB,NULL,4096:::OID,e'~\\x16eF'::NAME:::NAME,'1 year 7 mons 925 days 00:15:40.084379':::INTERVAL,'213.191.185.165/16':::INET,'-73 years -7 mons -692 days -18:26:57.373198':::INTERVAL,90004:::OID,'62 years 11 mons 849 days 09:07:15.24373':::INTERVAL,1560:::OID,'9bb933f3-8aaa-4cae-b3cf-c09fd3240b84':::UUID,1560:::OID,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID,0.4116644884997049971:::DECIMAL,(-0.9217039121028213753):::DECIMAL,1009:::OID,'2003-07-02 22:06:29.000174+00:00':::TIMESTAMPTZ), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ b0f13c9bbef3e6628471d887672be7c7658f6511:

          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 258009 [runnable]:
          | runtime.gopark(0xc001883fb0?, 0x2?, 0xf8?, 0x3e?, 0xc001883fa4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc001883e30 sp=0xc001883e10 pc=0x465216
          | runtime.selectgo(0xc001883fb0, 0xc001883fa0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc001883f70 sp=0xc001883e30 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc001883fe0 sp=0xc001883f70 pc=0xd2f2ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc001883fe8 sp=0xc001883fe0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 257938 [runnable]:
          | runtime.gopark(0xc00159efb0?, 0x2?, 0xc7?, 0x52?, 0xc00159efa4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00159ee30 sp=0xc00159ee10 pc=0x465216
          | runtime.selectgo(0xc00159efb0, 0xc00159efa0, 0xc0009a6e70?, 0x0, 0xc00010ff20?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00159ef70 sp=0xc00159ee30 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc00159efe0 sp=0xc00159ef70 pc=0xd2f2ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00159efe8 sp=0xc00159efe0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 257988 [runnable]:
          | runtime.gopark(0xc00106e7b0?, 0x2?, 0xf8?, 0xe6?, 0xc00106e7a4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00106e630 sp=0xc00106e610 pc=0x465216
          | runtime.selectgo(0xc00106e7b0, 0xc00106e7a0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00106e770 sp=0xc00106e630 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc00106e7e0 sp=0xc00106e770 pc=0xd2f2ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00106e7e8 sp=0xc00106e7e0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | t\\t\\t\\t\\t\\t\\t\\t    table_name='table5153' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema3739.table5153', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema3739'\\n\\t\\t           AND table_name = 'table5153'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [QUERY: DROP TABLE schema4495.table4674 RESTRICT, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema2742.table5048 (col5048_5049,col5048_5050,col5048_5051,col5048_5052,col5048_5053,col5048_5054,col5048_5055,col5048_5056,col5048_5057,col5048_5058,col5048_5059,col5048_5060,col5048_5061,col5048_5062,col5048_5063,col5048_5064) VALUES (NULL,'2008-01-09 15:17:45.000867+00:00':::TIMESTAMPTZ,0.05700961500406265:::FLOAT8,ARRAY[e'\\x00':::STRING,e'\\t\\n9f*FhC':::STRING,e'w:\\x18T\\x1a\\x0b\\r\\x05':::STRING],'0101000040000000000000F87F000000000000F87F000000000000F87F':::GEOMETRY,2205:::OID,'02184a97-839b-42c6-a437-1bf3480111e5':::UUID,'8685884d-a646-4f69-b124-2e25f75d7f43':::UUID,90000:::OID,'1977-01-28':::DATE,'24:00:00':::TIME,''::VARCHAR,'1970-08-30':::DATE,'2012-12-31 18:18:56.000849+00:00':::TIMESTAMPTZ,(-2.1002867221832275):::FLOAT8,3802:::OID), Expected Errors: , Potential Errors:  QUERY: CREATE SCHEMA schema5287 AUTHORIZATION root, Expected Errors: , Potential Errors:  QUERY: ALTER SEQUENCE schema3739.seq5207 RENAME TO schema3739.seq5207, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema3739.table5153 (col5153_5155,col5153_5156,col5153_5157,col5153_5158,col5153_5159,col5153_5160,col5153_5161,col5153_5162,col5153_5163,col5153_5164,col5153_5165,col5153_5166,col5153_5167,col5153_5168) VALUES ('55 years 11 mons 138 days 06:30:45.120108':::INTERVAL,'1975-06-01':::DATE,1.0330995321273804:::FLOAT8,B'11100001111',1894786193:::INT8,'BOX(-1.768983998993107 -0.17063296603575273,0.7357922491711375 0.4416433630277291)':::BOX2D,'}'::\"char\",'2482b987-b89c-4be7-8143-8a4ab54121de':::UUID,1266:::OID,90002:::OID,90000:::OID,ARRAY[e'\\'':::STRING,e'\\x0f':::STRING,'R':::STRING],3802:::OID,'82 years 222 days 04:51:46.34076':::INTERVAL),('-52 years -8 mons -31 days -24:40:02.464009':::INTERVAL,'1991-10-29':::DATE,(-0.11074919253587723):::FLOAT8,B'10011110011',1806223354:::INT8,'BOX(-0.7807303332530433 0.3384270663732014,-0.12424586671507853 1.2347847070465505)':::BOX2D,'X'::\"char\",'9b060de9-0564-464c-b1b2-26083a979845':::UUID,21:::OID,1082:::OID,1082:::OID,ARRAY[e'\\n':::STRING,'A':::STRING,'t':::STRING,e'\\x1a':::STRING],1266:::OID,'-25 years -8 mons -376 days -08:33:38.118848':::INTERVAL),('52 years 6 mons 661 days 18:09:36.704261':::INTERVAL,'1981-09-06':::DATE,(-0.25688132643699646):::FLOAT8,B'00010000100',(-232383820):::INT8,'BOX(-1.509601589655769 -0.1631420385226183,0.20403427394885937 -0.07889506018150583)':::BOX2D,';'::\"char\",'8f8b2e9f-66dc-4c88-b021-23aab76f63d3':::UUID,18:::OID,26:::OID,4089:::OID,ARRAY[',':::STRING,'}':::STRING,e'\\x0b':::STRING],17:::OID,'-32 years -359 days -18:18:34.919879':::INTERVAL), Expected Errors: , Potential Errors: ]: ERROR: schema \"schema5287\" (1265): expected matching namespace entry, found none (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ e5917605f751c569ddd90062250a40cafd7ccf30:

          |  1449.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1450.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          |  1450.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1450.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1450.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1451.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          |  1451.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1451.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1451.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1452.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1452.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1452.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1452.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1453.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          |  1453.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1453.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1453.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1454.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          |  1454.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1454.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1454.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1455.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          |  1455.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1455.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1455.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1456.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          |  1456.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1456.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1456.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1457.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1457.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1457.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1457.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1458.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          |  1458.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1458.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1458.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
          |  1459.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
          |  1459.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
          |  1459.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
          |  1459.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 2372698da1dfacb90f60c6a63f2c1298d1db16b8:

          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00060f7e8 sp=0xc00060f7e0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 230458 [runnable]:
          | github.com/jackc/puddle.(*Resource).Destroy.func1()
          |     github.com/jackc/puddle/external/com_github_jackc_puddle/pool.go:71 fp=0xc0012edfe0 sp=0xc0012edfd8 pc=0xe0d840
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0012edfe8 sp=0xc0012edfe0 pc=0x495701
          | created by github.com/jackc/puddle.(*Resource).Destroy
          |     github.com/jackc/puddle/external/com_github_jackc_puddle/pool.go:71 +0x85
          |
          | goroutine 230161 [runnable]:
          | runtime.gopark(0xc0022d1778?, 0x3?, 0x0?, 0x30?, 0xc0022d174a?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0022d15c8 sp=0xc0022d15a8 pc=0x465216
          | runtime.selectgo(0xc0022d1778, 0xc0022d1744, 0xc002250be8?, 0x0, 0x3f7a6a0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0022d1708 sp=0xc0022d15c8 pc=0x474cf2
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002cabe00, {0x5510e28, 0xc0010ea500})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0022d17b8 sp=0xc0022d1708 pc=0x1a91e6e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0022d17e0 sp=0xc0022d17b8 pc=0x1a9216e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0022d17e8 sp=0xc0022d17e0 pc=0x495701
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 230409 [runnable]:
          | runtime.gopark(0xc0014b27b0?, 0x2?, 0xf8?, 0x26?, 0xc0014b27a4?)
          |     GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0014b2630 sp=0xc0014b2610 pc=0x465216
          | runtime.selectgo(0xc0014b27b0, 0xc0014b27a0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0014b2770 sp=0xc0014b2630 pc=0x474cf2
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0014b27e0 sp=0xc0014b2770 pc=0xd2f2ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0014b27e8 sp=0xc0014b27e0 pc=0x495701
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | \n                           IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema6065.table6545\"]] :[]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema6065.table6545'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema6065' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table6545' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema6065.table6545', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema6065'\\n\\t\\t           AND table_name = 'table6545'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema6065.table6545 (col6545_6546,col6545_6547,col6545_6548,col6545_6549,col6545_6550,col6545_6551,col6545_6552,col6545_6553,col6545_6554,col6545_6555,col6545_6556,col6545_6557,col6545_6558,col6545_6559,col6545_6560,col6545_6561,col6545_6562,col6545_6563,col6545_6564) VALUES (4267112677082135895:::INT8,NULL,1700:::OID,'1ac1daf3-3b06-4fcd-9f6f-cb77735bd1cb':::UUID,19:::OID,(-2147483648):::INT8,4096:::OID,'1984-01-05':::DATE,'00:48:01.211766':::TIME,B'1111101010110111111110111101010010111001001',1043:::OID,'1993-07-04':::DATE,'871c:215f:fc2c:6b6c:5e2e:67e0:367f:ab78/73':::INET,'0'::\"char\",B'101101000111000111100000010','1975-03-15':::DATE,e'o\\x02'::STRING,(-0.5978192090988159):::FLOAT8,'[null, \"O``]@\", [{\"bar\": 1.5328551685767675}], null]':::JSONB), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 34dc56fbb5789b39be47b110bf22332c7f5654f6:

          | runtime.gopark(0xc0018dafb0?, 0x2?, 0x18?, 0x0?, 0xc0018dafa4?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc0018dae30 sp=0xc0018dae10 pc=0x466c16
          | runtime.selectgo(0xc0018dafb0, 0xc0018dafa0, 0xc000a106b0?, 0x0, 0xc002d0b200?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc0018daf70 sp=0xc0018dae30 pc=0x476adc
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0018dafe0 sp=0xc0018daf70 pc=0xd497ac
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0018dafe8 sp=0xc0018dafe0 pc=0x4981e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 427855 [runnable]:
          | runtime.gopark(0xc0027b6f78?, 0x3?, 0x0?, 0x30?, 0xc0027b6f4a?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc0027b6dc8 sp=0xc0027b6da8 pc=0x466c16
          | runtime.selectgo(0xc0027b6f78, 0xc0027b6f44, 0xc001316c40?, 0x0, 0x3fea260?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc0027b6f08 sp=0xc0027b6dc8 pc=0x476adc
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002b76af0, {0x5b894a8, 0xc001316c40})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0027b6fb8 sp=0xc0027b6f08 pc=0x1ac236e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0027b6fe0 sp=0xc0027b6fb8 pc=0x1ac266e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0027b6fe8 sp=0xc0027b6fe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 424696 [runnable]:
          | runtime.gopark(0xc003069f78?, 0x3?, 0x0?, 0x30?, 0xc003069f4a?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc003069dc8 sp=0xc003069da8 pc=0x466c16
          | runtime.selectgo(0xc003069f78, 0xc003069f44, 0xc001316c40?, 0x0, 0x3fea260?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc003069f08 sp=0xc003069dc8 pc=0x476adc
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002a51e50, {0x5b894a8, 0xc001316c40})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc003069fb8 sp=0xc003069f08 pc=0x1ac236e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc003069fe0 sp=0xc003069fb8 pc=0x1ac266e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc003069fe8 sp=0xc003069fe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | a)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: CREATE TYPE schema4777.enum5397 AS ENUM ('sfaa', 'a'), Expected Errors: , Potential Errors:  QUERY: SELECT t0.col4619_4632 AS col0,t1.col4619_4627 AS col1,t1.col4619_4631 AS col2,t0.col4619_4625 AS col3,t1.col4619_4632 AS col4,t1.col4619_4621 AS col5,t0.col4619_4622 AS col6,t1.col4619_4622 AS col7,t0.col4619_4628 AS col8,t0.col4619_4629 AS col9,t1.col4619_4628 AS col10,t1.col4619_4624 AS col11,t1.col4619_4632 AS col12,t0.col4619_4630 AS col13 FROM public.table4619 AS t0 ,public.table4619 AS t1  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO public.table4619 (col4619_4620,col4619_4621,col4619_4622,col4619_4623,col4619_4624,col4619_4625,col4619_4626,col4619_4627,col4619_4628) VALUES (25:::OID,'0101000020E6100000000000000000F87F000000000000F87F':::GEOGRAPHY,'1998-07-03 06:57:55.000061':::TIMESTAMP,'010700000009000000010200000003000000006AB210FBD9A1C16A0309C8A156F1C100A4999BD3ED6CC1D8A03B3B0754D04130AA9C5A9262E7413003CEBF50C0EC410105000000040000000102000000020000005891CBEC522C0042AA89E6845C3FF141C0097D6A588ABA4164FDCDB58002F6410102000000050000000005C41313C9FB411414D37B9568E441B0965D019670C841F88C8795252B00426B91038BF792F3C180BA7B674F0A00429E7238ECF75AF3C182AEFCD973DAF8418E472EE38EF1F1C1C4324416BAB3EE4101020000000200000070690B698ABFDA417C74C5405B26EE4168A1AEA7EE49D9C1204DEB69BC37EE410102000000050000003613DB1428E8F8411512ADD4CA9202C2F249189E3960F9411E23B6C4BE34FAC1C48453DEB61EF8413046510B222CCBC1B8CDAB68367EE64140D46447337DA4C1F246506D54F100C29084848E5CFEE9410105000000050000000102000000070000000BF5E861BBBCF4C10E5E4FBA6EC3F7C18FC6BBA428C9F1C1F553EBA80291FCC1C83830AD0CFDD04134CF09B2BE3DEBC1A6BA14DFB5680042389F33E55A96F7C142EC571F8F3B01424CF765B43ABAF2C18E6CC796B038E0C1A0A4F0582E8FE9419AE19A17E42DE4C1943594FA426DE64101020000000200000010FACC7DF544FD417622AC1B6C89EDC1FCB4C5E4AF02DCC178812FC41161F2410102000000020000004E1377382167E6C1525CEA1FD484FBC16C13BB178E18ED41CB57E118F089F8C101020000000600000066E493358EA0F841B65ED0E6040D01C284D1B5CDAFBCFD41B0B06CC8FEC7C34164B4BAC713CFEA41D03D14D1578DCF41042AFE9579460042DA5BEA776F82F14150528FE2D8CFD2C1680114FDFA69FA41082D3AC2486FE0C1F08700ACCF93DE41010200000002000000F48F3B67802EE341680956FEF7F7E94100BEDED08AB3D1C130AF1B2E1B9AC24101010000001E2AB5601B86F94186A2585BF4310142010600000002000000010300000001000000090000004B284E13A309F8C170516A5D91ECE5C19583E59FC4ADF6C12FCE2A1CF8C4FEC1ACBD0890EBFFEA417FDC908D5C5C01C2EC1FACF230DAF3410EA60490CF3DFBC1B800764FB3F6F7413216A94434E4F3C164F403F73B7F0042D009C361A04B024228F4C4F91F53D0C1009B833CF98285C164EC198FD34CF2C1C0D4E7FF834FA7414B284E13A309F8C170516A5D91ECE5C10103000000010000000D000000AE4B0892EF35FBC1789B8F6BE394FFC1D8A430EAF7C8E5C168B158DA1A35F6C10C44D8EC0760EA41E3256097CD87F0C15020361717AEFC41E02CCB55B68802C28A3504909E51F141407C04740F01EFC12AC02A36BAAFFA4181770181171EF5C1467EE9AB623DF2417E39485F732AE7C196B93D6C90160142002088331201AAC1687A363978B5E74106D5218CCAABF041800B7E7DC049AE4150B4011DFFABF44197E4E8F6425AFEC1CCB6E05FDC5400423BBAFF9446D9F8C1E602B836DE62F941AE4B0892EF35FBC1789B8F6BE394FFC1010300000001000000050000009C1B0F0DAE68FBC1102475BEBEADD4C1021FFB297D06F841B45D8E49A5C4EC41C063143CCC8FC4C108B2631DF195F341B001231A8FCADBC1ECBAE551ED47EE419C1B0F0DAE68FBC1102475BEBEADD4C101010000003EE829BCBE8BF14118E758ED32A5D041010400000006000000010100000080131776790CE041840C56DD8584E5C1010100000038F66EE50926E4C1685CFF8FC188F8410101000000E83453E24C61EA4148F8FC441A79EA410101000000D4B07E5CF498D6C1387C762A2F0DE0410101000000AA560FE68B44F8413E3EF85AEA1F02C201010000004AC6B4B30ACB01C2145BFD169527E3410101000000A08AAB6A6431BFC15AC8A336F782F341':::GEOMETRY,2202:::OID,1043:::OID,2205:::OID,(-2168):::INT8,(-700623831):::INT8), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ a0bfa6dafcc206301d3a21887c374db63b377075:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
    schemachange_random_load.go:165,schemachange_random_load.go:59,test_runner.go:928: output in run_142521.580064015_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
        (1) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1974
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:162
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:59
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:928
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) output in run_142521.580064015_n1_workload_run_schemachange
        Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
          | stderr:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 84)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149)" []] :false
          | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM public.table5138 WHERE (col5138_5150, col5138_5154, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152)= ( SELECT  (col5138_5150, col5138_5154, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152) FROM (VALUES( 'Hn'::STRING,'80 years 323 days 16:47:01.421334':::INTERVAL,'01:15:32.898716+10:48:00':::TIMETZ,'':::STRING,'hn':::STRING,'box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::STRING,869:::OID,'BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149) ) )" []] :false
          | QUERY ["SELECT array[((col5138_5150, col5138_5154, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152))::STRING] FROM (VALUES('Hn'::STRING,'80 years 323 days 16:47:01.421334':::INTERVAL,'01:15:32.898716+10:48:00':::TIMETZ,'':::STRING,'hn':::STRING,'box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::STRING,869:::OID,'BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149)" []] :[["(\"80 years 323 days 16:47:01.421334\",01:15:32.898716+10:48:00,\"\",hn,\"box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)\",869,\"BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)\",Hn)"]]
          | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table5138'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table5138' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table5138', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table5138'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
          |
          | ===========================
          | Previous statements [QUERY: INSERT INTO public.table5138 (col5138_5149,col5138_5150,col5138_5151,col5138_5152,col5138_5153,col5138_5154) VALUES ('BOX(-1.3510600150968646 0.7064702248189542,-0.2299858840764486 1.4366532836712582)':::BOX2D,'55 years 8 mons 65 days 07:33:21.024335':::INTERVAL,1560:::OID,'P'::STRING,NULL,'11:49:02.79375-01:04:00':::TIMETZ),('BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D,'80 years 323 days 16:47:01.421334':::INTERVAL,869:::OID,'Hn'::STRING,1161285865:::INT8,'01:15:32.898716+10:48:00':::TIMETZ), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 4, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152) FROM (VALUES( 'Hn'::STRING,'80 years 323 days 16:47:01.421334':::INTERVAL,'01:15:32.898716+10:48:00':::TIMETZ,'':::STRING,'hn':::STRING,'box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::STRING,869:::OID,'BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149) ) )\" []] :false\nQUERY [\"SELECT array[((col5138_5150, col5138_5154, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152))::STRING] FROM (VALUES('Hn'::STRING,'80 years 323 days 16:47:01.421334':::INTERVAL,'01:15:32.898716+10:48:00':::TIMETZ,'':::STRING,'hn':::STRING,'box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::STRING,869:::OID,'BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149)\" []] :[[\"(\\\"80 years 323 days 16:47:01.421334\\\",01:15:32.898716+10:48:00,\\\"\\\",hn,\\\"box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)\\\",869,\\\"BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)\\\",Hn)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table5138'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table5138' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table5138', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table5138'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO public.table5138 (col5138_5149,col5138_5150,col5138_5151,col5138_5152,col5138_5153,col5138_5154) VALUES ('BOX(-1.3510600150968646 0.7064702248189542,-0.2299858840764486 1.4366532836712582)':::BOX2D,'55 years 8 mons 65 days 07:33:21.024335':::INTERVAL,1560:::OID,'P'::STRING,NULL,'11:49:02.79375-01:04:00':::TIMETZ),('BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D,'80 years 323 days 16:47:01.421334':::INTERVAL,869:::OID,'Hn'::STRING,1161285865:::INT8,'01:15:32.898716+10:48:00':::TIMETZ), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:35:55.508552",
          |  "ops": null,
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "WATCH DOG: failed to get session information: timeout: context canceled\n"
          | }
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 4780ee15194e7d22b661ac92254b13cd2b71dcb1:

          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 232326 [runnable]:
          | runtime.gopark(0xc001389fb0?, 0x2?, 0x20?, 0x0?, 0xc001389fa4?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc001389e30 sp=0xc001389e10 pc=0x466c16
          | runtime.selectgo(0xc001389fb0, 0xc001389fa0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc001389f70 sp=0xc001389e30 pc=0x476adc
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc001389fe0 sp=0xc001389f70 pc=0xd4cccc
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001389fe8 sp=0xc001389fe0 pc=0x4981e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 230672 [runnable]:
          | runtime.gopark(0xc002393fb0?, 0x2?, 0x60?, 0x3e?, 0xc002393fa4?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc002393e30 sp=0xc002393e10 pc=0x466c16
          | runtime.selectgo(0xc002393fb0, 0xc002393fa0, 0xc001bdb130?, 0x0, 0xc001860ba0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc002393f70 sp=0xc002393e30 pc=0x476adc
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc002393fe0 sp=0xc002393f70 pc=0xd4cccc
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002393fe8 sp=0xc002393fe0 pc=0x4981e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 232338 [runnable]:
          | runtime.gopark(0xc00145ffb0?, 0x2?, 0xc7?, 0x6c?, 0xc00145ffa4?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc00145fe30 sp=0xc00145fe10 pc=0x466c16
          | runtime.selectgo(0xc00145ffb0, 0xc00145ffa0, 0xc001f5aae0?, 0x0, 0xc0012cdd40?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc00145ff70 sp=0xc00145fe30 pc=0x476adc
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc00145ffe0 sp=0xc00145ff70 pc=0xd4cccc
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00145ffe8 sp=0xc00145ffe0 pc=0x4981e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 65E135D40178A3233055953C0B848CD97C08ED1C1026C36145ABEF9412EF91B6C14BB6440AA4D00B0E7E350400C490929713FEDC13A92FBDB59E6F9C1743B79B76EA54640C43C7A4551204F4080C15D693899D04124A1B9498090F14180DF5EC3C57E09C06CAE2A4154403D40685BABB6A800D2C182174226780DFFC1506B6E4D10EA41C05A7E61D9D06146C0780CD2B02B47FAC140A105101EB6A3C101030000C0010000000900000094E2BAFF40895FC09183FB929C6546C00027B0B84E5D8D41A23FD5BEFFABE7C1C0FD9D8D91E343C070BBF0B1CD253CC0B03AB2D52084F441D0C3D5D4E9EFCE41208459F8257F2D40D80A19E6592536C0C6F8C38B407BF3C160F6267381F2B9C17043A00C32F54840A226FD55D9AF52C06081D8923CACC7C136DE3735D11AE6C1B4F845C8099365403438C48C69DD354000631E1C0653F1418AA61C0E5A3301C26AA77736C06B64403C27A47DBD0739404A3F369B59E400C29C303FFD0D95FE410EA990A90E0550C0D22DDA7902544140C889712BAB66D3C166A23EC9AFCCF541962A6CCC75FD5DC0C491DE1A666F424050605649B903D5C148C0B40FD594D84194E2BAFF40895FC09183FB929C6546C00027B0B84E5D8D41A23FD5BEFFABE7C101030000C0010000000B000000C48BABA1144162C0BA60FB28F08B54C01CCBD8DFE6E3EC4140238F005289E741568ABAD387C75FC02C0491A0C8A655C0D0B13DCFAC61F941B45A55F1A66CD7C1809BB9C4444017C0E1331EFB4A9443C040C45114C7B4F7C1A4F9DEE40280FF4120A7D257698F3840FA445DC059E63FC0D0BFB64EB37DE8C1E051EA71165BC2419222AC5EBD4450401C530F598DDC49C0D8EF42778BF0FE4122F4DD1E29C3ECC15C7A1C7A57FA6540908FB2D4C01D2DC0FA7A307407B80142A8A7745D4E3CE741BCC7A46351D65240187417EB251D35401E621AC592C1E7C115FC8CDB938CF8C1981F17BF20755A4018638DAFFE7153401E300FEA8FB8F041ACB84DA34B930142C054648E886C0CC028C7C9EEB4592140A4BD39E93227E3416858A8140F87D0C1985E7285C66145C0B49B650ED29D3040787B93FF3CADDBC1C626F238E3540242C48BABA1144162C0BA60FB28F08B54C01CCBD8DFE6E3EC4140238F005289E74101030000C0010000000C000000AB48115BB4155DC0ACC38146D44430C06CC65C04A36CDEC135A9536A3131FAC1E2156CBA58B560C0B4E706022F5C50C05CBBE203E281E0C15200B1C34E3700C29C4CFBAC2A4D45C0F0FC1D75FB2534C0CC26EAEC912BEAC1849567D3E3DF00C24E03E6D7140750C01081EF27955656C0FE70CD6D995BE6C1F84B6A3440AAE4410C0C555B398D5B404EF3A2A6D4B646C0E074135D0426EDC1403C98342CB9BB41B8C986AF6FFA5C4080317B74C70EFE3FE0245129CF3CD7C153061EB9FB0401C26A49114124A96240C8EABC05A3B355402063D7296F3DF541D0A2822371CFCEC1E864ECF7ABCF354074499423DDAE4340E0B40CEC0A36D24188DEEC0D62000042CCFA82C1156752C0E4E252D06BCA4840AFC962FD60C2F1C16CA4CE940227F0C1005072F73AE644C0CC81AE89E5623640FEF4C1B8E53D00C2F0C077322269C0C1FCEF3DA674E860C0885C9CECFC605340C08D392DC961A0C10058B1328040DE41AB48115BB4155DC0ACC38146D44430C06CC65C04A36CDEC135A9536A3131FAC101030000C001000000090000004EABFCFDEEA258C046356B07222A50C0B002AD6C5DB4E4411F08114FFCF7F2C19019CF1399F242C0D1CCB2C528DF49C03EE4DC8E76D50042CEFBF0753BBB0142F4C7A66FB68A5940F9D0CCD0C34146C066B3D49D017E00422C0E6CE17C47F2C14C62501388005B4073C69282802F43C08AFE72B550F0F7418F484B196D4E00C24020FD0C07A35940BCD4CCEA488B30C0E0498F5DE69DFB4190F9CEE3E483FA4180DEF831A45A654098790C2B5916444041A880B5055700C2DE7AD1B75C9AF3C15682667ED85C4FC0ECE480D80D164E402C67CE3D6242E64160EC45DFF2DEE041CCE48696E8B661C0E8E0F38958124340205BE6B49A4FD441B140E0738977FFC14EABFCFDEEA258C046356B07222A50C0B002AD6C5DB4E4411F08114FFCF7F2C101030000E0E6100000010000000C000000039774A7310566C080216A4A1D6451C06060E026E548C9C1523812EDA127014210A5593F645A54C07CEA84F2971748C0E04A242A3B94CCC124728F3BD340F841D64158F6B1FE51C00CD09121E98E47C08C264B2791BAFF4134501BEB005BFE41102330B14C5436C05086FEA32F9A51C0FCB58B98FCBCF241005CDE5FE025CE41FC5D711A2C625740A2CED6577F0856C0488D21B196BFE2C166F0FC5E3063F4C15CB1302878DC454047DF5581F70445C0710D3718E93AF0C108F86DAF5F810242CE9219757FFD634064133CA6E5EF3AC01E7A60251983E3C1F807D79B14B7D0C1A8F25886B9615740AC3F852DFF0353407E1855136F92F74154C1CA069A4DFB41585BF0CEB17837C050B9D19F1E694A401C9CEC0F92E8F84159445F3FDE0A00C2F8FEBB23821A50C03C1F962277C94D40D0AC68C67B23FD41C85140B5DD2BDFC1C3AD8A8CC4D257C0341EA4AB255E474000453DACA899B8C140746A926C0BA5C1039774A7310566C080216A4A1D6451C06060E026E548C9C1523812EDA1270142':::GEOGRAPHY,'1976-08-31 06:38:20.000497+00:00':::TIMESTAMPTZ,'0101000020E610000058006723731F6340141D3A6807BF4A40':::GEOGRAPHY,B'0001',700:::OID), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 4780ee15194e7d22b661ac92254b13cd2b71dcb1:

          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).isConnectionActive(0xc00295ef50, {0x5b9b188, 0xc001f28700})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:70 +0x2c5 fp=0xc0026ccf08 sp=0xc0026cce50 pc=0x1aca665
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc00295ef50, {0x5b9b188, 0xc001f28700})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:100 +0x110 fp=0xc0026ccfb8 sp=0xc0026ccf08 pc=0x1aca8f0
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0026ccfe0 sp=0xc0026ccfb8 pc=0x1acabce
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0026ccfe8 sp=0xc0026ccfe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 361790 [runnable]:
          | runtime.gopark(0xc0012d9f78?, 0x3?, 0x0?, 0x30?, 0xc0012d9f4a?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc0012d9dc8 sp=0xc0012d9da8 pc=0x466c16
          | runtime.selectgo(0xc0012d9f78, 0xc0012d9f44, 0xc001f28700?, 0x0, 0x3ff6880?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc0012d9f08 sp=0xc0012d9dc8 pc=0x476adc
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002a58f00, {0x5b9b188, 0xc001f28700})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0012d9fb8 sp=0xc0012d9f08 pc=0x1aca8ce
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0012d9fe0 sp=0xc0012d9fb8 pc=0x1acabce
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0012d9fe8 sp=0xc0012d9fe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 362698 [runnable]:
          | runtime.gopark(0xc000ee4f78?, 0x3?, 0x0?, 0x30?, 0xc000ee4f4a?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc000ee4dc8 sp=0xc000ee4da8 pc=0x466c16
          | runtime.selectgo(0xc000ee4f78, 0xc000ee4f44, 0xc001f28700?, 0x0, 0x3ff6880?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc000ee4f08 sp=0xc000ee4dc8 pc=0x476adc
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc001d4d130, {0x5b9b188, 0xc001f28700})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000ee4fb8 sp=0xc000ee4f08 pc=0x1aca8ce
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000ee4fe0 sp=0xc000ee4fb8 pc=0x1acabce
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000ee4fe8 sp=0xc000ee4fe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | lse\nQUERY [\"SELECT array[((col4667_4670))::STRING] FROM (VALUES(1560:::OID) ) AS T(col4667_4670)\" []] :[[\"1560\"]]\nQUERY [\"SELECT num_nulls(col4667_4669, col4667_4670, col4667_4671) \u003e 0 FROM (VALUES(1042:::OID,1560:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4669,col4667_4670,col4667_4671)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema2196.table4667 WHERE (col4667_4669, col4667_4670, col4667_4671)= ( SELECT  (col4667_4669, col4667_4670, col4667_4671) FROM (VALUES( 1042:::OID,1560:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4669,col4667_4670,col4667_4671) ) )\" []] :false\nQUERY [\"SELECT array[((col4667_4669, col4667_4670, col4667_4671))::STRING] FROM (VALUES(1042:::OID,1560:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4669,col4667_4670,col4667_4671)\" []] :[[\"(1042,1560,ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff)\"]]\nQUERY [\"SELECT num_nulls(col4667_4671) \u003e 0 FROM (VALUES('ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4671)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema2196.table4667 WHERE (col4667_4671)= ( SELECT  (col4667_4671) FROM (VALUES( 'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4671) ) )\" []] :true\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema2196.table4667'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema2196' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4667' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema2196.table4667', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema2196'\\n\\t\\t           AND table_name = 'table4667'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema2196.table4667 (col4667_4668,col4667_4669,col4667_4670,col4667_4671) VALUES ('21:01:08.892855':::TIME,26:::OID,2202:::OID,'4da9:7de5:f95b:48b1:2cb4:6d67:91b5:511a/58':::INET), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema2196.table4667 (col4667_4668,col4667_4669,col4667_4670,col4667_4671) VALUES ('19:08:25.259379':::TIME,26:::OID,25:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema2196.table4667 (col4667_4668,col4667_4669,col4667_4670,col4667_4671) VALUES ('15:48:16.396418':::TIME,700:::OID,2950:::OID,'5d9c:87cb:d9f2:fa54:e793:78c3:5a6c:9c80/1':::INET),('23:30:34.155906':::TIME,1042:::OID,1560:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET),('06:04:28.279302':::TIME,1560:::OID,19:::OID,'dabd:5b8d:5c07:c150:7a3e:a601:16e6:6007/56':::INET), Expected Errors: 23505, Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ e786cba2b137a671d3846cf7a33e7b9dea2854e6:

          | goroutine 301350 [runnable]:
          | runtime.gopark(0xc001dcafb0?, 0x2?, 0x0?, 0x0?, 0xc001dcafa4?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc001dcae30 sp=0xc001dcae10 pc=0x466c16
          | runtime.selectgo(0xc001dcafb0, 0xc001dcafa0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc001dcaf70 sp=0xc001dcae30 pc=0x476adc
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc001dcafe0 sp=0xc001dcaf70 pc=0xd5000c
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001dcafe8 sp=0xc001dcafe0 pc=0x4981e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | goroutine 302286 [runnable]:
          | runtime.gopark(0xc001547f78?, 0x3?, 0x0?, 0x30?, 0xc001547f4a?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc001547dc8 sp=0xc001547da8 pc=0x466c16
          | runtime.selectgo(0xc001547f78, 0xc001547f44, 0xc000fd6568?, 0x0, 0x4012820?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc001547f08 sp=0xc001547dc8 pc=0x476adc
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc00252b630, {0x5bb8328, 0xc0011ae080})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc001547fb8 sp=0xc001547f08 pc=0x1acf4ae
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc001547fe0 sp=0xc001547fb8 pc=0x1acf7ae
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001547fe8 sp=0xc001547fe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          | panic: dumping stacks, we failed to terminate threads on time.
          |
          | goroutine 302420 [running]:
          | panic({0x397efe0, 0x5b80860})
          |     GOROOT/src/runtime/panic.go:987 +0x3ba fp=0xc000f06f08 sp=0xc000f06e48 pc=0x463bba
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc00274a5a0, {0x5bb8328, 0xc0011ae080})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:96 +0x175 fp=0xc000f06fb8 sp=0xc000f06f08 pc=0x1acf535
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000f06fe0 sp=0xc000f06fb8 pc=0x1acf7ae
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000f06fe8 sp=0xc000f06fe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | IMESTAMP)) AS t(col4894_4895,col4894_4896,col4894_4897,col4894_4898,col4894_4899,col4894_4900);\" []] :[[\"\\bGEWY\"]]\nQUERY [\"SELECT num_nulls(col4894_4896, col4894_4899, col4894_4904, col4894_4900, col4894_4897) \u003e 0 FROM (VALUES(1043:::OID,'-35 years -4 mons -4 days -21:58:48.967541':::INTERVAL,'-2.4307332038879395'::FLOAT8,'1990-10-20 01:57:42.00058':::TIMESTAMP,'59 years 11 mons 491 days 22:21:03.698774':::INTERVAL) ) AS T(col4894_4896,col4894_4899,col4894_4904,col4894_4900,col4894_4897)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema3925.table4894 WHERE (col4894_4896, col4894_4899, col4894_4904, col4894_4900, col4894_4897)= ( SELECT  (col4894_4896, col4894_4899, col4894_4904, col4894_4900, col4894_4897) FROM (VALUES( 1043:::OID,'-35 years -4 mons -4 days -21:58:48.967541':::INTERVAL,'-2.4307332038879395'::FLOAT8,'1990-10-20 01:57:42.00058':::TIMESTAMP,'59 years 11 mons 491 days 22:21:03.698774':::INTERVAL) ) AS T(col4894_4896,col4894_4899,col4894_4904,col4894_4900,col4894_4897) ) )\" []] :false\nQUERY [\"SELECT array[((col4894_4896, col4894_4899, col4894_4904, col4894_4900, col4894_4897))::STRING] FROM (VALUES(1043:::OID,'-35 years -4 mons -4 days -21:58:48.967541':::INTERVAL,'-2.4307332038879395'::FLOAT8,'1990-10-20 01:57:42.00058':::TIMESTAMP,'59 years 11 mons 491 days 22:21:03.698774':::INTERVAL) ) AS T(col4894_4896,col4894_4899,col4894_4904,col4894_4900,col4894_4897)\" []] :[[\"(1043,\\\"-35 years -4 mons -4 days -21:58:48.967541\\\",-2.4307332038879395,\\\"1990-10-20 01:57:42.00058\\\",\\\"59 years 11 mons 491 days 22:21:03.698774\\\")\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema3925.table4894'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema3925' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4894' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema3925.table4894', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema3925'\\n\\t\\t           AND table_name = 'table4894'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema3925.table4894 (col4894_4895,col4894_4896,col4894_4897,col4894_4898,col4894_4899,col4894_4900) VALUES ('09:20:42.364954-06:06:00':::TIMETZ,1184:::OID,'50 years 4 mons 562 days 02:01:31.545969':::INTERVAL,(-0.41339758038520813):::FLOAT8,'-41 years -668 days -17:29:34.473611':::INTERVAL,'1970-11-03 03:19:17.000794':::TIMESTAMP),('08:37:57.269533-11:22:00':::TIMETZ,1043:::OID,'59 years 11 mons 491 days 22:21:03.698774':::INTERVAL,(-0.8522640466690063):::FLOAT8,'-35 years -4 mons -4 days -21:58:48.967541':::INTERVAL,'1990-10-20 01:57:42.00058':::TIMESTAMP), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 0eaeeb773474716753781289788fdd087fb9b166:

          | runtime.gopark(0xc000fa5f78?, 0x3?, 0x0?, 0x30?, 0xc000fa5f4a?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc000fa5dc8 sp=0xc000fa5da8 pc=0x466c16
          | runtime.selectgo(0xc000fa5f78, 0xc000fa5f44, 0xc00126eb80?, 0x0, 0x4010ca0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc000fa5f08 sp=0xc000fa5dc8 pc=0x476adc
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc001372d70, {0x5bb4908, 0xc00126eb80})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000fa5fb8 sp=0xc000fa5f08 pc=0x1acf44e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000fa5fe0 sp=0xc000fa5fb8 pc=0x1acf74e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000fa5fe8 sp=0xc000fa5fe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 370714 [runnable]:
          | runtime.gopark(0xc000fa3f78?, 0x3?, 0x0?, 0x30?, 0xc000fa3f4a?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc000fa3dc8 sp=0xc000fa3da8 pc=0x466c16
          | runtime.selectgo(0xc000fa3f78, 0xc000fa3f44, 0xc00126eb80?, 0x0, 0x4010ca0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc000fa3f08 sp=0xc000fa3dc8 pc=0x476adc
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc0026e9400, {0x5bb4908, 0xc00126eb80})
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000fa3fb8 sp=0xc000fa3f08 pc=0x1acf44e
          | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000fa3fe0 sp=0xc000fa3fb8 pc=0x1acf74e
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000fa3fe8 sp=0xc000fa3fe0 pc=0x4981e1
          | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
          |     github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
          |
          | goroutine 370820 [runnable]:
          | runtime.gopark(0xc002256fb0?, 0x2?, 0x20?, 0x0?, 0xc002256fa4?)
          |     GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc002256e30 sp=0xc002256e10 pc=0x466c16
          | runtime.selectgo(0xc002256fb0, 0xc002256fa0, 0x0?, 0x0, 0x0?, 0x1)
          |     GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc002256f70 sp=0xc002256e30 pc=0x476adc
          | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc002256fe0 sp=0xc002256f70 pc=0xd5016c
          | runtime.goexit()
          |     GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002256fe8 sp=0xc002256fe0 pc=0x4981e1
          | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
          |     github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | lic.table4473 WHERE (col4473_4486)= ( SELECT  (col4473_4486) FROM (VALUES( 25:::OID) ) AS T(col4473_4486) ) )\" []] :false\nQUERY [\"SELECT array[((col4473_4486))::STRING] FROM (VALUES(25:::OID) ) AS T(col4473_4486)\" []] :[[\"25\"]]\nQUERY [\"SELECT num_nulls(col4473_4486) \u003e 0 FROM (VALUES(701:::OID) ) AS T(col4473_4486)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM public.table4473 WHERE (col4473_4486)= ( SELECT  (col4473_4486) FROM (VALUES( 701:::OID) ) AS T(col4473_4486) ) )\" []] :false\nQUERY [\"SELECT array[((col4473_4486))::STRING] FROM (VALUES(701:::OID) ) AS T(col4473_4486)\" []] :[[\"701\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table4473'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4473' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table4473', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table4473'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: DROP SCHEMA \"schema1254\" CASCADE, Expected Errors: , Potential Errors:  QUERY: CREATE TYPE schema1884.enum4643 AS ENUM ('saad', 'sff', 'sfs', 'af', 'a'), Expected Errors: , Potential Errors:  QUERY: INSERT INTO public.table4473 (col4473_4475,col4473_4476,col4473_4477,col4473_4478,col4473_4479,col4473_4480,col4473_4481,col4473_4482,col4473_4483,col4473_4484,col4473_4485,col4473_4486,col4473_4487,col4473_4488,col4473_4489) VALUES ('BOX(-1.5414831460613732 -1.6678590931387252,1.8135343231405447 0.05206416541382236)':::BOX2D,2206:::OID,e'\\x00'::CHAR,B'01011','17:12:13.425157':::TIME,'3000-01-01 00:00:00+00:00':::TIMESTAMPTZ,(-2147483648):::INT8,NULL,'1984-04-10 12:26:43.000985':::TIMESTAMP,'\\x00':::BYTES,e';zf\\x04'::STRING,90001:::OID,1042:::OID,(-1842093163):::INT8,(-1.6645545959472656):::FLOAT8),('BOX(-1.9285756037988755 -0.22387667790538246,0.9358243017520091 -0.19399173084054755)':::BOX2D,3802:::OID,':'::CHAR,B'01000','09:11:08.305014':::TIME,'1978-07-31 13:58:33.000484+00:00':::TIMESTAMPTZ,(-3905200269684727671):::INT8,true,'1984-07-03 08:16:13.000177':::TIMESTAMP,'\\x2e62c2':::BYTES,'\"'::STRING,25:::OID,1043:::OID,2147483647:::INT8,0.47048094868659973:::FLOAT8),('BOX(-1.150100758472085 -0.3469992550086455,-0.6873019350113381 -0.16640408981756305)':::BOX2D,19:::OID,'B'::CHAR,B'01000','19:02:36.961733':::TIME,'2020-11-22 10:01:04.000248+00:00':::TIMESTAMPTZ,2147483647:::INT8,false,'2013-05-20 06:30:26.000857':::TIMESTAMP,'\\x58b71d02642942':::BYTES,'jp7e'::STRING,701:::OID,2950:::OID,(-442943547):::INT8,(-1.545388102531433):::FLOAT8), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
          | }
        Wraps: (4) SSH_PROBLEM
        Wraps: (5) Node 1. Command with error:
          | ``````
          | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
          | ``````
        Wraps: (6) exit status 255
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #89146 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ 762c1b86fe1c3a70338cee4a91c9e2e4c5e0fcfe:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(schemachange_random_load.go:165).runSchemaChangeRandomLoad: output in run_134446.781924515_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #90038 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema blocks-22.2.0-beta.5 branch-release-22.2.0 release-blocker] - #89493 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema blocks-22.2.0-beta.5 branch-release-22.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ bb4f8cd5396e73c31d370c0d0ac1881263c5156f:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(schemachange_random_load.go:165).runSchemaChangeRandomLoad: output in run_131806.397728074_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #90038 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema blocks-22.2.0-beta.5 branch-release-22.2.0 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ a3b578466bcdcc5d2bbea628d33c337d87b9f4b8:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(schemachange_random_load.go:165).runSchemaChangeRandomLoad: output in run_133511.803815656_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #90038 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.2.0]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.schemachange/random-load failed with artifacts on master @ c5decaeb69155399647909163451f3d8ca7858b6:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(schemachange_random_load.go:165).runSchemaChangeRandomLoad: output in run_135358.653649291_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #90038 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.2.0]

This test on roachdash | Improve this report!