search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.13k stars 3.81k forks source link

compose: TestComposeCompare failed #120869

Closed cockroach-teamcity closed 6 months ago

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ 76e54f8e0db3e6882623d0a2e21142d83b4f1687:

        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     --- FAIL: TestCompare/postgres (12.83s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:201: open /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/_tmp/018812dfcf0ce383ac3e799915a61396/TestComposeCompare1395546253/002/postgres.log: no such file or directory
        test_1        |         compare_test.go:203: compare cockroach1 to postgres: unexpected diff:
        test_1        |               []any(Inverse(func1, []any{
        test_1        |             -   string("010300000000000000"),
        test_1        |             +   string("010300000001000000050000000000000000000840000000000000224000000000000008400000000000000840000000000000184000000000000008400000000000001840000000000000224000000000000008400000000000002240"),
        test_1        |               }))
        test_1        |             cockroach1:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         st_makeenvelope(3.0:::FLOAT8::FLOAT8::FLOAT8, 9.0:::FLOAT8::FLOAT8::FLOAT8, 6.0:::FLOAT8::FLOAT8::FLOAT8, 3.0:::FLOAT8::FLOAT8::FLOAT8)::GEOMETRY
        test_1        |                             AS col16162
        test_1        |                 )
        test_1        |                     AS tab8466
        test_1        |             ORDER BY
        test_1        |                 col16162 DESC;
        test_1        |             
        test_1        |             postgres:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         st_makeenvelope(3.0::FLOAT8::FLOAT8::FLOAT8, 9.0::FLOAT8::FLOAT8::FLOAT8, 6.0::FLOAT8::FLOAT8::FLOAT8, 3.0::FLOAT8::FLOAT8::FLOAT8)::GEOMETRY
        test_1        |                             AS col16162
        test_1        |                 )
        test_1        |                     AS tab8466
        test_1        |             ORDER BY
        test_1        |                 col16162 DESC;
        test_1        |             
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (641.26s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #119902 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.2.3-rc] - #116150 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-master] - #112154 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.2] - #109400 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-36916

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ c013b0d428ae23ac3542d950d9359cfc2a0246a5:

        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:201: open /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/_tmp/018812dfcf0ce383ac3e799915a61396/TestComposeCompare3916548645/002/postgres.log: no such file or directory
        test_1        |         compare_test.go:203: compare cockroach1 to postgres: unexpected diff:
        test_1        |               []any(Inverse(func1, []any{
        test_1        |             -   float32(0),
        test_1        |             +   float32(0.012158541940152645),
        test_1        |                 string("0010000000000000000000000"),
        test_1        |               }))
        test_1        |             cockroach1:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         ts_rank('''RBCWyJF'':311A,468A,709B ''c'' ''jtKGQbFnUw'' ''noFTOWsf'':868C,935A ''whfoLGls'':454A,747C ''xlGkyf'':21A,218C,777A':::TSVECTOR::TSVECTOR::TSVECTOR, '''c'' | ''aonN'' & ''vgQbzwn'' <-> !''MoOlpp'' <-> ''uBRex''':::TSQUERY::TSQUERY::TSQUERY)::FLOAT4
        test_1        |                             AS col38276,
        test_1        |                         B'0010'::BIT(25) AS col38277
        test_1        |                 )
        test_1        |                     AS tab27332
        test_1        |             ORDER BY
        test_1        |                 col38276 NULLS LAST, col38277 ASC;
        test_1        |             
        test_1        |             postgres:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         ts_rank('''RBCWyJF'':311A,468A,709B ''c'' ''jtKGQbFnUw'' ''noFTOWsf'':868C,935A ''whfoLGls'':454A,747C ''xlGkyf'':21A,218C,777A'::TSVECTOR::TSVECTOR::TSVECTOR, '''c'' | ''aonN'' & ''vgQbzwn'' <-> !''MoOlpp'' <-> ''uBRex'''::TSQUERY::TSQUERY::TSQUERY)::FLOAT4
        test_1        |                             AS col38276,
        test_1        |                         B'0010'::BIT(25) AS col38277
        test_1        |                 )
        test_1        |                     AS tab27332
        test_1        |             ORDER BY
        test_1        |                 col38276 NULLS LAST, col38277 ASC;
        test_1        |             
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (655.79s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #119902 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.2.3-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ c013b0d428ae23ac3542d950d9359cfc2a0246a5:

        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:201: open /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/_tmp/018812dfcf0ce383ac3e799915a61396/TestComposeCompare229158440/002/postgres.log: no such file or directory
        test_1        |         compare_test.go:203: compare cockroach1 to postgres: unexpected diff:
        test_1        |               []any(Inverse(func1, []any{
        test_1        |             -   string("010300000000000000"),
        test_1        |             +   string("010300000001000000050000000000000000001C4000000000C069D0400000000000001C40000000000000000000000000000014400000000000000000000000000000144000000000C069D0400000000000001C4000000000C069D040"),
        test_1        |                 nil,
        test_1        |               }))
        test_1        |             cockroach1:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         st_makeenvelope(7.0:::FLOAT8::FLOAT8::FLOAT8, pow(7.0:::FLOAT8::FLOAT8::FLOAT8, 5.0:::FLOAT8::FLOAT8::FLOAT8)::FLOAT8::FLOAT8, 5.0:::FLOAT8::FLOAT8::FLOAT8, 0.0:::FLOAT8::FLOAT8::FLOAT8)::GEOMETRY
        test_1        |                             AS col😠343452,
        test_1        |                         NULL::INT2VECTOR AS """col343453"
        test_1        |                 )
        test_1        |                     AS tab191965
        test_1        |             ORDER BY
        test_1        |                 col😠343452 DESC, """col343453" ASC NULLS LAST;
        test_1        |             
        test_1        |             postgres:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         st_makeenvelope(7.0::FLOAT8::FLOAT8::FLOAT8, pow(7.0::FLOAT8::FLOAT8::FLOAT8, 5.0::FLOAT8::FLOAT8::FLOAT8)::FLOAT8::FLOAT8, 5.0::FLOAT8::FLOAT8::FLOAT8, 0.0::FLOAT8::FLOAT8::FLOAT8)::GEOMETRY
        test_1        |                             AS col😠343452,
        test_1        |                         NULL::INT2VECTOR AS """col343453"
        test_1        |                 )
        test_1        |                     AS tab191965
        test_1        |             ORDER BY
        test_1        |                 col😠343452 DESC, """col343453" ASC NULLS LAST;
        test_1        |             
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (882.47s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #119902 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.2.3-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ 21996d7bf12f05147bea7ea4e0b27b2f9a6e24ed:

        postgres_1    | 2024-03-31 05:31:44.781 UTC [98] ERROR:  function bytes(bytea) does not exist at character 61
        postgres_1    | 2024-03-31 05:31:44.781 UTC [98] HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
        postgres_1    | 2024-03-31 05:31:44.781 UTC [98] STATEMENT:  SELECT
        postgres_1    |         *
        postgres_1    |     FROM
        postgres_1    |         (
        postgres_1    |             SELECT
        postgres_1    |                 NULL::TEXT AS "co'l58187",
        postgres_1    |                 bytes(('\x41'::BYTEA::BYTEA::BYTEA || '\x44'::BYTEA::BYTEA::BYTEA)::BYTEA::BYTEA)::BYTEA AS col58188
        postgres_1    |             FROM
        postgres_1    |                 postgres.public.table2 AS "tab%q31206"
        postgres_1    |             ORDER BY
        postgres_1    |                 "tab%q31206".col2_0 ASC NULLS FIRST
        postgres_1    |         )
        postgres_1    |             AS "tab""31207"
        postgres_1    |     ORDER BY
        postgres_1    |         "co'l58187" NULLS LAST, col58188 NULLS LAST
        test_1        | --- FAIL: TestCompare (826.48s)
        test_1        |     compare_test.go:132: Checking connection to: postgres
        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     --- FAIL: TestCompare/postgres (222.81s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:212: SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         '{"D": null, "F": null, "G": [[8], null, {"D": {"H": {"G": [{"H": {"E": "A", "I": false, "J": {"C": {"A": 0, "H": {"B": null, "D": {"E": 3, "F": [0]}, "H": 1}, "J": {"G": {"B": {"D": true}, "H": false, "I": {"A": {"C": {"G": [{"A": {"D": {"A": {"G": true, "H": null}, "F": {"C": {"E": {"H": {"C": true, "H": {"B": "I"}}}, "J": {"C": {"G": "D"}}}, "E": [false]}}}, "H": "G", "I": {"A": true, "E": "B", "G": "E"}}, [[{"E": false}, {"C": {"D": true, "F": true, "G": true}}], {"G": false, "J": null}, {"C": 2, "E": {"B": null, "G": {"A": {"C": {"B": {"H": {"B": false, "G": {"G": "J"}}, "J": {"G": null, "J": 1}}, "D": "C", "J": true}, "D": {"A": true, "D": false}, "I": true}, "I": {"B": 3}, "J": [7, [{"A": 4}, {"A": true, "D": {"E": {"A": {"C": null, "D": {"I": {"D": true}}, "E": false}, "G": {"B": {"A": true, "B": {"F": {"H": {"E": {"G": [{"I": "H", "J": {"B": "I", "G": {"B": {"A": false, "F": null, "I": "J"}, "C": {"I": true}, "I": {"G": "H"}}, "J": {"B": true, "F": [{"H": {"D": null, "G": 8}, "I": 9}]}}}]}, "F": {"A": "D", "D": {"G": {"B": {"G": {"A": {"J": true}, "E": {"C": null, "G": true, "H": {"G": false, "H": "G"}}}}, "G": {"B": {"D": false, "J": 1}}}, "H": {"D": "E", "I": {"B": false, "C": "I", "D": {"E": [false, true, {"F": [null, 7], "H": false, "I": {"B": true, "G": {"A": null, "E": true, "I": null}}}]}}}}}}, "J": {"I": {"A": 4, "D": "H", "G": {"B": [{"C": {"B": "J", "E": {"F": {"H": false}, "H": [{"E": true, "F": "D"}, {"D": 9, "H": [true, {"D": false, "F": {"D": [false, "F", "A"], "E": {"C": {"B": true, "D": {"J": 4}}, "F": [{"C": null, "D": {"B": null, "C": false, "F": {"G": {"I": false}, "I": 8}}}, {"D": {"I": [false]}, "I": {"E": null, "G": [true, {"E": 1, "G": false, "H": {"C": {"B": true}, "I": {"B": false, "E": [false], "F": {"E": false}}}}, {"E": {"B": 5, "C": {"E": {"B": {"F": null}, "C": {"D": {"C": true, "E": {"F": {"B": {"C": true, "D": {"F": {"B": {"E": ["A", false], "F": null}, "E": [5, "D"]}}}, "D": {"D": true}, "F": [{"A": null}, true, 9]}, "G": "E"}, "F": "F"}, "H": {"B": {"B": 4}, "G": false, "I": 7}}}, "G": [false, "I", true]}, "J": {"C": false}}}], "I": {"C": {"I": {"B": {"B": false, "C": {"B": [{"J": {"D": 9}}, ["G", true, {"I": [null], "J": {"E": null, "F": 4, "I": "F"}}]], "F": {"F": {"A": {"E": [{"F": false, "H": true}], "F": {"B": {"B": true, "J": {"A": {"H": {"A": {"C": {"D": true, "H": {"D": {"D": {"G": true}, "F": false}, "F": "A"}, "I": "A"}, "E": {"B": {"D": null, "E": [null, true]}, "G": {"C": true}, "H": 2}}, "I": true}}}}, "D": null, "F": true}, "J": {"D": [true, false]}}, "G": false}, "I": {"J": {"B": 4}}}}, "G": "I"}, "I": {"A": {"G": {"B": {"C": "I"}, "H": {"H": false, "I": [{"A": {"A": "B"}, "D": null}, false, false], "J": {"H": {"A": 0}}}}, "H": false}}}}, "H": {"B": {"H": {"A": "H", "G": null, "H": false}}, "F": {"D": {"B": {"H": {"H": {"E": {"H": 8, "J": ["J", false]}}}, "J": null}, "C": {"D": [{"G": true}, 5], "G": {"C": true, "H": null}, "J": [{"B": true, "F": [{"C": {"E": null, "G": false, "J": "J"}, "E": {"C": [true, {"C": {"A": null, "F": false, "I": null}}], "D": "H"}, "H": {"G": {"D": null}}}, [{"C": null, "I": {"B": true, "F": {"B": 2, "F": 9}, "I": {"J": "A"}}}], true], "J": true}]}, "D": false}}, "J": [{"C": {"E": 4, "J": false}}, "B", {"B": {"B": [["B", true, null], null, {"G": {"F": "B", "H": 0}, "I": false}], "G": "F", "H": {"D": [{"B": [{"C": true, "J": 2}]}, {"G": {"A": false, "H": "I", "I": "A"}}, {"B": {"D": {"I": null, "J": true}, "E": {"A": [false, {"E": {"A": 0, "I": true}, "F": {"F": 2}, "J": 9}], "E": 4}}, "I": true}], "G": 9}}, "D": null, "I": {"B": {"C": {"A": {"J": {"I": "C"}}, "F": false, "H": [{"A": false, "E": {"B": [null, {"G": false}, "C"], "C": null, "J": null}}, true, {"D": [{"D": [{"C": {"B": {"B": {"C": [1, {"A": true}], "H": null, "J": false}}, "E": false}, "I": "J"}], "G": [{"B": true, "C": {"I": null}, "E": [{"B": {"A": 3, "F": false, "I": 6}, "D": 6}]}, null, {"D": {"D": {"C": "B"}}, "H": 7, "I": true}], "I": true}, 2, 6], "G": [{"A": {"C": {"A": [{"A": 3}, {"A": 5, "D": "D"}], "F": {"F": null, "I": null, "J": null}, "I": true}, "E": null, "F": "B"}}, {"B": [false, null]}], "H": {"A": {"A": true, "F": false, "H": {"B": {"B": {"C": 2, "E": {"E": {"H": 2}}}}}}, "C": false, "E": {"A": {"A": "E", "C": {"B": null}, "E": {"A": true, "E": {"B": 8, "H": false}}}, "F": {"D": "E", "G": [null, true], "I": 6}}}}]}, "G": true, "J": {"E": [{"D": {"G": false}}], "I": {"I": {"B": true, "G": {"D": {"C": {"F": {"A": {"J": {"A": {"J": {"B": "H", "D": true}}, "C": "A"}}, "B": "H"}, "H": {"F": ["A"]}}, "H": [{"B": {"D": null, "H": {"B": {"E": {"A": {"B": {"B": false, "H": 5}, "D": {"C": false, "H": 7}}}, "F": [true, [4, 0, {"D": "C"}], [null, {"E": {"B": {"H": 6}, "D": false, "G": "C"}, "F": false, "I": {"A": {"A": {"B": [{"B": {"G": {"C": {"F": null}, "I": 0}, "H": false, "I": {"G": {"B": 5, "I": {"B": {"B": null, "C": {"B": "C", "F": {"B": {"B": "J"}, "H": null}}, "F": {"C": false}}, "C": true, "F": [null, false, {"C": true, "H": {"A": [{"A": {"D": {"B": {"F": null, "H": false}, "C": null, "H": "J"}}, "E": 7, "F": false}], "E": false, "I": false}}]}}}}, "D": 0}, {"J": false}], "F": true, "G": [false, {"C": false, "E": false}]}, "G": [false, ["I", 2], 1]}, "C": {"C": {"E": "E", "H": [{"F": {"B": false, "C": [[{"A": null, "F": true}], "D", 3], "H": {"A": "D"}}, "G": {"B": {"H": true}, "D": {"E": null, "H": "J"}, "G": null}}, {"I": "J"}, {"B": 3}], "I": [{"C": {"C": 6}, "J": true}]}, "E": false, "F": {"F": {"A": 9, "C": "B"}, "G": false, "I": {"C": false, "I": false}}}, "G": true}}, null]], "G": false}, "H": {"B": {"D": {"D": "G", "E": {"C": {"B": {"A": 1, "C": "B", "J": {"A": 1, "B": {"B": {"E": true, "G": "C", "I": false}, "E": "C", "J": true}}}, "H": {"C": null, "J": false}, "J": "G"}, "F": null}, "J": true}, "E": false, "F": "B"}, "G": {"E": {"E": {"C": false, "D": {"B": {"A": null, "D": {"E": {"B": {"C": 5, "I": null}, "H": {"E": {"F": null, "H": [false], "J": 0}, "H": {"F": {"B": null}}}, "I": {"F": {"A": null, "D": {"A": false, "G": {"A": 4, "J": [null, [{"D": {"A": false, "D": "B", "E": true}, "G": [null, {"C": {"E": {"E": true, "G": 6}, "G": 7}}, [false]]}], {"F": false}]}, "J": 4}}, "I": false}}, "J": false}, "E": {"E": false}}, "E": {"A": null}, "H": true}, "J": "D"}}, "I": null}, "I": null}}, "I": {"C": {"D": "E", "E": null, "I": 9}, "D": false}}, "F": {"H": {"E": {"B": {"C": "C", "E": 5, "H": false}, "D": [3, {"C": 8, "H": [true, null]}], "E": {"D": null, "E": null}}, "H": false}}}, "H"]}}}}}}, "C": "I"}}]}, "I": 7}}}]}, "H": 8}}, true], "J": [[false], {"A": {"A": [[{"D": {"C": {"F": 8, "H": {"B": {"C": {"E": "F"}, "J": true}}}, "F": 5, "J": {"E": false, "G": "G"}}}, true, 6]], "D": "C"}, "I": {"E": "D"}}, 1]}]}, "H": {"D": false, "E": {"C": "G", "I": {"D": true}}, "J": true}}}]}}}}}, "C": 8}, "D": [{"A": {"C": {"I": [[{"B": "J", "E": 0}, {"B": 4}]], "J": {"G": {"C": {"G": 4}, "J": [{"C": {"C": {"H": 8}, "F": 1, "H": null}, "J": {"I": {"E": {"D": {"J": false}, "F": "A"}}}}]}, "I": {"A": "F"}}}, "J": null}, "F": {"I": true}, "G": true}, [false, {"E": [false]}, {"C": [null], "E": [[[true]], {"H": false}, {"G": ["I", {"G": {"A": {"E": true, "H": true, "J": {"G": {"C": {"B": "A", "I": [{"A": null}, {"D": false, "E": {"J": 4}, "G": "D"}, "A"]}, "D": {"D": null}}, "H": 1, "I": {"A": {"E": false, "G": [true, true, 6]}, "E": {"A": {"C": 8, "I": {"F": [true, {"B": {"G": null}}, {"G": {"D": {"B": "I"}, "E": "B"}, "H": "F", "I": null}], "G": true, "H": {"D": "E"}}}, "G": {"C": {"C": null, "D": true, "J": {"A": {"G": {"C": true}}, "B": true, "J": null}}, "F": null, "H": 9}}, "G": true}}}, "D": null, "I": 4}, "H": {"D": {"F": {"F": {"A": true}}, "J": false}, "E": "J", "I": null}}]}], "G": "D"}]], "G": 4}}}, "I": [{"J": true}]}, {"G": 6, "H": true}]]}}}]], "I": null, "J": [[{"D": false, "G": {"A": false, "F": [[{"C": {"J": {"A": [{"B": 4, "D": {"H": {"C": {"B": {"D": ["D", {"D": {"C": true, "D": {"C": null, "F": {"A": true, "C": 8, "D": {"F": true, "J": {"D": [true, {"I": [{"J": true}]}], "I": "H"}}}}}}, 0], "F": null}, "C": [[null, [true, {"C": {"H": {"E": null}, "I": {"A": {"I": null}, "D": ["E"], "I": [true, {"D": {"D": "D", "I": {"H": "J", "I": {"E": {"C": {"D": "J"}, "E": null}}}}, "J": [null]}]}}, "I": true}, 6], {"E": {"H": "F"}}]]}, "H": {"A": "D", "F": [{"H": false}, "E"], "J": null}}}}, {"D": "E"}], "C": {"A": "A", "B": {"C": true, "D": true, "E": [[{"G": "E"}, {"F": {"D": {"E": {"H": {"D": "J", "H": {"C": {"C": true}, "E": 8, "G": true}}}, "H": {"E": "C", "I": null}}}, "G": {"G": {"E": "D", "I": [{"B": 0, "D": {"A": "A", "F": true}, "G": {"G": [{"C": {"I": 1, "J": [[true, {"E": null}]]}}], "H": "G"}}, [null, {"E": true}]]}, "J": true}}]]}, "I": "H"}, "H": {"E": {"J": {"A": "I", "F": false}}, "F": {"B": false, "G": [["J"], {"A": true, "G": 5, "H": "C"}], "I": false}}}}, "H": "J", "I": {"D": {"H": {"I": {"H": true}}}, "E": {"B": null, "F": false}}}], {"J": 7}]}}], {"A": null, "E": {"C": {"G": "H"}, "G": {"E": 7, "I": [{"C": null, "E": {"D": {"B": [{"A": "E"}], "H": {"D": true, "H": {"G": 2}}, "I": {"B": false}}}, "G": {"F": true}}, null, true], "J": {"J": false}}}, "J": 1}, 8]}, "I": {"F": "A", "H": [{"G": {"A": {"B": {"C": {"A": [false, null, {"D": "D", "F": {"H": false}, "G": true}], "C": "H", "G": null}, "I": null, "J": {"B": ["F", false]}}, "F": 4}, "C": "I", "D": null}, "J": null}, true]}}, "I": [null]}}}}, "H": true, "I": "B"}}}, "A", {"C": "A", "D": {"D": [[{"C": false, "G": "G", "J": true}, {"A": true, "I": true, "J": null}, 0], true], "G": {"D": {"D": {"E": true, "H": [{"C": {"B": 7, "H": "I"}, "H": {"I": 1}}], "I": null}, "J": {"H": {"J": null}}}, "H": {"E": [7, true, true]}}, "J": 3}}], "I": {"J": [{"G": {"D": true, "J": "H"}, "I": true, "J": {"C": [{"A": null, "I": "G"}], "D": {"A": true, "G": null, "I": false}, "H": [[false, true], {"F": "B", "I": true}, {"I": true}]}}, {"C": {"B": {"B": true, "I": [false, {"F": false, "J": [{"B": null, "I": 0}, {"E": {"A": false, "B": {"C": {"E": {"D": false}, "I": 7}}, "I": {"A": {"D": "D", "G": null, "H": {"F": {"F": 4, "I": true}}}, "D": false, "G": null}}}, false]}, {"H": [true, {"C": null}, false]}], "J": 4}, "C": null}, "D": [{"I": "I"}, null, {"A": false, "J": [null, true, {"I": true}]}]}, {"C": {"H": false, "I": {"B": null, "C": "C"}}, "E": {"C": {"B": {"B": false, "D": false, "F": "G"}, "I": null}}}]}}, "I": null}, "F": {"F": true, "I": false}, "G": "A"}]}':::JSONB::JSONB
        test_1        |                             AS 🙀col58236
        test_1        |                 )
        test_1        |                     AS tab31239
        test_1        |             ORDER BY
        test_1        |                 🙀col58236 DESC NULLS LAST
        test_1        |         compare_test.go:213: cockroach1: ping: conn closed
        test_1        | FAIL
        postgres_1    | 2024-03-31 05:34:31.304 UTC [98] LOG:  could not receive data from client: Connection reset by peer
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (850.49s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #119902 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.2.3-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ 21996d7bf12f05147bea7ea4e0b27b2f9a6e24ed:

        test_1        |     compare_test.go:132: Checking connection to: postgres
        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     --- FAIL: TestCompare/postgres (0.00s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        | panic: geos: no locations to init GEOS [recovered]
        test_1        |     panic: geos: no locations to init GEOS
        test_1        | 
        test_1        | goroutine 151 [running]:
        test_1        | testing.tRunner.func1.2({0x2274240, 0xc000d54500})
        test_1        |     GOROOT/src/testing/testing.go:1545 +0x238
        test_1        | testing.tRunner.func1()
        test_1        |     GOROOT/src/testing/testing.go:1548 +0x397
        test_1        | panic({0x2274240?, 0xc000d54500?})
        test_1        |     GOROOT/src/runtime/panic.go:914 +0x21f
        test_1        | github.com/cockroachdb/cockroach/pkg/sql/randgen.encodeInvertedIndexHistogramUpperBounds(0x46d77a0?, {0x320ec98?, 0xc000d5a900?})
        test_1        |     github.com/cockroachdb/cockroach/pkg/sql/randgen/mutator.go:375 +0x436
        test_1        | github.com/cockroachdb/cockroach/pkg/sql/randgen.randHistogram(0x31cd5a0?, 0x46d77a0)
        test_1        |     github.com/cockroachdb/cockroach/pkg/sql/randgen/mutator.go:308 +0x12c
        test_1        | github.com/cockroachdb/cockroach/pkg/sql/randgen.statisticsMutator.func1(0xc000d2ac00)
        test_1        |     github.com/cockroachdb/cockroach/pkg/sql/randgen/mutator.go:225 +0x7f
        test_1        | github.com/cockroachdb/cockroach/pkg/sql/randgen.statisticsMutator(0xc000d39920, {0xc000d540a0?, 0x2, 0xc000d39920?})
        test_1        |     github.com/cockroachdb/cockroach/pkg/sql/randgen/mutator.go:258 +0x89d
        test_1        | github.com/cockroachdb/cockroach/pkg/sql/randgen.MultiStatementMutation.Mutate(0x2072780?, 0x24774b0?, {0xc000d540a0?, 0x4a0a168?, 0x2?})
        test_1        |     github.com/cockroachdb/cockroach/pkg/sql/randgen/mutator.go:96 +0x28
        test_1        | github.com/cockroachdb/cockroach/pkg/sql/randgen.RandCreateTables(0xc00075d040?, {0x24774b0, 0x5}, 0x2, 0x80?, {0xc000d0ba40, 0x3, 0x7f2128405108?})
        test_1        |     github.com/cockroachdb/cockroach/pkg/sql/randgen/schema.go:93 +0x131
        test_1        | github.com/cockroachdb/cockroach/pkg/internal/sqlsmith.randTablesN(0xc000090cc8?, 0x1?, {0x0, 0x0})
        test_1        |     github.com/cockroachdb/cockroach/pkg/internal/sqlsmith/setup.go:118 +0x15e
        test_1        | github.com/cockroachdb/cockroach/pkg/internal/sqlsmith.randTables(0xc00017ca00?)
        test_1        |     github.com/cockroachdb/cockroach/pkg/internal/sqlsmith/setup.go:98 +0x2f
        test_1        | github.com/cockroachdb/cockroach/pkg/internal/sqlsmith.init.wrapCommonSetup.func28(0x265dc28?)
        test_1        |     github.com/cockroachdb/cockroach/pkg/internal/sqlsmith/setup.go:51 +0x1b
        test_1        | github.com/cockroachdb/cockroach/pkg/compose/compare/compare.TestCompare.func2(0xc000d10ea0)
        test_1        |     github.com/cockroachdb/cockroach/pkg/compose/compare/compare/compare_test.go:143 +0xe2
        test_1        | testing.tRunner(0xc000d10ea0, 0xc000d39890)
        test_1        |     GOROOT/src/testing/testing.go:1595 +0xff
        test_1        | created by testing.(*T).Run in goroutine 9
        test_1        |     GOROOT/src/testing/testing.go:1648 +0x3ad
        compare_test_1 exited with code 2
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 2
--- FAIL: TestComposeCompare (19.51s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #121592 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.2] - #121531 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-24.1] - #121530 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-master] - #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #119902 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.2.3-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ c9fff6f54f3094ca4eacd788dabb6f8ab61cacfc:

        postgres_1    |         col4825 ASC,
        postgres_1    |         col4826 DESC NULLS FIRST,
        postgres_1    |         col4827 DESC NULLS FIRST,
        postgres_1    |         col4828 DESC NULLS LAST,
        postgres_1    |         col4829 ASC NULLS LAST,
        postgres_1    |         col4830,
        postgres_1    |         col4831 ASC NULLS FIRST,
        postgres_1    |         col4832 NULLS LAST,
        postgres_1    |         "!co%ql4833" NULLS FIRST,
        postgres_1    |         "cOl4834" NULLS FIRST,
        postgres_1    |         col4835 NULLS LAST,
        postgres_1    |         "c ol4836",
        postgres_1    |         col4837 NULLS FIRST,
        postgres_1    |         " col4838" DESC NULLS LAST,
        postgres_1    |         col4839 DESC NULLS LAST
        postgres_1    | 2024-04-05 05:21:33.281 UTC [79] ERROR:  could not identify an ordering operator for type void at character 69
        postgres_1    | 2024-04-05 05:21:33.281 UTC [79] HINT:  Use an explicit ordering operator or modify the query.
        postgres_1    | 2024-04-05 05:21:33.281 UTC [79] STATEMENT:  SELECT * FROM (SELECT NULL::VOID AS "c�ol4840") AS tab2696 ORDER BY "c�ol4840" ASC
        test_1        | --- FAIL: TestCompare (763.96s)
        test_1        |     compare_test.go:132: Checking connection to: postgres
        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     --- FAIL: TestCompare/postgres (158.91s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:212: SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         '{"D": {"F": {"A": {"B": {"E": 7, "G": {"F": [{"E": {"E": true, "H": [true, 6, false]}}, {"G": true, "I": {"A": {"E": {"A": {"G": false, "H": {"A": "E"}, "I": {"A": {"I": 0}, "F": null}}}, "H": {"B": {"E": null, "F": 9, "G": true}, "F": {"D": [{"C": 8, "I": "G", "J": 1}]}}}, "B": null, "H": {"J": {"A": {"C": {"G": true, "H": [{"I": null}, {"C": "B", "H": [true, false, {"H": true, "I": 7}], "J": null}]}, "F": false}, "H": {"A": {"F": null}, "J": null}}}}}, {"C": {"C": 2, "F": false, "I": {"D": {"H": {"C": {"E": true}}, "I": null}}}, "F": "G"}]}}, "E": [{"D": {"D": [{"J": {"B": 3, "C": {"B": "D", "F": {"F": {"D": {"C": true, "E": true, "G": null}, "G": {"A": [false, false, {"E": null, "G": true, "I": {"C": false, "J": {"A": "B", "B": "F", "G": {"A": ["H", null, false], "D": {"D": [3, {"B": {"I": {"D": {"H": {"C": {"E": {"B": true, "F": {"A": true, "E": true}, "I": [{"B": false, "E": {"A": {"E": {"B": {"D": 7, "E": 0, "J": true}, "J": [{"E": false}, true, true]}, "H": {"I": [{"C": [[[false, true, {"C": true, "H": false}], {"I": "J"}, {"I": {"B": false, "H": {"H": {"H": null}}}, "J": null}], 6], "D": true, "E": {"A": [[{"C": [true], "H": "C"}], {"B": "E", "J": null}], "B": false, "D": [null, {"I": {"B": "A", "C": true}}, {"F": {"H": {"A": {"C": 0, "H": true}, "E": "A", "F": "E"}, "J": 6}}]}}, 1, [{"F": {"C": false, "I": {"C": {"I": true}, "H": true}}, "J": true}, {"B": {"E": {"F": null, "I": {"A": false, "J": {"G": {"D": "D", "F": true, "J": {"G": {"C": ["F", {"F": {"B": [{"A": {"C": null}, "B": {"C": [["A"]], "D": {"G": false, "I": 3, "J": false}, "F": "F"}, "D": 9}]}, "G": {"H": {"I": "D", "J": [[false, false, {"D": null, "F": true, "G": true}], 2, [false, null]]}}, "H": true}]}}}, "H": {"I": null}}}}, "J": 1}, "G": 0}]], "J": [{"H": {"E": false, "F": true}}, false]}, "I": false}}, "G": [{"C": {"B": {"A": false, "C": {"G": 1}, "H": {"F": {"A": true, "C": {"F": true, "G": {"H": {"B": null, "D": false}, "J": {"J": {"B": false, "C": null, "G": 5}}}, "J": false}, "H": {"C": ["J", 9, true], "I": {"E": false, "F": null, "J": false}}}}}, "C": "J"}, "H": "I"}]}, {"D": {"D": null}, "E": null}, {"B": [[1], "F", "D"], "F": "D"}]}, "G": {"G": null, "J": null}, "J": null}, "G": {"G": {"E": null}}, "J": "F"}}}}, "C": {"B": {"A": 9, "F": {"C": null}}, "E": true, "H": true}, "G": {"B": {"A": {"B": {"C": {"B": true, "F": true}, "I": {"F": "A"}, "J": {"A": 7, "F": null}}}, "E": {"A": {"A": {"E": null}, "H": 4}, "G": {"I": 0}}, "H": "F"}, "J": [[{"J": 1}]]}}, false], "I": "F"}}}}}], "C": [{"A": 8, "H": [{"D": "J", "G": null}, 3, 6]}, {"B": {"A": false}, "E": {"I": true}}, [{"A": false, "H": {"D": {"E": {"D": {"A": {"G": true}, "G": false, "J": {"A": false, "C": {"G": {"G": {"E": {"D": {"A": {"H": true, "I": [{"A": {"E": "H", "F": [{"C": false, "F": {"D": null, "I": {"C": false, "D": [{"C": 0}, {"A": null, "C": {"A": {"H": {"A": {"D": {"G": {"A": 4, "H": [{"H": "B"}], "J": true}}}, "F": [{"A": false, "I": {"C": null, "D": {"B": true, "E": null}, "E": true}, "J": {"E": [2]}}, 3, {"C": {"A": {"B": null, "D": {"A": null, "F": {"B": {"B": {"H": null, "I": {"H": {"C": false}}}}, "C": false}}}, "I": {"A": [null], "H": "I"}}, "H": false, "I": true}]}, "J": {"C": 8, "E": true, "I": {"G": false}}}, "C": {"B": {"C": {"A": 9}}, "J": false}, "G": {"D": {"A": {"B": {"E": 5}, "C": "J"}}}}}], "J": {"H": {"G": true}, "I": {"A": {"I": {"E": null, "H": 2}}, "C": null}}}}, "G": null}, false], "G": {"D": {"G": {"D": null, "H": {"G": {"A": {"F": null}, "D": "A", "H": "E"}, "H": true}}, "H": false, "I": null}, "I": {"H": true, "J": [{"C": 2, "F": {"C": {"I": "C", "J": {"C": [false, 9]}}, "E": {"C": null, "D": {"C": [[{"D": {"E": [{"I": null}, false, {"B": 4, "C": true}]}, "J": 1}, null], {"F": 8, "H": 0}], "F": false}, "G": {"H": {"D": [null, {"A": "B", "C": null}, null], "J": [true, true, {"A": [{"I": {"G": true}}, {"A": true, "E": true}, [[3, {"A": 2, "H": {"H": true}}, {"C": false, "I": true, "J": "G"}], {"F": [{"G": {"A": {"E": false}, "F": {"E": [{"F": {"C": "J", "D": {"D": {"F": true, "G": {"B": {"C": null}, "C": {"H": {"A": {"A": null, "D": {"E": {"C": 8, "F": ["A", {"B": false, "G": "A"}, true], "J": [false, null]}, "J": {"B": null, "F": null}}, "H": {"C": 0, "F": "J", "H": {"A": false, "C": {"H": "F"}, "I": 4}}}}, "I": {"B": true, "J": 5}}}, "I": {"H": [{"D": 8}, [{"B": [null, {"F": [{"H": true, "J": {"A": {"A": {"H": [{"H": 8}, {"E": {"C": {"B": "B", "C": {"A": {"F": true, "G": {"A": false, "C": {"C": null, "F": {"G": null, "H": {"B": false, "H": {"B": {"H": ["F", {"A": {"C": 4, "E": "A"}, "F": [{"A": true}, {"C": [true], "J": "C"}], "J": 3}], "J": {"A": {"D": {"G": {"J": {"C": 6, "G": {"A": null, "D": {"F": {"D": {"C": {"E": false, "G": {"B": "I", "D": {"A": {"B": false, "D": false}, "G": "F"}, "J": {"C": false, "J": {"F": "A"}}}}}, "E": {"D": {"D": {"H": {"D": "I"}}, "I": {"A": true, "I": {"B": "C"}}}, "G": {"B": 5, "F": null}}, "J": false}, "J": "E"}, "J": [null, null]}}}}, "I": {"D": [false, {"D": {"E": "D"}, "F": true, "H": "B"}, {"A": ["B"], "B": 1}]}}, "E": [null, false]}}}}, "J": 8}}}}, "E": [{"E": true, "F": true, "I": {"C": false}}, {"J": 1}, null]}, "I": {"A": {"F": {"A": 0, "J": {"F": true, "H": {"G": null}}}}, "H": false, "J": null}}, "F": [4, {"H": "C"}]}, "G": false}, [{"A": 8, "D": true}, null, null]]}, "G": false, "H": "D"}, "D": true, "F": 8}}]}, 3]}, false], [false, 7, "A"]]}}, "I": true}, "G": {"E": true}}, "J": false}, {"D": true}], "G": {"A": 7, "E": "D", "G": null}}, "H": {"G": "B"}}, "H": false}, {"C": {"I": [{"B": [[[{"E": {"C": 0, "F": true, "G": true}, "J": {"A": {"E": {"G": 7}, "H": true}, "G": "C", "J": 2}}], "G"], null], "D": "G", "F": 7}], "J": {"G": false, "J": {"J": [{"F": {"F": [{"D": {"A": null, "D": null}}, {"A": {"F": false, "H": {"C": {"C": 0}, "F": null, "I": false}}, "D": [[null, null]], "F": 7}, {"B": {"F": null}, "H": 3}]}}, [{"B": [{"B": 7, "G": {"H": {"G": {"C": {"I": false}, "H": {"C": "D"}}}}}], "C": 1}, [{"G": {"C": {"D": {"A": true, "D": "G", "F": true}, "J": {"D": 7, "F": false, "I": {"E": null}}}}}, [{"D": {"A": false, "E": {"A": {"H": "F", "J": {"D": {"E": {"F": [5], "I": {"B": {"C": false, "G": 7}, "C": "D", "F": true}}, "H": [{"D": true}]}, "G": false}}, "B": null, "C": 4}, "H": false}, "H": null}, 5]]]]}}}, "D": {"C": null, "I": null}}], "G": {"D": {"F": {"H": {"E": null, "G": {"B": {"C": null}, "J": {"I": false}}, "J": null}}, "H": 9}}}, false]], "D": [false], "I": null}]}}}}, "H": null}, null]}}}, "B": {"G": true, "H": {"D": false, "H": {"H": {"B": {"J": "B"}}}}}, "J": null}, false]}, "F": false}, "I": false}, "H": {"F": null, "I": "H"}, "J": "D"}}}, "E": [{"H": false, "I": {"I": "C"}}, false, false]}}}, "F": 0}, "G": {"A": {"F": [null], "H": {"G": {"H": [{"C": [{"D": {"A": [8], "C": {"B": null, "D": false, "H": {"A": [[1, null, {"E": false, "J": false}]], "E": {"E": 7}}}, "F": {"H": false}}, "I": {"A": false}}, "G", [null, {"D": true, "G": {"B": null}, "I": {"A": false}}]], "D": null, "G": false}, [9], {"B": 0, "G": {"J": {"G": {"A": [null, true, {"C": "H"}], "J": 7}, "H": 1}}, "I": {"D": {"D": {"B": {"A": {"E": {"A": {"A": {"A": {"A": true, "G": false, "J": false}, "C": {"B": 7}, "G": {"E": "G", "F": "F", "H": {"A": "G", "I": null}}}, "J": null}, "D": {"D": {"C": true, "D": {"B": true, "G": false, "J": true}, "H": "A"}, "H": {"G": false}}}, "F": {"F": {"I": null}}}, "I": [{"B": null, "E": "H", "G": 3}, {"I": {"A": true, "H": {"B": {"H": "A", "I": {"C": 1, "G": [5]}}}, "I": {"A": [{"A": null, "D": {"C": false}, "J": {"F": 6}}], "D": {"G": {"F": {"B": 7, "D": true}, "G": [{"E": {"E": true, "G": {"A": null}}}]}}, "E": false}}}], "J": 3}, "D": {"B": {"A": {"A": null, "D": null}, "B": {"H": {"D": {"E": false}}}, "C": {"D": true, "I": false, "J": "A"}}}, "I": false}, "F": [{"C": 3, "J": null}, {"F": {"C": true, "D": "F", "G": false}}]}, "E": {"H": false, "I": null}, "F": false}}]}}, "I": {"A": false, "B": 3}}, "G": 0}}}, {"F": {"C": [true, false, 5], "F": {"A": true, "D": {"D": {"B": [{"J": {"J": {"C": {"I": true}, "G": {"E": null, "F": true, "H": false}}}}, "F"], "E": {"E": true, "J": "C"}, "F": {"I": {"C": true, "D": true, "E": true}}}, "G": 4, "I": [1, 0]}, "E": {"G": {"G": {"C": {"A": ["H"], "D": {"C": {"G": false, "H": [true]}, "D": {"B": {"A": [{"C": {"A": {"C": "J", "H": null, "J": null}}, "H": 0}, [null, {"D": 0}, true]]}, "G": true}, "J": ["C", "H", 6]}, "J": {"G": false, "I": false}}, "I": [9, 4, 2]}}, "I": [5]}}, "G": {"F": {"A": true, "E": {"D": "E"}}}}}, false]]}, "I": true}, "I": {"H": false, "I": false}}}, "I": [false, {"B": {"A": "H", "E": "F", "F": {"G": 3}}, "C": false, "I": false}, {"C": null, "F": "H"}]}}, true, [null, {"E": {"H": true, "J": {"C": {"F": {"C": {"H": {"D": false, "H": [{"F": [6, true], "I": "G", "J": {"B": {"C": {"B": null, "E": null, "H": [true, false]}, "G": [["D", {"B": [{"B": {"B": null, "C": {"A": 3, "E": false, "H": null}}, "I": "A"}, true], "J": {"A": {"C": {"C": 6}, "D": {"G": {"D": 2}}, "E": [false]}, "C": 8, "J": "F"}}, false], {"G": {"I": ["G", {"F": 0, "G": true, "H": 8}]}}]}, "D": "A", "J": {"B": true, "I": {"A": "F", "F": {"C": 8}}}}}, {"G": false}, {"D": true, "F": [[[null, {"I": true}]]], "I": {"F": "G", "H": [true, false, [[{"B": {"C": {"E": null, "G": false}, "E": {"C": [[{"E": {"D": false}, "G": 8, "H": 3}], "F"], "E": true}, "F": false}, "C": {"D": {"D": 5}, "H": false, "J": {"D": {"I": [{"B": true, "G": null}, "B", false]}}}, "J": null}], true]]}}]}}, "F": {"D": {"B": "H", "J": null}, "E": false}}}, "H": false}}, "J": "C"}]], "E": {"A": null, "D": {"C": false, "I": {"H": {"A": null, "B": 1}, "I": false}}, "I": [[{"F": {"F": [false, [{"I": "H"}], "F"], "G": {"B": {"J": true}, "D": {"A": {"J": [{"H": {"G": {"H": {"A": [[{"E": 4, "I": true}, true, [null]], {"B": {"A": 6, "J": {"E": {"B": false, "F": "G"}}}, "D": {"H": {"G": "G", "H": 9, "J": {"B": {"H": {"C": {"E": "F", "F": [{"F": {"A": {"F": 9, "G": null, "H": null}}, "H": {"C": {"D": {"A": false}}, "J": true}}, [{"C": false, "D": ["J"]}], [{"D": 8, "H": true}, [4, [null, {"F": true}, false], {"G": [{"F": {"A": true, "D": "E", "J": ["F", 3, 7]}}, 3]}]]], "J": {"G": {"F": false}}}, "F": "A", "J": [true, "E", false]}, "I": {"B": {"G": 7, "J": {"C": {"I": 1, "J": {"J": 9}}, "G": false}}}}, "E": false, "G": {"A": true, "F": 7, "G": false}}}}}], "G": false, "H": {"C": {"D": false, "E": false, "I": 6}, "E": {"A": null, "I": "E"}, "J": {"D": {"D": true, "E": "I"}, "H": false}}}}}, "I": false}, {"B": true, "D": {"B": false, "C": "H", "J": 2}}]}, "F": 2}}, "J": {"G": {"H": "C"}}}, "H": null}], true, {"C": {"A": {"D": [6], "E": {"G": null, "I": "F"}}, "D": {"D": {"E": [{"A": [{"A": {"D": true, "G": {"F": {"F": {"B": 2, "D": [true]}}, "J": 4}}}, 2], "B": false, "G": ["G"]}, null, {"B": null, "F": {"A": false, "I": {"G": false}}, "H": null}], "F": {"D": null, "E": null}, "J": [9, {"B": ["D", {"C": "F", "E": [true]}, {"A": {"B": false, "H": {"B": null, "H": [2, ["D"], true]}, "J": {"E": true}}}], "E": "I", "F": true}]}, "G": true, "J": null}}, "J": {"B": 9, "D": false}}]}, "G": {"D": "A", "J": true}}, "G": {"A": "I", "D": true}}], "J": 1}, "C": "F"}, "J": false}, "I": {"F": "D", "G": {"B": {"B": 0, "H": true}, "F": {"C": {"D": "B"}, "E": {"A": true, "C": {"C": {"B": false}, "E": false, "I": [{"B": {"C": {"F": false}}, "G": {"C": {"C": {"B": true, "F": "H"}, "D": null, "I": {"C": null, "D": "D", "J": {"A": {"C": false}}}}, "D": {"A": [{"F": null, "G": {"A": null}}, {"D": null, "E": {"B": {"C": true, "I": {"E": {"H": {"A": {"G": "A", "H": {"A": true, "H": {"A": {"A": false, "G": null, "J": {"J": null}}, "C": {"A": null, "C": true}, "I": "F"}, "I": {"F": [{"C": {"C": [[[{"B": {"E": 0, "F": {"B": false}, "G": {"A": {"F": null}, "I": {"D": {"B": 4, "C": "F", "F": false}, "J": [null, false, null]}}}, "F": false, "I": {"A": "D"}}, "F", {"B": false, "H": 3}], {"I": {"D": [{"F": {"A": 1}, "G": null}, null], "J": {"C": "F"}}, "J": {"J": 3}}, {"A": [false, 3]}], {"C": null, "G": {"A": true, "I": "J"}}, null], "G": false, "H": "E"}, "G": 3, "J": [true, {"E": [null, null]}, {"F": 1, "I": null}]}]}}, "J": 9}, "B": false}, "J": [true, {"J": {"A": {"C": null, "I": null}, "E": null}}]}, "F": {"B": {"C": {"B": {"B": {"J": true}, "D": [{"C": 1, "H": {"D": {"D": [true]}}, "J": {"F": null, "H": "F"}}, ["I", {"D": true, "J": "F"}], true]}, "F": {"H": null}, "G": true}, "D": null}}}, "J": [null]}, "H": 4}}], "C": {"A": "G", "B": {"E": {"E": true, "F": {"A": 6, "D": {"A": true, "G": [{"D": {"D": false, "E": 6, "I": false}, "I": "B"}], "J": {"B": true, "E": {"B": true, "G": null, "J": null}}}, "J": true}}, "G": 3, "H": null}, "J": [{"D": false, "E": [{"B": {"D": {"C": 6, "D": true}}, "J": false}, 3]}]}, "D": true}}}, [{"B": "A", "E": null}], {"B": true, "D": false, "E": {"A": 3, "E": [true], "H": null}}]}, "J": 8}}, "G": {"A": {"C": {"E": {"B": "J", "E": {"C": {"D": false}, "F": true}}}}, "F": {"D": {"D": {"I": 0}}}}}}}':::JSONB::JSONB
        test_1        |                             AS "c%pol4877"
        test_1        |                 )
        test_1        |                     AS """tab""2723"
        test_1        |             ORDER BY
        test_1        |                 "c%pol4877" DESC NULLS LAST
        test_1        |         compare_test.go:213: cockroach1: ping: conn closed
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (788.36s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #121592 compose: TestComposeCompare failed [no locations to init GEOS] [C-test-failure O-robot T-dev-inf branch-release-23.2] - #121531 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-24.1] - #121530 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-master] - #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ c9fff6f54f3094ca4eacd788dabb6f8ab61cacfc:

        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     --- FAIL: TestCompare/postgres (77.20s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:201: open /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/_tmp/018812dfcf0ce383ac3e799915a61396/TestComposeCompare4289381550/002/postgres.log: no such file or directory
        test_1        |         compare_test.go:203: compare postgres to cockroach1: unexpected diff:
        test_1        |               []any(Inverse(func1, []any{
        test_1        |             -   string("010300000001000000050000000000000000001C4000000000000010400000000000001C40000000000000104000000000000010400000000000001040000000000000104000000000000010400000000000001C400000000000001040"),
        test_1        |             +   string("010300000000000000"),
        test_1        |               }))
        test_1        |             postgres:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         st_makeenvelope(7.0::FLOAT8::FLOAT8::FLOAT8, 4.0::FLOAT8::FLOAT8::FLOAT8, 4.0::FLOAT8::FLOAT8::FLOAT8, 4.0::FLOAT8::FLOAT8::FLOAT8)::GEOMETRY
        test_1        |                             AS col81259
        test_1        |                 )
        test_1        |                     AS "t?ab53646"
        test_1        |             ORDER BY
        test_1        |                 col81259 DESC;
        test_1        |             
        test_1        |             cockroach1:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         st_makeenvelope(7.0:::FLOAT8::FLOAT8::FLOAT8, 4.0:::FLOAT8::FLOAT8::FLOAT8, 4.0:::FLOAT8::FLOAT8::FLOAT8, 4.0:::FLOAT8::FLOAT8::FLOAT8)::GEOMETRY
        test_1        |                             AS col81259
        test_1        |                 )
        test_1        |                     AS "t?ab53646"
        test_1        |             ORDER BY
        test_1        |                 col81259 DESC;
        test_1        |             
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (709.00s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #121592 compose: TestComposeCompare failed [no locations to init GEOS] [C-test-failure O-robot T-dev-inf branch-release-23.2] - #121531 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-24.1] - #121530 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-master] - #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ c9fff6f54f3094ca4eacd788dabb6f8ab61cacfc:

        postgres_1    |         "c%vol167046" ASC
        postgres_1    | 2024-04-07 05:23:51.567 UTC [78] ERROR:  could not identify an ordering operator for type void at character 325
        postgres_1    | 2024-04-07 05:23:51.567 UTC [78] HINT:  Use an explicit ordering operator or modify the query.
        postgres_1    | 2024-04-07 05:23:51.567 UTC [78] STATEMENT:  SELECT
        postgres_1    |         *
        postgres_1    |     FROM
        postgres_1    |         (
        postgres_1    |             SELECT
        postgres_1    |                 'B'::TEXT::TEXT AS "co'l167066",
        postgres_1    |                 tab98272."col2_\\u1EAC9" AS "\\xadco_l😢167067",
        postgres_1    |                 tab98272.co🙁l2_10 AS "co l(167068",
        postgres_1    |                 NULL::VOID AS "c*%vol167069"
        postgres_1    |             FROM
        postgres_1    |                 postgres.public.table2 AS tab98272
        postgres_1    |         )
        postgres_1    |             AS "t%va'b98273"
        postgres_1    |     ORDER BY
        postgres_1    |         "co'l167066" DESC, "\\xadco_l😢167067" ASC, "co l(167068" ASC, "c*%vol167069"
        test_1        | --- FAIL: TestCompare (1450.64s)
        test_1        |     compare_test.go:132: Checking connection to: postgres
        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     --- FAIL: TestCompare/postgres (844.87s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:212: SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         '{"I": {"B": "H", "D": {"C": {"B": {"D": [true, "H"], "H": {"A": "D", "G": {"B": {"B": {"A": {"A": [{"D": null, "H": "E"}, {"C": [null, "J"]}], "I": {"I": null}}, "H": true}, "F": null, "J": {"D": {"G": [{"C": "F", "I": "C"}, {"H": {"B": false, "C": {"F": {"F": true, "I": [{"A": {"C": {"E": [{"H": {"A": true, "G": [{"I": {"E": {"I": {"I": {"C": {"C": {"I": [{"C": {"D": null}, "F": false, "I": [6, null, [8, {"A": 4, "G": {"H": {"C": {"C": false, "F": {"C": "D", "E": {"F": {"B": {"B": ["E"], "E": {"H": null}}, "G": true}, "H": [true, false]}}, "J": {"A": {"F": {"D": false, "J": "D"}}, "G": 3}}, "G": {"D": false, "J": {"B": true}}}, "J": [{"B": {"G": "F"}}]}, "H": {"I": [[false, [null]], 3]}}]]}]}, "D": ["G", {"E": "B", "F": false, "J": [[{"B": null, "G": {"A": true, "G": {"I": {"C": 9, "E": [false, "G"], "F": {"A": {"B": [6, false]}, "C": 5, "E": 6}}}, "I": false}}, {"C": {"A": {"C": true, "F": "A"}, "I": "D"}}], true, [true, 4]]}, 5], "H": {"H": null, "I": true, "J": 6}}, "D": false, "G": {"E": null, "I": 7, "J": 5}}}}, "G": "G", "J": {"F": {"J": {"C": "B", "F": {"D": [{"D": [{"C": [[true, {"A": null, "F": {"E": "B", "F": {"A": false, "D": {"E": [{"B": "A"}, [{"B": {"B": {"A": {"A": 2, "G": {"A": {"G": null}, "G": true}, "H": {"D": {"F": false, "G": {"A": null, "H": null, "I": {"D": {"G": {"I": ["E", {"A": 9, "H": {"A": [null, true], "F": null, "H": "G"}}, {"I": {"G": null, "I": false}}], "J": true}}, "G": ["C"], "J": [null]}}}, "H": true, "J": {"I": null, "J": true}}}, "H": {"A": {"B": 3, "G": {"A": {"E": {"I": false}, "F": {"C": {"B": false, "F": true}, "D": {"E": false}, "E": [false, {"A": {"F": null, "I": {"A": {"C": false}, "B": {"D": null, "H": {"B": 6, "D": true}}, "C": {"E": null}}}, "B": {"D": {"I": null, "J": {"H": {"G": {"C": {"F": "F"}}}, "I": {"D": {"F": {"D": null, "E": 1, "G": 7}}}, "J": {"C": false, "F": null, "H": {"E": "A", "F": {"A": 9, "I": {"G": null}}}}}}, "E": {"E": {"J": {"C": {"G": {"B": {"C": {"F": {"C": null, "F": true, "I": {"G": {"B": {"A": true}}}}, "I": {"I": null}, "J": {"A": [null, null], "J": {"D": false}}}, "D": {"D": "B", "E": {"I": [{"A": null, "C": {"J": false}, "E": {"H": {"F": {"E": false, "I": {"B": [null, true, false], "H": {"E": "F"}, "I": 5}}}, "I": {"D": {"B": {"C": {"B": true, "E": {"B": false, "J": 4}}, "D": null, "G": {"C": false, "H": false}}}, "J": {"B": {"H": null, "I": {"F": 7}, "J": [1, {"H": "F", "I": false}, {"C": {"A": false}, "H": true}]}, "H": false}}}}, 5]}}, "H": [false, true, [null, null, true]]}, "J": false}}, "I": false}}}, "H": {"I": {"A": {"C": 8, "E": true}}}}, "C": {"C": false, "E": true}}, true]}}, "C": {"A": 2}, "H": "I"}, "I": {"H": 3}}, "C": {"B": [false, false], "E": false, "H": [{"E": [[false, {"B": {"C": {"F": {"A": {"E": false}, "H": {"C": {"G": {"A": null, "E": {"F": null}, "I": {"A": "C", "C": [{"D": {"F": 4}}, {"H": "A"}, {"C": "I", "F": {"B": 7, "H": 3}, "I": {"A": "C", "H": {"B": true}}}]}}, "H": null, "I": 2}, "H": {"I": [null, {"F": {"H": "I"}, "H": {"F": null, "I": true}, "J": {"J": [false, "G", false]}}]}}, "I": [{"A": 0, "D": 7, "G": [{"C": false, "I": false, "J": {"A": {"A": {"A": "B", "J": [null, {"B": {"E": true, "J": true}, "E": "C", "I": {"A": [[false]]}}]}, "E": false, "F": null}, "C": [{"J": {"G": {"A": false}}}], "E": false}}, null]}, null]}, "I": {"E": {"H": {"B": 9}, "I": true}, "G": "F"}}, "J": true}, "F": true}, {"D": {"E": {"A": {"D": "E", "G": true, "I": {"E": {"I": {"D": {"B": null, "E": null, "I": {"D": false}}, "E": {"G": {"G": {"B": "F", "C": {"J": {"B": {"A": 5, "F": [[{"H": 6, "J": [{"D": false, "E": true, "J": [{"D": true, "F": true, "H": false}]}]}], 2]}, "H": {"G": 3}, "J": true}}, "I": [{"B": {"C": [{"C": {"I": 0}, "E": true}, {"H": true}], "H": {"C": "C", "D": {"E": false}, "G": "F"}, "J": "B"}, "E": {"G": null}}, "D"]}}}, "F": {"G": {"F": {"A": false, "C": [{"E": 1}, true], "I": {"C": "B", "D": false}}, "H": true, "I": {"E": false, "I": {"C": {"B": "C"}, "E": "J", "I": true}}}, "I": false}}}, "F": {"C": "C", "G": {"A": true, "B": false, "G": [[false, {"B": null, "G": {"D": {"F": {"B": "G", "G": true, "H": {"E": false}}, "I": false}, "E": {"A": null, "E": "B", "F": "C"}, "H": {"G": null, "I": {"C": {"C": {"B": {"D": {"A": {"B": null, "I": "G"}, "D": {"F": false, "H": false}, "F": 8}, "F": 2}, "I": "B"}, "D": {"H": "D"}, "F": {"D": {"H": {"B": 7, "H": true}}, "G": null, "I": "F"}}}}}}, {"B": "A", "F": {"D": false}}]]}, "J": {"C": {"I": "D"}, "G": {"C": "I"}}}}}, "E": {"E": {"D": [3]}}, "I": "D"}, "G": {"G": false}, "H": 4}, "F": true, "H": {"C": false, "D": {"D": true, "E": null, "I": null}}}]], "H": [null, {"C": true, "E": 9, "F": {"H": [{"E": {"B": true, "J": {"E": {"D": true, "F": {"F": 0, "G": 1, "H": [{"A": {"C": {"C": {"F": null, "G": false}, "H": 3}}}, {"A": {"A": {"E": 5, "F": false, "I": {"H": true, "I": {"A": 1, "E": {"B": {"A": {"A": {"G": {"G": {"E": "E", "F": {"H": {"B": [true]}}}, "H": false}}, "E": [{"A": null, "I": true}, true], "I": false}, "B": {"F": {"G": "G", "H": {"C": "F", "H": false}, "J": true}, "I": null}}, "C": "E", "E": true}}}}, "B": {"E": [{"B": 5, "C": false, "D": "A"}], "J": 1}, "F": null}, "I": false}, {"I": [{"D": null, "E": {"A": true, "D": {"C": 9, "G": "H"}, "H": {"E": {"B": {"B": {"A": "G", "G": 6, "I": 6}, "H": {"C": true, "J": {"B": {"E": {"I": {"F": {"I": [{"C": ["C", true], "D": {"C": true, "J": false}}, 6, {"B": [{"E": {"E": null, "I": {"C": "A", "I": {"A": {"B": {"F": [3, null]}, "F": false, "I": {"G": false}}, "H": {"E": {"A": false, "B": "B", "E": false}, "H": null}, "J": 7}}}, "G": null, "J": {"F": true, "H": true, "J": {"J": {"D": "G", "F": {"F": [null, {"G": {"B": false, "F": true}, "I": [{"E": 3, "G": "B"}, {"B": [{"B": true, "D": true, "F": "J"}, {"B": false}, {"B": true, "H": "I", "I": {"F": {"B": "A", "D": false, "F": "C"}, "H": {"D": {"A": false, "B": {"A": false, "J": true}}}, "I": {"E": {"A": false, "D": {"J": {"I": 6}}, "H": false}, "I": false}}}], "H": {"B": {"D": {"C": {"F": {"J": [{"D": 7}]}}, "J": {"B": "D"}}, "E": {"H": true}}, "G": {"A": [null, false, 4]}}}, {"B": false, "H": "F"}]}], "J": {"B": [1], "J": true}}}}}}, null], "C": {"C": {"C": {"G": [false, true], "J": null}}, "F": [[false, {"G": null}], true]}, "G": true}]}}}, "H": false}, "E": null}}}}, "F": true}}}]}]}, "G": {"C": [null, {"C": {"A": {"B": {"E": {"H": {"A": {"C": ["A", [{"G": null, "H": {"F": false, "J": {"D": false, "H": 0}}}], false], "I": false}, "I": null}}}, "C": null}, "F": {"A": 4, "E": 8}, "H": null}, "E": {"I": 7}, "F": "D"}, false], "D": 2, "H": "D"}}, "H": {"B": null}}}, "G": null, "J": {"H": false}}, null, true]}}]}]}, "I": {"H": null}}}, "G": true, "J": null}, "H": {"A": null, "B": {"B": [{"C": {"G": {"C": true}}, "J": [{"B": true, "F": {"D": null, "E": {"C": {"B": {"A": true, "B": {"G": [false, [6], {"F": 4, "H": true}], "I": "G"}}, "J": {"H": false}}, "G": {"C": 9, "E": false, "G": {"B": false}}, "J": {"A": "B", "I": "I"}}}, "G": false}, {"I": {"E": false, "J": {"D": {"E": {"C": [{"F": {"B": 3, "E": {"B": false, "F": "H"}, "J": {"J": null}}}], "F": false}}, "G": "A"}}}, [true, {"J": 1}, 8]]}, {"A": false, "D": {"E": 5, "I": {"C": {"F": null, "J": 7}, "D": true, "H": {"J": "D"}}}, "G": 5}, {"B": {"J": true}, "G": ["B", true], "I": 8}]}, "F": true}}, false], null], "G": false}, "E": {"G": [true, {"E": {"C": true, "E": "G"}, "H": [false], "I": 6}, [true]]}}, "I": false}}], 6, {"D": {"G": 8}, "E": {"B": {"D": {"D": {"C": "D"}, "H": {"E": {"D": 5, "J": "A"}, "F": "F"}}, "E": true, "J": [{"F": false}, "E", 6]}, "D": true, "E": "J"}, "I": false}], "F": {"B": [{"F": [{"B": {"A": true}, "D": null}]}, null], "C": {"A": 0, "B": false, "C": {"F": 3}}}, "H": null}, [7, {"B": true}], "F"], "G": true, "I": {"H": null}}, "D"], "F": null}}}, "G": null}}}, {"I": false}, {"B": "J"}], "H": [null, null]}, "J": {"A": [4, {"E": {"A": false, "E": {"A": {"A": "J", "I": null, "J": {"D": {"A": [[true, {"H": {"D": {"A": {"F": [{"G": {"D": {"E": {"A": [false, {"D": 2, "I": {"J": null}, "J": {"A": {"B": 8, "J": null}}}, {"A": true, "I": true}], "H": {"A": {"G": {"D": {"H": {"F": null}, "J": {"G": "H", "H": {"G": true, "H": false, "I": {"B": {"B": [{"D": {"E": {"D": {"B": {"C": "H", "E": false}}, "F": "B", "H": {"D": {"I": {"A": true, "D": {"A": {"C": true, "F": "H", "J": {"F": null}}, "D": {"C": {"A": null, "C": false, "J": {"B": {"F": {"C": {"E": false, "I": false}, "F": 2, "I": [2, null]}}, "I": 1}}, "G": true, "J": null}, "H": {"I": "D"}}}, "J": {"J": {"I": {"C": true}, "J": {"A": "J", "D": false, "E": null}}}}, "E": {"E": {"B": {"A": {"H": {"B": 8, "E": {"D": "J"}, "G": {"C": false, "G": true}}}, "G": {"J": "E"}}, "F": "I", "G": {"C": false}}}, "G": false}}, "F": 5, "H": [false, {"B": {"B": true, "C": 6, "J": null}, "H": 3}, {"B": [false], "E": {"I": {"H": false, "J": {"J": {"D": {"F": {"G": false}}, "I": {"I": null, "J": false}}}}}, "F": false}]}, "E": {"B": false}, "I": {"A": [{"F": {"H": {"F": [null, false], "I": false}, "J": "I"}}, "G", {"J": {"A": [null, null, true], "C": false}}], "D": false, "E": true}}]}, "E": [null, false, false], "H": {"E": true, "F": false, "H": {"B": false, "C": ["J", {"H": {"A": true, "D": 2}}], "H": [false, "J", {"B": {"G": [7, {"A": 2, "G": null}, {"E": {"J": [{"C": null, "F": {"E": {"A": [null, [{"A": false, "E": {"A": "C", "B": null, "C": false}}, {"B": false, "E": [null, "I", false], "G": null}], [5, true]]}, "J": "G"}, "I": false}, "F", [{"B": {"A": null, "G": true}, "J": false}]]}}]}, "C": {"A": {"B": {"H": {"E": {"E": 2, "F": {"B": null, "G": [{"H": false}, null]}}, "F": "D", "I": true}}, "F": {"D": [false, 1, {"A": true, "G": {"E": {"B": null, "I": true}, "F": true}, "H": null}], "F": null}}, "F": null}}]}}}}}}, "E": {"F": {"E": ["D"]}}, "H": 5}}, "F": {"E": false, "G": true}, "H": {"E": null, "J": null}}}}, "G": false}}, false], "H": 4}, "F": [0, "B"]}}}, {"A": false, "J": true}], {"G": [false, true], "J": {"A": {"D": null, "I": [{"F": {"E": null, "J": null}, "I": "G"}, {"F": {"C": {"F": "C", "H": "G"}, "J": true}}, {"F": {"D": {"C": {"F": {"H": 7}}, "D": [2], "J": {"B": 9, "F": "C", "G": [0]}}}, "G": {"E": "A", "J": [1]}}], "J": {"B": true, "J": 6}}, "C": true, "J": "C"}}], "I": "I"}}}, "J": null}}, "F": {"E": {"C": true, "F": {"A": 6, "E": true}}}}], "B": {"D": {"F": null}, "F": {"B": {"E": "C", "H": true, "J": null}}}, "J": {"A": {"F": {"B": true, "F": {"C": {"C": 3, "F": [{"B": 9, "E": {"B": "A"}, "F": true}, {"C": "I"}, [true]]}, "J": 0}}}, "C": false, "D": {"D": null, "H": {"E": 4}}}}}, {"D": false, "E": {"B": null, "D": {"G": 1}, "H": [true, null]}, "G": 9}, {"B": 9, "C": 3}], "G": {"D": null, "E": ["E", {"H": {"D": {"A": {"F": 1}, "E": {"C": 7}}}}]}, "I": {"C": "A", "G": false}}}, "B": "D", "F": {"B": {"E": true, "H": {"J": [null]}, "I": {"F": {"G": {"G": 4}}}}, "D": [9, 4, [false]], "G": {"G": "I"}}}], "J": {"G": true}}}, "J": null}}, {"J": [{"F": {"E": {"H": {"B": {"A": {"E": {"E": [{"B": {"F": 4, "H": {"F": {"B": null, "J": {"A": [true], "E": {"B": {"J": "F"}, "G": true}}}, "H": true, "I": true}}}]}, "I": {"F": {"B": {"A": [null], "G": 3}, "D": {"B": {"B": {"B": {"I": 0, "J": [false, {"J": "G"}, 4]}, "D": null}, "J": {"B": true, "D": "A"}}, "E": null, "F": {"C": {"D": {"G": {"E": "H", "I": ["H", false, null]}, "I": null}}, "D": {"B": [{"G": false, "H": {"B": 5}}, {"E": {"E": null}, "G": true, "H": {"D": {"A": {"F": {"A": {"H": {"C": {"A": {"J": true}}, "D": "J", "F": null}, "I": {"B": {"B": {"I": {"H": "C"}}, "D": true, "H": {"A": 2, "D": "I", "I": {"B": {"C": [{"A": [{"I": null}, {"D": {"D": false}, "G": {"D": [{"B": {"D": {"A": true, "D": {"A": {"B": {"J": null}, "J": null}, "G": [false], "I": null}, "G": false}}}], "H": {"E": "E", "J": {"F": "A", "G": [{"D": "J"}], "J": {"A": null, "E": {"B": [null], "J": false}, "J": false}}}}, "I": {"E": {"J": 0}}}], "G": 8}], "E": {"C": 4, "D": {"H": {"B": false, "D": {"B": {"B": {"H": 4}, "E": {"B": null, "D": null, "F": [{"F": true, "G": "I"}, [[null]]]}}}}}}, "F": {"H": false}}, "F": {"H": true}}}}, "E": {"B": {"D": [3, "F"], "I": {"E": {"D": {"C": {"A": 6, "J": null}}, "G": {"D": "G", "I": {"B": {"G": [true], "H": {"B": {"E": 6}}}, "I": null}, "J": {"E": {"C": {"I": {"D": 9}}, "I": [{"F": {"I": {"A": false, "F": {"J": "B"}, "I": {"E": {"A": [{"D": {"D": true}, "E": true, "F": {"D": {"F": {"C": {"J": 2}, "G": {"B": [{"F": 4, "G": {"A": [false], "E": false, "I": false}, "I": null}, false], "C": "C", "I": "F"}}}, "J": 2}}], "B": ["I"], "E": 5}, "F": false, "G": [[{"C": true, "H": {"F": null, "I": null}, "I": false}], {"B": "C", "H": "J", "I": 0}]}}, "J": false}, "H": true}, "A"]}, "F": true}}}, "G": [{"A": {"E": {"D": null}, "F": "A"}, "F": "G"}, {"A": {"D": ["J", true], "I": {"A": {"I": {"I": {"A": {"B": [true], "C": false, "H": 6}, "B": false, "H": {"F": {"E": 5, "H": 1, "I": true}}}}}}}}, {"C": "C", "D": {"D": [{"C": {"D": {"D": false, "F": "I", "I": {"B": "B", "H": [1]}}, "E": 5}, "H": null}], "F": true, "H": false}, "H": "C"}], "H": {"D": "H"}}}}, "F": "D"}}, "F": "I"}, "J": [{"B": false}]}, "E": true, "I": "F"}}}, true], "G": false}}}, "G": 1}}, "J": {"G": true}}}, "C": {"B": ["C", false], "J": {"E": ["I", {"C": {"C": null, "E": false}, "F": [{"G": {"C": null, "I": {"A": true}}, "H": false}, {"A": {"A": null}, "I": true}, false], "J": false}], "G": {"B": {"D": {"B": "G", "I": true}, "F": [null], "I": {"H": {"E": true}, "I": [[{"G": {"D": "C", "F": null, "I": {"D": 1, "H": 2}}, "H": {"I": {"E": {"H": "D"}, "G": false, "J": [null, 8, {"I": {"B": {"A": 5}, "C": {"F": [{"A": [{"D": null, "F": true, "J": true}, true, {"C": {"C": {"A": {"H": "J"}, "G": false}, "D": null}, "G": null}], "F": null, "H": {"E": {"C": false, "F": "D"}, "G": null, "H": "A"}}, "G", "E"]}}}]}}}, {"C": 7, "D": 4}, {"D": {"C": {"B": true, "G": {"B": 6, "G": true, "J": true}, "H": null}}, "G": 3}]]}}, "C": {"B": 0, "E": false, "F": 6}, "D": null}}}, "F": "F"}}, "H": false, "I": {"B": {"J": {"A": {"B": {"A": {"C": {"J": {"G": {"B": null, "C": {"A": {"C": {"B": {"G": false, "J": 9}, "E": "E"}, "E": true}, "C": "D", "H": false}, "I": {"B": [{"C": false, "J": null}], "J": [{"D": {"I": "J", "J": [1, false]}, "J": "B"}, false, null]}}}}, "E": {"B": true}, "J": false}, "F": {"H": null, "I": "A", "J": 7}}}, "J": [{"C": {"B": {"C": true, "H": false, "J": {"F": false, "H": {"H": {"A": {"G": {"A": [false], "B": false, "G": null}, "I": {"H": [null, [[true], {"D": {"B": {"B": true, "C": {"C": "J", "E": false, "F": [true, false, {"C": {"H": [null, {"A": "D", "C": false, "I": true}]}, "D": "A", "H": {"D": [[true]], "E": false, "I": false}}]}}, "D": "H"}}]]}, "J": {"F": {"H": false}}}, "E": {"B": {"E": null, "G": [false, {"D": true, "E": [true, false, {"H": true, "J": false}], "I": {"B": {"H": false, "I": 7}}}, {"G": true, "H": {"A": {"E": {"E": "C", "H": [true, false, [{"C": {"G": {"D": {"A": false, "H": true}}, "J": {"A": true, "E": null}}, "J": [5, {"E": 5, "J": null}, {"H": {"H": "E"}}]}, {"D": [0, {"A": {"F": false, "G": 5}, "H": {"D": {"H": true}}, "I": {"D": {"E": [false], "I": "J", "J": [{"C": {"C": [{"I": [{"C": false, "H": {"A": {"C": {"B": {"B": null, "H": "D"}, "C": {"B": {"G": true}, "D": {"H": 7}, "I": true}}, "F": true}, "F": {"G": null}, "J": 9}}]}], "J": null}, "D": null, "G": null}, "C"]}, "G": {"B": "G", "H": {"A": true, "I": [7, [true]]}}}}], "I": [false, {"A": [{"F": false}], "C": {"H": {"G": {"A": "A", "C": "J", "H": null}}, "I": [false, {"E": false, "F": true}]}, "D": true}, null]}, [false, [false], {"A": {"D": true, "H": 6, "J": false}}]]]}, "J": false}, "G": [{"F": {"D": {"D": [null, null], "G": "A"}, "G": null, "I": true}}, true]}}]}}, "J": {"E": {"B": null, "F": false, "J": [{"E": null, "I": {"A": {"B": {"B": null, "H": {"A": false}, "I": null}, "G": false, "J": [{"G": "H"}, {"C": {"J": {"B": null, "C": {"C": {"E": {"B": "B"}, "G": false}}}}, "H": {"C": {"A": {"A": {"I": {"E": {"A": false, "F": "F", "I": {"I": [[5], {"G": {"D": {"D": {"A": false}, "J": null}, "F": 0}, "H": "J", "I": false}], "J": {"E": false, "J": true}}}}}, "E": {"D": true, "E": {"J": false}}, "G": 3}, "I": [{"B": 5, "E": {"F": {"D": {"D": null, "G": true, "I": null}, "E": null, "F": null}, "I": {"B": {"I": 3}}}}]}, "F": ["J", {"A": {"C": true, "H": {"H": {"A": {"I": {"F": false, "I": {"D": null, "G": false}, "J": [false]}}}}, "J": null}, "E": {"I": {"C": "H", "H": {"D": 9}}}}]}}]}, "G": {"H": false}, "I": "I"}}, "A", {"A": {"B": null, "D": false, "G": [4, true]}, "I": [[7, {"G": true}], {"B": 7}]}]}, "F": {"B": null, "D": true, "F": [{"A": false, "I": [9], "J": "H"}]}, "J": 7}}}}}, "H": true}}, {"H": {"C": {"A": true, "I": {"D": {"H": true, "I": {"C": null, "D": {"H": false}}}, "F": {"H": "F"}, "I": {"E": [6, {"A": false, "F": {"A": true, "I": {"F": {"A": true, "F": null, "G": "G"}, "G": {"A": {"A": {"B": [{"A": 4}, "E"], "J": [{"B": {"F": null, "J": {"A": "B"}}, "F": {"G": {"B": {"A": 2, "C": {"I": "C"}, "G": {"D": true, "E": "F", "J": "I"}}, "D": {"B": [{"H": "B"}, {"A": {"G": {"B": false, "C": [[null, {"A": true, "F": {"F": {"G": {"B": {"B": true, "J": false}, "C": {"I": {"G": 2, "I": [{"B": {"B": {"B": 6, "H": null, "J": true}, "E": "E", "H": {"D": true, "I": {"F": {"H": {"E": false}}, "J": "J"}}}, "G": {"D": [{"B": 9}, "I", false]}}, false, {"G": 9, "I": 9}], "J": [{"C": {"I": "A"}, "G": {"C": {"I": false}}}, 5, {"A": "J"}]}}, "D": ["G"]}}, "I": {"C": true}}, "I": null}], {"D": 2}], "E": {"A": {"H": null, "I": {"B": {"C": 5, "E": null, "G": false}}}, "J": {"H": "H"}}}}, "B": {"B": null, "I": true, "J": null}}], "G": {"C": "D", "D": {"A": {"J": 0}}, "E": {"E": {"A": {"E": {"F": {"H": {"C": true}}, "J": {"C": [[{"B": 0}]]}}, "I": null}, "D": 8, "J": {"E": {"F": false}, "J": {"C": null}}}}}}, "G": {"E": false}}, "H": {"D": "H", "J": {"A": [{"A": null, "J": false}, {"F": {"J": ["D", {"C": {"A": "I", "J": null}}, {"C": true, "E": {"H": "D", "I": true}}]}, "J": {"C": {"A": "D", "D": {"E": null, "F": "F", "H": {"F": "I", "H": true}}}, "G": {"A": true, "F": null, "H": {"F": true, "H": [{"F": true, "H": {"C": null, "D": ["G"], "E": {"B": {"D": [null], "E": "E"}, "D": {"H": true}, "H": false}}}, {"G": "H", "H": {"I": {"F": {"B": {"F": false}}}}, "J": [false, {"F": {"H": {"C": 7, "G": [{"B": {"A": {"B": null, "F": false, "H": null}, "C": null, "G": false}, "C": false, "J": true}, "A"]}}, "H": "J"}]}], "I": {"A": {"B": {"H": [true, null, {"C": {"C": false, "H": [{"C": "A"}, [true, {"H": {"B": true, "C": "G"}, "I": "F"}, "I"]]}, "D": null, "E": {"A": false, "D": 9}}], "I": null}}, "D": {"I": null}}}}}}, [null, {"C": {"C": [3, {"F": null, "G": false}], "I": {"A": 2, "C": 2}}, "G": {"A": 9, "D": false}}]], "D": {"B": false, "G": ["I", {"B": {"C": {"E": false, "F": 1, "J": {"C": [[{"A": ["I"], "I": true, "J": {"G": false}}], {"E": {"B": {"E": {"H": {"A": 9, "B": {"G": false, "H": {"A": true, "E": "C", "F": {"B": {"G": true, "J": {"D": true, "G": ["G", {"A": {"C": false, "G": 0}}]}}, "D": [{"A": {"B": {"C": {"F": {"D": {"D": [true, {"I": {"A": "E", "E": {"D": {"D": "A", "F": {"A": [{"B": "C", "E": [["I", null], 4], "J": {"J": {"C": {"C": {"J": "E"}, "D": [null, null, [null]], "F": false}, "I": {"E": "F", "I": {"E": {"B": 8, "E": {"F": {"G": {"D": {"C": null, "G": {"A": {"A": {"C": "H"}}, "D": 3, "G": {"I": false}}, "H": "J"}, "G": true, "H": {"D": {"B": {"C": false}, "F": "C", "G": false}}}, "J": null}}, "J": {"J": {"E": {"G": 3}, "J": false}}}, "G": {"A": 6, "C": "E"}}}}}}, true, 8]}, "I": 0}, "H": {"B": [{"A": "G", "D": null, "F": {"F": [[false]], "H": {"H": false, "I": false}}}], "F": false}, "J": 2}}}, {"E": true, "H": false}], "G": {"G": {"G": {"E": false, "I": {"E": null, "H": null, "I": true}}, "I": 2, "J": 6}, "H": {"A": "B"}}}, "F": {"F": true}, "I": {"H": {"D": true, "I": false}}}, "G": {"B": "J", "F": {"B": 0, "E": {"D": [{"C": {"C": true}, "I": {"B": {"A": {"G": 9}, "B": false}}, "J": {"C": {"A": {"E": {"F": false}, "G": {"A": "J", "F": false, "H": {"B": 7}}}, "C": null, "H": {"A": {"E": {"F": null, "I": {"H": 6}}}, "H": null}}, "F": null, "J": true}}, {"D": {"D": true, "F": {"H": {"I": "F"}, "I": {"A": 6, "I": "E"}}}}], "E": [null]}}}}}}, "D": {"D": {"D": false, "G": {"H": {"I": 9}}, "J": {"F": {"I": [{"A": [{"D": {"C": [{"E": false, "G": false, "H": {"H": null}}], "G": "G"}, "G": {"B": "J", "F": {"H": 6, "I": [[false, {"C": false}]]}, "I": 5}}], "D": {"C": true, "F": false}, "E": {"A": {"G": "D"}}}, false]}, "G": {"F": "E", "H": {"B": {"G": {"E": false}, "J": {"D": true}}, "J": {"G": {"B": null, "F": [7, [true, {"A": {"D": false, "G": false, "I": "E"}, "B": {"H": false}, "F": {"F": true}}], {"D": {"I": {"B": "J", "E": null, "J": 0}}, "G": false}]}}}}}}}, "G": {"B": {"A": {"D": 3, "F": false}, "E": [[null, [{"E": {"F": {"D": true, "F": null, "J": {"C": true, "F": {"C": 3}, "J": {"I": true}}}, "G": null, "J": null}, "H": "I", "I": null}, {"C": {"D": {"H": {"A": [6, {"F": {"F": [{"A": null, "E": "F", "I": {"B": null, "C": 0, "J": {"H": [{"C": {"A": "H", "B": false, "E": 0}}], "J": true}}}], "G": {"H": {"C": 9, "G": false}}, "I": 1}, "H": {"D": null}}, {"A": {"A": {"G": 9}}, "F": {"E": [null]}}]}}, "G": {"B": {"C": {"D": {"I": 5}}}, "C": {"G": null, "H": false}, "H": {"D": false}}, "I": false}, "I": {"C": {"C": null}, "E": ["E"]}}], false]]}, "G": {"B": false, "C": {"I": [{"F": [{"C": {"B": {"A": {"B": 0, "C": 4, "J": {"F": 6, "G": null}}, "G": 1, "H": {"B": 4, "H": null}}, "F": "D", "I": {"C": {"B": {"A": {"I": {"A": false, "D": "C"}}, "I": true}, "H": false, "I": 7}, "E": [null, {"A": null}], "J": ["C", false, null]}}, "H": null}], "H": [true, true, [false, {"B": {"C": false, "D": null}, "E": false, "I": {"F": [false, {"E": 4, "I": {"C": {"A": null}, "D": "A", "I": false}}], "H": {"D": null, "F": 1, "G": {"B": true, "D": 2}}}}, null]]}]}}}}, false, false], "E": 6}}, "J": null}}}}, "E": 5}, "F": {"J": false}}], "F": null, "H": 1}}}, "J": {"E": ["G", 8]}}], "I": {"D": false, "H": {"C": {"E": 7}, "F": 6, "J": {"D": null, "E": null}}}}}}}, "H": true}]}, "I": null}, "G": {"F": 7}, "I": {"D": null, "E": "C", "F": {"A": {"G": {"C": true, "H": true}, "H": {"B": {"A": {"I": false, "J": false}, "E": {"A": null, "C": {"H": {"B": {"A": true, "E": true}, "E": "H", "G": true}, "J": {"E": true, "F": null}}}}, "C": {"A": [0, [{"E": {"A": "H", "C": false}, "H": {"A": false, "C": "C", "I": null}}, "A", "G"], {"A": false, "H": {"B": {"A": {"G": [{"I": [[{"D": {"A": true}, "F": [true, false, [{"F": 7, "I": null}]], "G": {"B": {"B": null}, "C": "G", "F": false}}, "J", [{"B": {"J": {"A": [{"F": false}, {"H": null}], "G": {"E": true, "G": {"B": {"E": [{"B": null, "J": "I"}, {"A": false, "B": true}], "J": {"C": 3}}, "C": {"E": ["G", {"B": null}, {"A": false, "G": true, "H": {"A": [{"B": 3, "C": null}, null, "I"], "G": null, "I": {"D": [{"D": false}, "J"], "J": {"D": null, "E": true}}}}]}}}}}, "C": [{"G": "D"}], "J": null}, 0]]]}, null, {"D": [{"F": {"B": [{"C": [{"J": {"J": false}}], "D": {"E": true, "J": {"H": true, "J": [{"G": {"C": [{"B": true, "G": {"H": "B"}}], "G": {"G": {"A": null, "D": {"I": null}, "J": {"A": "G", "H": false}}, "I": false}}}, {"I": null}]}}, "G": {"C": [3], "H": {"B": true, "C": {"J": {"A": "C", "H": {"A": false, "F": "J"}}}}}}, false, {"I": 3}], "I": [{"A": null}, {"C": {"B": 5, "E": {"A": null, "B": true}}}, {"A": false, "B": [true, 6], "F": true}]}}, "B", {"A": false, "C": true, "D": {"D": false, "G": false}}], "G": {"B": [true, "B", false], "H": {"B": "F", "C": true}}}]}, "G": {"D": false, "G": 7}}, "F": true, "H": [true, {"E": {"A": {"D": {"B": null, "D": null, "G": "E"}, "H": {"C": 2, "F": {"A": 8}}}, "G": {"A": false, "E": false, "H": {"A": [{"E": true, "I": true}, 5]}}}, "H": [null]}, 4]}}]}, "I": {"E": false}}}, "G": true}}}, "I": null}}, "G": {"F": 4, "J": {"E": false}}}], "J": false}}, "J": {"B": 9, "H": {"H": 8, "J": null}, "J": [true]}}}, "I": {"I": true}}, {"D": [false]}]}}, "I": {"H": "J"}, "J": [7, false]}}}, null]}]}}}, "D": {"E": {"F": null, "J": {"D": {"B": false, "E": "A"}}}}, "F": true}, "J": {"I": {"H": {"C": 3}, "I": "D"}}}}, "D": false, "H": [8, 5]}}}, "J": false}':::JSONB::JSONB
        test_1        |                             AS col167070
        test_1        |                 )
        test_1        |                     AS ta😏b98274
        test_1        |             ORDER BY
        test_1        |                 col167070 ASC NULLS LAST
        test_1        |         compare_test.go:213: cockroach1: ping: conn closed
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (1476.30s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #121592 compose: TestComposeCompare failed [no locations to init GEOS] [C-test-failure O-robot T-dev-inf branch-release-23.2] - #121531 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-24.1] - #121530 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-master] - #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ c9fff6f54f3094ca4eacd788dabb6f8ab61cacfc:

        postgres_1    |             FROM
        postgres_1    |                 postgres.public."ta%pble1" AS "Tab66142"
        postgres_1    |             ORDER BY
        postgres_1    |                 "Tab66142".col1_0 ASC NULLS FIRST
        postgres_1    |         )
        postgres_1    |             AS tab66143
        postgres_1    |     ORDER BY
        postgres_1    |         "col%q130576" ASC NULLS LAST, "c""ol130577" NULLS LAST, "co""l130578" DESC, "%qcol130579" ASC
        test_1        | --- FAIL: TestCompare (677.26s)
        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     compare_test.go:132: Checking connection to: postgres
        test_1        |     --- FAIL: TestCompare/postgres (72.54s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:201: open /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/_tmp/018812dfcf0ce383ac3e799915a61396/TestComposeCompare3365578124/002/postgres.log: no such file or directory
        test_1        |         compare_test.go:203: compare cockroach1 to postgres: unexpected diff:
        test_1        |               []any(Inverse(func1, []any{
        test_1        |             -   nil,
        test_1        |             +   string("01010000000000000000205CC00000000000E05040"),
        test_1        |               }))
        test_1        |             cockroach1:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (SELECT st_pointfromgeohash('C':::STRING::STRING::STRING, NULL::INT4::INT4)::GEOMETRY AS "'col130642") AS tab66188
        test_1        |             ORDER BY
        test_1        |                 "'col130642" DESC;
        test_1        |             
        test_1        |             postgres:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (SELECT st_pointfromgeohash('C'::TEXT::TEXT::TEXT, NULL::INT4::INT4)::GEOMETRY AS "'col130642") AS tab66188
        test_1        |             ORDER BY
        test_1        |                 "'col130642" DESC;
        test_1        |             
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_cockroach1_1 ... 
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (702.10s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #121592 compose: TestComposeCompare failed [no locations to init GEOS] [C-test-failure O-robot T-dev-inf branch-release-23.2] - #121531 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-24.1] - #121530 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-master] - #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ e3276dc8c5e041aabc787bba43193e638103f31f:

        test_1        |     compare_test.go:132: Checking connection to: postgres
        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     --- FAIL: TestCompare/postgres (153.84s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:201: open /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/_tmp/018812dfcf0ce383ac3e799915a61396/TestComposeCompare2859608782/002/postgres.log: no such file or directory
        test_1        |         compare_test.go:203: compare cockroach1 to postgres: unexpected diff:
        test_1        |               []any(Inverse(func1, []any{
        test_1        |             -   s"-0290-02-24 03:13:43.145224 +0000 UTC",
        test_1        |             +   s"0002-06-05 03:01:00.998782 +0000 UTC",
        test_1        |               }))
        test_1        |             cockroach1:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         make_timestamp(2:::INT8::INT4::INT4, 6:::INT8::INT4::INT4, 5:::INT8::INT4::INT4, 3:::INT8::INT4::INT4, 1:::INT8::INT4::INT4, cosd((2.0:::FLOAT8::FLOAT8::FLOAT8 ^ (9.0:::FLOAT8::FLOAT8::FLOAT8 / 6.0:::FLOAT8::FLOAT8::FLOAT8)::FLOAT8::FLOAT8)::FLOAT8::FLOAT8)::FLOAT8::FLOAT8)::TIMESTAMP
        test_1        |                             AS col127047
        test_1        |                 )
        test_1        |                     AS t😰ab81165
        test_1        |             ORDER BY
        test_1        |                 col127047;
        test_1        |             
        test_1        |             postgres:
        test_1        |             SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         make_timestamp(2::INT8::INT4::INT4, 6::INT8::INT4::INT4, 5::INT8::INT4::INT4, 3::INT8::INT4::INT4, 1::INT8::INT4::INT4, cosd((2.0::FLOAT8::FLOAT8::FLOAT8 ^ (9.0::FLOAT8::FLOAT8::FLOAT8 / 6.0::FLOAT8::FLOAT8::FLOAT8)::FLOAT8::FLOAT8)::FLOAT8::FLOAT8)::FLOAT8::FLOAT8)::TIMESTAMP
        test_1        |                             AS col127047
        test_1        |                 )
        test_1        |                     AS t😰ab81165
        test_1        |             ORDER BY
        test_1        |                 col127047;
        test_1        |             
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_cockroach1_1 ... 
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (783.84s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #122079 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-master] - #121592 compose: TestComposeCompare failed [no locations to init GEOS] [C-test-failure O-robot T-dev-inf branch-release-23.2] - #121531 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-24.1] - #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ e3276dc8c5e041aabc787bba43193e638103f31f:

        postgres_1    | waiting for server to shut down...2024-04-15 05:20:50.261 UTC [49] LOG:  received fast shutdown request
        postgres_1    | .2024-04-15 05:20:50.262 UTC [49] LOG:  aborting any active transactions
        postgres_1    | 2024-04-15 05:20:50.264 UTC [49] LOG:  background worker "logical replication launcher" (PID 56) exited with exit code 1
        postgres_1    | 2024-04-15 05:20:50.266 UTC [51] LOG:  shutting down
        postgres_1    | 2024-04-15 05:20:50.354 UTC [49] LOG:  database system is shut down
        postgres_1    |  done
        postgres_1    | server stopped
        postgres_1    | 
        postgres_1    | PostgreSQL init process complete; ready for start up.
        postgres_1    | 
        postgres_1    | 2024-04-15 05:20:50.383 UTC [1] LOG:  starting PostgreSQL 13.5 (Debian 13.5-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
        postgres_1    | 2024-04-15 05:20:50.383 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
        postgres_1    | 2024-04-15 05:20:50.383 UTC [1] LOG:  listening on IPv6 address "::", port 5432
        postgres_1    | 2024-04-15 05:20:50.386 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
        postgres_1    | 2024-04-15 05:20:50.391 UTC [68] LOG:  database system was shut down at 2024-04-15 05:20:50 UTC
        postgres_1    | 2024-04-15 05:20:50.396 UTC [1] LOG:  database system is ready to accept connections
        postgres_1    | 2024-04-15 05:30:53.649 UTC [98] ERROR:  invalid byte sequence for encoding "UTF8": 0x00
        postgres_1    | 2024-04-15 05:30:53.649 UTC [98] STATEMENT:  CREATE TABLE "t""a ble2" (col2_0 VARCHAR, " col2_%v1" INET NOT NULL, col2_2 OID NULL, col2_3 REGROLE, col😇2_4 TSVECTOR NULL, "co{l2_5" FLOAT4 NOT NULL, "c""o%vl2%v_6" TEXT NULL GENERATED ALWAYS AS (lower(CAST(col😇2_4 AS TEXT))) STORED, "col2_
7" TEXT GENERATED ALWAYS AS (lower(col2_0)) STORED, "co,l2_8" TEXT GENERATED ALWAYS AS (lower(col2_0)) STORED, col2_9 TEXT NULL GENERATED ALWAYS AS (lower(CAST(col😇2_4 AS TEXT))) STORED, "col 2_10" TEXT NULL GENERATED ALWAYS AS (lower(CAST(col😇2_4 AS TEXT))) STORED);
        postgres_1    |     CREATE INDEX ON "t""a   ble2" (" col2_%v1" ASC, "co{l2_5" ASC, (CASE WHEN col2_3 IS NULL THEN e'\x00'::TEXT ELSE e':Y%me\x7f4]'::TEXT END), "col2_
7", "col 2_10", col2_2 DESC, "co,l2_8", "c""o%vl2%v_6", col2_9 ASC);
        postgres_1    |     CREATE INDEX ON "t""a   ble2" (col2_0 ASC);
        postgres_1    |     CREATE INDEX ON "t""a   ble2" (" col2_%v1" ASC, "co,l2_8" ASC, col2_0 ASC, "co{l2_5" DESC);
        postgres_1    |     CREATE INDEX ON "t""a   ble2" (col2_2 ASC, col2_9 DESC, " col2_%v1") INCLUDE (col2_0, col2_3, "co{l2_5");
        postgres_1    |     
        test_1        | --- FAIL: TestCompare (606.44s)
        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     compare_test.go:132: Checking connection to: postgres
        test_1        |     --- FAIL: TestCompare/postgres (1.74s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:172: CREATE TABLE "t""a ble2" (col2_0 VARCHAR, " col2_%v1" INET NOT NULL, col2_2 OID NULL, col2_3 REGROLE, col😇2_4 TSVECTOR NULL, "co{l2_5" FLOAT4 NOT NULL, "c""o%vl2%v_6" TEXT NULL GENERATED ALWAYS AS (lower(CAST(col😇2_4 AS TEXT))) STORED, "col2_
7" TEXT GENERATED ALWAYS AS (lower(col2_0)) STORED, "co,l2_8" TEXT GENERATED ALWAYS AS (lower(col2_0)) STORED, col2_9 TEXT NULL GENERATED ALWAYS AS (lower(CAST(col😇2_4 AS TEXT))) STORED, "col 2_10" TEXT NULL GENERATED ALWAYS AS (lower(CAST(col😇2_4 AS TEXT))) STORED);
        test_1        |             CREATE INDEX ON "t""a   ble2" (" col2_%v1" ASC, "co{l2_5" ASC, (CASE WHEN col2_3 IS NULL THEN e'\x00'::TEXT ELSE e':Y%me\x7f4]'::TEXT END), "col2_
7", "col 2_10", col2_2 DESC, "co,l2_8", "c""o%vl2%v_6", col2_9 ASC);
        test_1        |             CREATE INDEX ON "t""a   ble2" (col2_0 ASC);
        test_1        |             CREATE INDEX ON "t""a   ble2" (" col2_%v1" ASC, "co,l2_8" ASC, col2_0 ASC, "co{l2_5" DESC);
        test_1        |             CREATE INDEX ON "t""a   ble2" (col2_2 ASC, col2_9 DESC, " col2_%v1") INCLUDE (col2_0, col2_3, "co{l2_5");
        test_1        |             
        test_1        |         compare_test.go:173: postgres: exec: ERROR: invalid byte sequence for encoding "UTF8": 0x00 (SQLSTATE 22021)
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (631.52s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #122339 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1] - #122079 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-master] - #121592 compose: TestComposeCompare failed [no locations to init GEOS] [C-test-failure O-robot T-dev-inf branch-release-23.2] - #121531 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-24.1] - #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

compose.TestComposeCompare failed with artifacts on release-23.2.4-rc @ e3276dc8c5e041aabc787bba43193e638103f31f:

        postgres_1    |             FROM
        postgres_1    |                 postgres.public."ta/ble😀4" AS "ta%vb 118614"
        postgres_1    |             WHERE
        postgres_1    |                 NULL::BOOL
        postgres_1    |             ORDER BY
        postgres_1    |                 "ta%vb 118614"."col%c84_0"
        postgres_1    |         )
        postgres_1    |             AS tab118615
        postgres_1    |     ORDER BY
        postgres_1    |         col186050 DESC, col186051 ASC, col😪186052 DESC, col186053 ASC NULLS FIRST, col186054 ASC NULLS LAST
        postgres_1    | 2024-04-17 05:24:59.993 UTC [78] ERROR:  could not identify an ordering operator for type refcursor at character 178
        postgres_1    | 2024-04-17 05:24:59.993 UTC [78] HINT:  Use an explicit ordering operator or modify the query.
        postgres_1    | 2024-04-17 05:24:59.993 UTC [78] STATEMENT:  SELECT
        postgres_1    |         *
        postgres_1    |     FROM
        postgres_1    |         (SELECT NULL::BIT(22) AS "co\\x1fl186068", NULL::REFCURSOR AS col186069, B'0010'::BIT(34) AS col186070) AS tab118630
        postgres_1    |     ORDER BY
        postgres_1    |         "co\\x1fl186068" DESC NULLS LAST, col186069 DESC NULLS LAST, col186070 ASC
        test_1        | --- FAIL: TestCompare (1224.50s)
        test_1        |     compare_test.go:132: Checking connection to: cockroach2
        test_1        |     compare_test.go:132: Checking connection to: postgres
        test_1        |     compare_test.go:132: Checking connection to: cockroach1
        test_1        |     --- FAIL: TestCompare/postgres (619.22s)
        test_1        |         compare_test.go:141: starting test: postgres
        test_1        |         compare_test.go:150: initializing connection: cockroach1
        test_1        |         compare_test.go:150: initializing connection: postgres
        test_1        |         compare_test.go:212: SELECT
        test_1        |                 *
        test_1        |             FROM
        test_1        |                 (
        test_1        |                     SELECT
        test_1        |                         '{"D": {"B": false, "F": {"A": [[{"J": null}, null, {"A": {"H": {"B": true, "E": {"B": false, "D": [{"E": {"B": {"C": ["E"], "E": {"B": {"A": 3}}, "G": false}, "C": {"A": {"A": {"D": 5, "E": null}, "H": 5}}, "I": true}}, {"G": {"B": 7, "G": {"B": {"F": {"C": true}}, "H": {"A": null}}}}], "I": {"A": true, "E": false, "H": {"D": "G", "E": {"J": "H"}}}}}}, "C": {"A": {"A": true, "C": false, "J": null}, "J": [false, [{"A": null, "E": [{"A": {"C": {"H": {"I": "E"}, "J": [null, {"F": [true, {"D": {"F": false, "G": {"C": [9], "E": 5}, "I": {"F": false}}, "J": {"A": [false, ["F", {"H": 0}]], "E": [{"I": false}, true]}}], "G": {"G": false}, "I": {"C": false, "G": false}}, false]}, "D": "D", "F": {"B": true, "G": {"A": true, "H": {"B": [4], "D": true}}, "I": [9, null]}}, "C": null}, {"A": 6, "E": {"D": {"B": {"G": "G", "H": 7}, "D": false}, "E": {"B": {"B": [4, {"F": {"A": null, "H": false, "I": true}}]}, "E": {"B": {"A": {"J": [[{"B": null, "E": {"A": {"B": {"A": {"D": false, "G": {"D": {"C": null, "G": 4}, "E": {"H": "J"}}}, "I": ["J", {"E": true, "F": {"C": "J", "J": 0}, "H": {"E": true, "H": 1, "J": {"A": {"C": true, "E": {"A": {"H": {"H": false}}, "H": false, "J": "J"}, "J": null}, "C": [false]}}}]}, "C": 9, "G": [{"D": "G", "H": false, "I": [["J", 9, {"B": false}]]}, "I"]}, "G": "F", "H": {"G": {"B": {"F": false}, "C": [true, {"E": [null, 3, {"H": "J"}], "I": true, "J": null}], "J": "B"}}}}, [[{"D": 4, "F": {"D": 6, "J": "C"}}, {"I": null, "J": {"F": {"A": 4, "I": {"D": {"G": {"A": {"C": {"D": [[{"F": {"C": "A", "J": {"F": [true, [{"E": 2, "H": true}, "A"], true]}}, "J": {"D": null}}], {"A": 1, "C": true}, false], "F": 6, "I": 7}, "H": false}, "G": {"F": null, "G": {"B": true, "F": [{"F": false}]}, "J": true}, "J": {"H": {"F": 4}}}, "I": {"H": 8}}, "H": false, "I": false}}, "H": {"I": {"C": [{"I": {"F": {"A": null, "D": {"F": {"C": {"B": 9, "C": [{"B": [{"A": {"D": 3, "F": 8, "I": [{"B": {"F": 6, "H": {"B": true, "H": 0}, "J": true}, "D": 9, "F": {"J": {"C": {"E": {"A": "C", "B": true, "J": {"C": {"A": 1, "F": {"H": 6}}}}}, "E": true}}}]}, "I": {"B": {"C": [{"B": {"H": false, "I": {"F": {"D": false}, "H": {"F": 4, "J": true}, "I": {"D": 4, "E": false}}, "J": [{"C": "G"}, false, {"B": {"I": false}}]}, "E": "G"}, true], "G": {"D": 0}, "I": [[{"F": {"D": [{"I": [{"J": ["E"]}]}, true], "G": false}, "I": "E"}, {"J": null}]]}, "F": true}}, "B"], "C": {"A": [{"E": {"A": ["G", true], "C": 2, "G": [1, "C", {"B": 6, "C": null}]}}]}}], "E": [[false, true, [null, {"B": 8, "C": true, "J": {"C": false, "G": "C"}}]], 2]}, "E": {"H": {"E": [{"I": true, "J": null}], "F": "A"}}}}, "H": [{"A": 6, "D": {"H": {"B": [{"F": {"I": [6, null]}}, {"C": {"A": 8, "H": {"G": [{"C": "I"}, null, {"A": {"B": true, "C": {"B": {"A": {"B": {"A": {"F": null, "I": "F"}, "C": {"B": {"C": {"I": "F"}}, "H": {"H": {"D": {"A": [null], "G": "I"}, "F": {"D": {"A": false, "G": null}, "H": true}}}}, "J": [{"A": [{"D": {"B": true, "J": {"F": null, "I": {"C": [false, false], "F": true, "G": "I"}}}, "J": "I"}], "I": {"A": true}}, [{"C": {"A": {"B": {"G": {"A": {"B": [{"C": null, "H": 2}, {"A": null, "C": {"F": {"C": {"E": {"G": [[null, [{"C": {"D": {"C": {"B": 8, "F": true}, "D": "G"}, "H": null}, "F": {"C": {"D": true, "G": {"B": 4, "D": true, "G": "I"}}, "G": 7}, "I": {"B": true, "C": 2}}, {"B": true, "F": "C"}], "H"], true, 8]}}, "F": true, "I": {"C": "H", "D": false, "E": {"B": true, "J": {"B": 4}}}}}}, {"G": {"G": false}}], "G": "J", "I": [false, {"B": {"F": true, "J": true}, "H": {"A": null, "G": {"E": null}}}, [{"I": [{"A": true, "D": false, "G": "A"}]}]]}, "C": {"C": "A"}}}, "G": {"F": null}, "J": false}, "I": ["H", true, true]}, "F": {"A": null}, "G": false}, {"B": 5, "H": true, "I": {"A": {"B": {"H": [false], "I": "I"}, "G": {"J": true}}, "G": {"B": {"G": "B"}, "E": {"B": {"C": {"D": {"I": false}, "I": {"J": [[{"J": true}, {"E": true, "G": true}, {"E": true, "J": true}]]}}, "F": true}, "C": {"B": {"C": true}, "G": false, "I": "F"}, "H": false}, "I": 8}}}], false]}, "E": {"E": {"F": true}, "G": {"H": "A"}}, "G": true}, "E": {"C": 6, "E": null, "H": [{"B": {"C": null, "H": [{"C": null, "J": 7}, {"C": false, "E": false, "H": true}]}, "F": "A"}, true]}}, "F": null, "J": {"I": false, "J": true}}, "F": "F"}, "H": "B"}]}}}, 1]}}}, {"A": false, "B": {"C": {"A": {"B": "A"}, "E": {"B": 9, "C": {"F": {"E": true, "G": 5, "H": false}}, "G": {"A": [{"C": "H", "D": "E"}, [{"E": null, "F": null, "I": {"F": {"A": {"A": true}, "C": [{"A": null, "D": {"B": "B", "E": false, "J": false}}], "G": "G"}, "I": [[null], {"F": 4, "I": "J", "J": null}]}}]]}}, "G": [{"G": {"E": {"A": "J"}, "I": {"A": true}}, "H": true, "J": {"C": "F"}}, 0]}, "F": {"A": false}, "H": "G"}, "G": false}, true]}}}, {"A": {"F": {"C": [{"A": {"F": {"C": {"C": {"H": {"E": false}}, "H": "G", "I": 9}, "D": {"A": true, "H": 9, "J": 4}}, "G": {"D": true, "G": {"D": {"H": {"D": {"A": "I", "B": {"J": {"C": false, "E": true}}}}, "J": null}, "H": [false, {"A": {"G": true, "H": {"A": 4, "C": "C", "G": {"J": 8}}}, "I": {"C": false}}, false]}}}, "C": true}], "I": {"B": {"G": {"F": null}}, "D": false}, "J": false}, "J": true}, "E": [{"G": {"G": {"A": {"B": false, "F": [{"H": "C"}, false], "H": [[{"A": [{"D": {"I": [{"C": null}, "C", 6], "J": {"H": "B"}}, "F": 2, "G": null}], "D": {"C": {"A": null, "B": [["E"], "A", true]}}, "J": false}], null, 1]}, "B": ["B", {"C": false, "I": 1}, [{"B": 9, "D": {"C": true, "E": 9}}, true]]}, "H": {"J": {"A": {"A": [6], "B": {"C": {"G": {"C": true}, "J": null}, "E": true}}, "B": {"C": {"E": null, "I": null, "J": [true, 3]}}, "G": {"E": {"B": true, "E": null}, "F": [[{"C": {"B": {"A": false, "D": {"E": {"A": 5}}, "E": "D"}, "H": {"A": [true, {"H": {"A": 7, "D": true}}, 1]}}, "J": {"C": {"E": [true], "F": true, "H": null}}}, {"B": null, "H": "G"}], false]}}}, "I": true}}, {"C": {"J": true}}], "H": true}, {"F": false}], "F": null, "J": true}}, "I": {"A": {"D": true, "F": {"F": 5}, "J": "C"}, "D": {"B": {"E": null, "I": ["I", [{"J": {"A": {"A": true, "G": null}, "F": {"J": {"A": {"J": 1}, "C": true, "D": {"F": true}}}}}, {"H": 4}], false]}, "F": {"G": 5, "I": false}, "I": "H"}, "I": false}}}, ["A", null]], {"C": 9, "D": {"A": {"A": false, "F": [{"A": 4}, null, true], "J": {"I": null}}}, "J": false}], {"F": "G", "G": {"D": {"C": {"E": null}, "E": {"F": "C", "J": "H"}, "I": {"F": {"A": [true], "E": "F", "I": {"B": {"C": ["H", {"B": {"E": {"H": {"A": null, "G": {"E": {"B": {"A": "J", "F": "E"}, "C": {"D": {"G": [9, {"J": 3}, {"D": {"D": {"A": {"A": {"F": {"A": true, "F": {"I": {"E": {"B": false}, "G": true, "I": {"F": {"H": {"A": true}, "J": "D"}}}}}}}}}}]}, "F": null}}, "F": false}}, "J": {"F": 9, "G": {"E": [{"C": false}, null, {"B": ["B", "E", {"B": false}], "I": [null, true], "J": {"H": {"F": {"C": 3, "D": "I"}}}}], "F": {"B": {"F": ["D", false], "H": {"A": {"H": 0, "J": [true, false]}}, "J": null}}, "I": {"A": true}}, "J": {"H": false, "J": 6}}}, "I": {"J": "I"}}, "E": {"C": true, "I": 5}}]}, "G": false, "I": 4}}, "G": {"G": false}}}, "F": 8}}]]}, "G": null, "H": {"B": 0, "D": 1, "H": false}}, "F": null, "H": {"G": {"E": [true]}, "J": null}}}}, "J": {"G": false, "J": {"G": [{"E": 4, "G": {"C": null, "E": null}, "H": true}, {"E": {"C": [{"B": 2}, {"B": false, "C": {"C": {"E": {"A": {"F": [{"A": {"G": {"G": 6, "H": {"C": [{"J": {"B": [{"I": 2, "J": false}, {"A": {"G": 1}, "E": false}], "G": {"G": 3, "H": "I"}, "H": {"H": true}}}], "D": null, "E": [{"G": false}, true, [{"B": {"A": true, "E": {"G": 7, "H": ["E"]}, "G": {"E": true, "F": false}}, "G": null}]]}, "I": {"A": {"C": {"J": "B"}}, "H": {"F": {"B": "A", "F": 4}}}}, "H": {"F": null, "I": {"D": "J"}}, "I": {"B": true, "H": {"D": true, "E": "J"}, "I": 5}}, "B": null, "E": {"C": null}}], "G": {"H": false}, "H": [[[{"B": null, "C": [[4], {"H": {"B": true, "E": {"F": "C", "H": 7}, "G": null}}, {"B": false, "F": {"F": {"H": {"E": {"F": false}}, "I": {"A": {"A": {"A": {"H": false}, "D": null}, "E": false}, "F": {"A": [{"A": 8, "G": [null, false]}, true, {"B": {"D": true, "G": {"F": {"D": true}, "J": {"H": true}}, "I": {"C": 4, "D": 1}}}], "C": {"A": {"A": null, "C": 5, "G": 8}, "E": {"F": {"B": "E", "C": {"D": {"A": {"E": {"A": {"D": {"A": true, "D": false, "H": false}}, "E": {"E": "H", "G": 3}, "I": "F"}, "J": null}}, "I": false, "J": {"B": 5}}, "G": [{"A": {"D": {"B": {"D": "J", "E": {"I": 2}}, "C": {"E": [{"G": 1, "I": {"E": true}}]}, "I": [{"A": {"J": null}, "B": "E", "E": {"B": true, "C": false, "F": {"B": {"D": {"B": 8, "C": {"J": {"A": {"A": [2, [[true, {"E": 5, "F": true}, 5], "B", [false]]], "J": {"H": {"I": true}}}, "C": null, "H": null}}, "F": "B"}, "G": "J"}, "G": {"A": {"G": {"E": false}}, "B": {"F": {"C": [false, false, false], "G": {"A": null}, "H": {"H": false}}}, "D": {"H": {"G": true, "H": {"D": false, "H": {"B": "F", "G": {"G": {"E": {"G": [{"E": null, "F": [6, {"F": false}, [{"I": true}, null, {"A": {"G": 4, "I": {"A": {"F": "E", "G": [5]}, "J": {"I": 6}}, "J": {"H": false}}, "F": false, "H": {"G": false, "H": "H"}}]], "J": 0}]}, "I": {"E": false, "F": {"C": {"A": true}, "F": 2}, "J": {"G": null, "H": {"C": {"D": null, "I": {"B": false, "H": false, "J": "J"}}, "E": {"A": null, "G": {"A": {"A": true}, "F": {"D": true, "E": true, "F": null}}}}, "J": {"I": true}}}}, "H": false, "I": "I"}}}, "J": {"A": true, "G": {"F": true, "J": {"B": {"H": 1}, "C": null}}, "H": 3}}}}}}}, {"A": {"I": [[true, {"F": false}], false]}, "B": {"B": {"D": "B"}, "D": 0}, "G": {"F": "B", "G": [{"E": "A", "G": [true, false, {"C": {"D": true, "I": null}, "F": [{"H": {"D": 2, "I": [{"E": 3, "F": {"D": {"C": {"C": true, "D": {"C": null, "I": {"G": [null], "H": "J"}}, "H": {"D": {"A": [{"E": {"C": true}}, 2], "D": true, "E": false}, "I": [7, {"D": [false, [false], {"D": null}]}, null]}}, "J": null}, "H": 7}}, "E", {"H": 3}], "J": {"B": {"B": {"B": 1, "C": [false, false]}}}}, "J": {"B": true, "F": true}}, {"E": [{"B": 5}], "H": false, "I": "A"}, {"C": {"B": 4, "C": "I"}, "D": [2]}], "I": true}]}, null]}}, null]}, "I": false, "J": {"B": {"C": {"C": {"A": {"F": "F", "H": null, "I": null}, "I": true, "J": true}, "H": true}, "G": "J", "I": {"A": false, "G": {"A": [false]}}}, "E": {"D": true, "G": true}, "H": {"A": [true, {"F": {"C": false, "I": [{"A": {"G": null}, "I": {"B": {"C": "C"}, "E": [{"A": {"B": ["G"]}, "E": {"E": {"G": {"G": {"F": "E"}, "I": 1}}, "I": {"G": true, "J": true}}}, {"H": 0, "J": {"B": [true, {"J": 1}], "G": {"C": null, "F": null, "H": "B"}}}], "J": 8}}]}, "G": {"B": "J", "H": 1, "I": 8}, "H": true}, [null, {"B": 1, "C": {"E": {"A": {"D": {"A": {"J": {"B": false, "C": {"A": {"I": false}, "B": false, "I": {"H": true, "J": "A"}}, "E": {"G": {"B": {"H": {"B": "D"}}, "J": {"E": {"B": {"J": {"A": 3}}, "J": {"H": null}}, "F": "H", "J": false}}, "H": ["G"], "J": {"C": false, "E": null}}}}, "H": 9}, "J": false}, "G": null, "I": 9}, "I": {"I": "I"}}}]], "C": {"C": null}, "D": {"B": [false, [true, {"C": {"H": {"F": {"C": {"A": true, "F": {"B": {"C": {"A": true, "B": {"D": true, "F": [true, null], "J": [{"H": 8}, [true, {"C": null}, {"A": {"A": {"C": [null], "D": true, "E": {"A": {"E": true}, "E": null}}, "G": 1}}]]}}, "D": [{"F": {"A": ["B"]}}]}, "E": null}, "J": {"B": 2}}, "H": {"A": 2}}, "J": {"D": {"D": true}}}, "J": {"A": {"G": {"B": {"A": "D", "D": {"B": 1, "J": true}, "F": true}, "D": 7}, "I": "F"}, "G": {"E": {"I": null}}}}, "E": "I"}], true], "E": ["J", {"E": 9, "F": "A"}, false], "I": {"J": 4}}}}}, "E": "F", "I": false}, {"F": true}, [{"I": true}]]}}}, "D": {"E": null, "F": null, "H": null}}}}}}], "E": 5}, {"E": {"I": false}, "J": true}, 8]], {"H": true}]}, "E": true}, "F": {"A": {"C": {"B": [false, 6, "C"], "H": {"D": [{"E": 6}, 3, true]}}}}}, "E": null}, "J": true}], "F": true}, "H": {"J": {"B": {"A": null, "C": {"J": {"D": {"A": {"B": "F"}, "G": null, "J": "A"}}}}, "H": true, "I": "A"}}, "J": {"F": true, "H": true, "I": {"G": {"A": true, "J": [{"D": false}, null, "A"]}}}}]}}}], "J": [true, 4]}, 9, 6]]}, "I": true}], false], "H": true, "J": true}}, "E": [[null, "I"], true], "G": {"B": [4, [{"D": {"D": {"B": null, "D": "G"}}, "E": false}, "H"], "I"], "H": true}}':::JSONB::JSONB
        test_1        |                             AS " co̭l'186089"
        test_1        |                 )
        test_1        |                     AS "t\nab118649"
        test_1        |             ORDER BY
        test_1        |                 " co̭l'186089" ASC NULLS LAST
        test_1        |         compare_test.go:213: postgres: ping: conn closed
        test_1        | FAIL
        compare_test_1 exited with code 1
        Stopping compare_postgres_1   ... 
        Stopping compare_cockroach2_1 ... 
        Stopping compare_cockroach1_1 ... 
        Stopping compare_postgres_1   ... done
        Stopping compare_cockroach2_1 ... done
        Stopping compare_cockroach1_1 ... done
        Aborting on container exit...

    compose_test.go:125: exit status 1
--- FAIL: TestComposeCompare (1249.45s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #122339 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1] - #122079 compose: TestComposeCompare failed [O-robot P-2 T-sql-foundations branch-master] - #121592 compose: TestComposeCompare failed [no locations to init GEOS] [C-test-failure O-robot T-dev-inf branch-release-23.2] - #121531 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-24.1] - #120925 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom-v2] - #120924 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-custom] - #120868 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.18-rc] - #119903 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-23.1.17-rc] - #89361 compose: TestComposeCompare failed [O-robot T-sql-foundations branch-release-22.2]

This test on roachdash | Improve this report!