search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

ccl/changefeedccl: TestChangefeedRandomExpressions failed #96052

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 1 year ago

ccl/changefeedccl.TestChangefeedRandomExpressions failed with artifacts on master @ 1d901eeca5c187f9d80fb52f985c1224de57762d:

    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE EXISTS (SELECT seed._timestamptz AS col🙂3928, e'\'DLJX\'':::TSQUERY AS col3929 FROM d.public.seed@[0] AS tab1473 WHERE (seed._inet::INET >> seed._inet::INET)::BOOL ORDER BY seed._enum DESC NULLS FIRST, seed._int4 DESC NULLS LAST, seed.tableoid ASC NULLS FIRST)
    changefeed_test.go:932: Skipping predicate EXISTS (SELECT seed._timestamptz AS col🙂3928, e'\'DLJX\'':::TSQUERY AS col3929 FROM d.public.seed@[0] AS tab1473 WHERE (seed._inet::INET >> seed._inet::INET)::BOOL ORDER BY seed._enum DESC NULLS FIRST, seed._int4 DESC NULLS LAST, seed.tableoid ASC NULLS FIRST) because it returned no rows
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (NOT (seed._inet::INET << seed._inet::INET)::BOOL)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE (NOT (seed._inet::INET << seed._inet::INET)::BOOL) to emit 91 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE seed.tableoid::OID IN (2306332459:::OID, 0:::OID, seed.tableoid, NULL)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE seed.tableoid::OID IN (2306332459:::OID, 0:::OID, seed.tableoid, NULL) to emit 100 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE '{"a": [[false, true, true], false, false, false], "fxO": {"a": "b", "a86B": {"a": 0.22875536143744696, "qNji8": null}, "kAOWw2Ga": null}, "pNO40qz": 0.6630852468307362}':::JSONB::JSONB NOT IN (SELECT '[[[null, 0.7566807655820729], [false, "cv7"], "jrqYA4BDX"], false, ["CYMBG2", false], false]':::JSONB::JSONB AS col3978 FROM d.public.seed@[0] AS tab1486 WHERE false GROUP BY seed._jsonb HAVING true)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE '{"a": [[false, true, true], false, false, false], "fxO": {"a": "b", "a86B": {"a": 0.22875536143744696, "qNji8": null}, "kAOWw2Ga": null}, "pNO40qz": 0.6630852468307362}':::JSONB::JSONB NOT IN (SELECT '[[[null, 0.7566807655820729], [false, "cv7"], "jrqYA4BDX"], false, ["CYMBG2", false], false]':::JSONB::JSONB AS col3978 FROM d.public.seed@[0] AS tab1486 WHERE false GROUP BY seed._jsonb HAVING true) to emit 100 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:939: Test tolerating create changefeed error: failed to start feed for job 0: pq: sub-query expressions not supported by CDC
    helpers_test.go:703: runtime error: invalid memory address or nil pointer dereference
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (seed._inet::INET << seed._inet::INET)::BOOL
    changefeed_test.go:932: Skipping predicate (seed._inet::INET << seed._inet::INET)::BOOL because it returned no rows
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE overlaps(current_timestamp():::DATE::DATE::DATE, '-26 years -8 mons -365 days -02:42:15.598445':::INTERVAL::INTERVAL, parse_date(e'Kj\x1cc+L':::STRING::STRING)::DATE::DATE, '56 years 509 days 11:50:12.885432':::INTERVAL::INTERVAL)::BOOL
    changefeed_test.go:927: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE overlaps(current_timestamp():::DATE::DATE::DATE, '-26 years -8 mons -365 days -02:42:15.598445':::INTERVAL::INTERVAL, parse_date(e'Kj\x1cc+L':::STRING::STRING)::DATE::DATE, '56 years 509 days 11:50:12.885432':::INTERVAL::INTERVAL)::BOOL because error pq: parse_date(): parsing as type date: could not parse "Kj�c+L"
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE seed._string LIKE seed._string
    changefeed_test.go:927: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE seed._string LIKE seed._string because error pq: LIKE regexp compilation failed: LIKE pattern must not end with escape character
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (seed._bool AND seed._bool)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE (seed._bool AND seed._bool) to emit 55 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE NULL::BOX2D NOT IN (SELECT 'BOX(-0.34990853140301237 -0.10626986308883635,1.0910843777296364 1.4774058953771299)':::BOX2D::BOX2D AS col4096 FROM d.public.seed@[0] AS tab1533 GROUP BY seed.tableoid, seed._date, seed._inet HAVING seed._bool ORDER BY seed._inet ASC NULLS FIRST, seed._inet DESC NULLS LAST)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE NULL::BOX2D NOT IN (SELECT 'BOX(-0.34990853140301237 -0.10626986308883635,1.0910843777296364 1.4774058953771299)':::BOX2D::BOX2D AS col4096 FROM d.public.seed@[0] AS tab1533 GROUP BY seed.tableoid, seed._date, seed._inet HAVING seed._bool ORDER BY seed._inet ASC NULLS FIRST, seed._inet DESC NULLS LAST) to emit 45 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:939: Test tolerating create changefeed error: failed to start feed for job 0: pq: sub-query expressions not supported by CDC
    helpers_test.go:703: runtime error: invalid memory address or nil pointer dereference
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (NOT EXISTS (SELECT seed._string AS "c
        ol4127"))
    changefeed_test.go:932: Skipping predicate (NOT EXISTS (SELECT seed._string AS "c
        ol4127")) because it returned no rows
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (NOT ((NULL::INET << seed._inet::INET)::BOOL AND ('11:50:40.46941-13:20:00':::TIMETZ < ((parse_timetz(seed._string::STRING, e'y\x0fEl1Yx':::STRING::STRING)::TIMETZ::TIMETZ - seed._interval::INTERVAL)::TIMETZ::TIMETZ - seed._interval::INTERVAL)::TIMETZ)))
    changefeed_test.go:927: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (NOT ((NULL::INET << seed._inet::INET)::BOOL AND ('11:50:40.46941-13:20:00':::TIMETZ < ((parse_timetz(seed._string::STRING, e'y\x0fEl1Yx':::STRING::STRING)::TIMETZ::TIMETZ - seed._interval::INTERVAL)::TIMETZ::TIMETZ - seed._interval::INTERVAL)::TIMETZ))) because error pq: parse_timetz(): unknown DateStyle parameter: y�el1yx
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (ltrim('$TJ':::STRING::STRING, to_char('1990-02-24 12:01:25.000111+00:00':::TIMESTAMPTZ::TIMESTAMPTZ, to_char('45 years 2 mons 967 days 06:24:22.4176':::INTERVAL::INTERVAL)::STRING::STRING)::STRING::STRING)::STRING::STRING || '1 day':::INTERVAL::INTERVAL)::STRING ILIKE NULL
    changefeed_test.go:932: Skipping predicate (ltrim('$TJ':::STRING::STRING, to_char('1990-02-24 12:01:25.000111+00:00':::TIMESTAMPTZ::TIMESTAMPTZ, to_char('45 years 2 mons 967 days 06:24:22.4176':::INTERVAL::INTERVAL)::STRING::STRING)::STRING::STRING)::STRING::STRING || '1 day':::INTERVAL::INTERVAL)::STRING ILIKE NULL because it returned no rows
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE ((e'5s\x0ezy':::STRING ILIKE seed._string) OR (seed._inet::INET >> '46c5:1434:55f5:acc6:9c0d:8b75:6937:93b5/10':::INET::INET)::BOOL)
    changefeed_test.go:927: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE ((e'5s\x0ezy':::STRING ILIKE seed._string) OR (seed._inet::INET >> '46c5:1434:55f5:acc6:9c0d:8b75:6937:93b5/10':::INET::INET)::BOOL) because error pq: LIKE regexp compilation failed: LIKE pattern must not end with escape character
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE seed._timestamptz::TIMESTAMPTZ IN (seed._timestamptz, '2003-06-21 17:35:58.000032+00:00':::TIMESTAMPTZ, '1974-06-25 22:30:16.000666+00:00':::TIMESTAMPTZ)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE seed._timestamptz::TIMESTAMPTZ IN (seed._timestamptz, '2003-06-21 17:35:58.000032+00:00':::TIMESTAMPTZ, '1974-06-25 22:30:16.000666+00:00':::TIMESTAMPTZ) to emit 92 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (seed._inet::INET << inet(seed._string::STRING)::INET::INET)::BOOL
    changefeed_test.go:927: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (seed._inet::INET << inet(seed._string::STRING)::INET::INET)::BOOL because error pq: inet(): could not parse "\x0f\f\x01oO^" as inet. invalid IP
    changefeed_test.go:955: 42 predicates checked: all had the same result in SELECT and CHANGEFEED
    --- FAIL: TestChangefeedRandomExpressions/kafka (25.58s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-23906

Epic CRDB-11732

HonoreDB commented 1 year ago

Weird, the test output here is for a successful test: it ends with predicates checked: all had the same result in SELECT and CHANGEFEED which is logged in the last line of the test. Wondering if the use of a panic/recover in closeFeedIgnoreError is confusing the test framework, or something about the output looks like an error, or the test was run multiple times and the wrong output was logged? This passes locally using ./dev test, so it's something specific to how the test ran in bazci.

miretskiy commented 1 year ago

dev stress perhaps to run this many times? Do we have a test seed printed somewhere to try to repro same run?

HonoreDB commented 1 year ago

I can't deterministically repro this, but running under stress+race is turning up some errors in rpc JSON encoding that are maybe somehow the root cause here?

1:

WARNING: DATA RACE
Write at 0x00c004c162f0 by goroutine 9774:
  github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchDirectScan).Release()
      github.com/cockroachdb/cockroach/pkg/sql/colfetcher/colbatch_direct_scan.go:147 +0x53
  github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecargs.(*NewColOperatorResult).Release()
      github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecargs/op_creation.go:139 +0x617
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).Release()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:703 +0x8d0
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Release()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:352 +0x47
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Cleanup()
      github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:401 +0x52b
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run.func2()
      github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:833 +0x61
  runtime.deferreturn()
      GOROOT/src/runtime/panic.go:476 +0x32
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun()
      github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1793 +0x2ec
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll()
      github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1544 +0x34b
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1641 +0xa84
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1262 +0x1624
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:744 +0x3a91
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:130 +0x18e
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2530 +0x50f
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129 +0x6cd
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:219 +0x217
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2022 +0xad7
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2024 +0xbd6
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1866 +0x424
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx.func1()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:171 +0x117

Previous read at 0x00c004c162f0 by goroutine 388:
  reflect.Value.Index()
      GOROOT/src/reflect/value.go:1416 +0x1ef
  encoding/json.arrayEncoder.encode()
      GOROOT/src/encoding/json/encode.go:915 +0xe7
  encoding/json.arrayEncoder.encode-fm()
      <autogenerated>:1 +0x90
  encoding/json.sliceEncoder.encode()
      GOROOT/src/encoding/json/encode.go:888 +0x4e2
  encoding/json.sliceEncoder.encode-fm()
      <autogenerated>:1 +0x90
  encoding/json.structEncoder.encode()
      GOROOT/src/encoding/json/encode.go:760 +0x2ba
  encoding/json.structEncoder.encode-fm()
      <autogenerated>:1 +0xdb
  encoding/json.ptrEncoder.encode()
      GOROOT/src/encoding/json/encode.go:944 +0x3c2
  encoding/json.ptrEncoder.encode-fm()
      <autogenerated>:1 +0x90
  encoding/json.structEncoder.encode()
      GOROOT/src/encoding/json/encode.go:760 +0x2ba
  encoding/json.structEncoder.encode-fm()
      <autogenerated>:1 +0xdb
  encoding/json.structEncoder.encode()
      GOROOT/src/encoding/json/encode.go:760 +0x2ba
  encoding/json.structEncoder.encode-fm()
      <autogenerated>:1 +0xdb
  encoding/json.ptrEncoder.encode()
      GOROOT/src/encoding/json/encode.go:944 +0x3c2
  encoding/json.ptrEncoder.encode-fm()
      <autogenerated>:1 +0x90
  encoding/json.(*encodeState).reflectValue()
      GOROOT/src/encoding/json/encode.go:359 +0x88
  encoding/json.(*encodeState).marshal()
      GOROOT/src/encoding/json/encode.go:331 +0x20b
  encoding/json.(*Encoder).Encode()
      GOROOT/src/encoding/json/stream.go:206 +0xcf
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.GRPCTransportFactory.func1()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:120 +0x2c4
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f6

2:

* ERROR: a panic has occurred!
* panic: reflect: slice index out of range
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:890
*   | encoding/json.(*encodeState).marshal.func1
*   |   GOROOT/src/encoding/json/encode.go:327
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:890
*   | reflect.Value.Index
*   |   GOROOT/src/reflect/value.go:1412
*   | encoding/json.arrayEncoder.encode
*   |   GOROOT/src/encoding/json/encode.go:915
*   | encoding/json.sliceEncoder.encode
*   |   GOROOT/src/encoding/json/encode.go:888
*   | encoding/json.structEncoder.encode
*   |   GOROOT/src/encoding/json/encode.go:760
*   | encoding/json.ptrEncoder.encode
*   |   GOROOT/src/encoding/json/encode.go:944
*   | encoding/json.structEncoder.encode
*   |   GOROOT/src/encoding/json/encode.go:760
*   | encoding/json.structEncoder.encode
*   |   GOROOT/src/encoding/json/encode.go:760
*   | encoding/json.ptrEncoder.encode
*   |   GOROOT/src/encoding/json/encode.go:944
*   | encoding/json.(*encodeState).reflectValue
*   |   GOROOT/src/encoding/json/encode.go:359
*   | encoding/json.(*encodeState).marshal
*   |   GOROOT/src/encoding/json/encode.go:331
*   | encoding/json.(*Encoder).Encode
*   |   GOROOT/src/encoding/json/stream.go:206
*   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.GRPCTransportFactory.func1
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:120
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   |   src/runtime/asm_amd64.s:1594
* Wraps: (2) panic: reflect: slice index out of range
* Error types: (1) *withstack.withStack (2) *errutil.leafError
*
panic: reflect: slice index out of range [recovered]
    panic: reflect: slice index out of range [recovered]
    panic: reflect: slice index out of range
HonoreDB commented 1 year ago

These stack traces look like they're specific to race builds so possibly unrelated. But surprising to see a panic deep into a standard library.

cockroach-teamcity commented 1 year ago

ccl/changefeedccl.TestChangefeedRandomExpressions failed with artifacts on master @ 639decf058c78f0afe0dba613d60670628c92e81:

    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (((true OR false) OR ((e'\x11':::STRING::STRING || (B'10001010101000111101110010000110011011000101011011111100010110011110101111010101011111'::VARBIT | B'00101101100011010001100011010110110101011011010'::VARBIT)::VARBIT::VARBIT)::STRING NOT ILIKE (st_asgeojson(e'&8\x07':::STRING::STRING, NULL::INT8, (-5254389604683114372):::INT8::INT8)::STRING::STRING || ((-1035320416713543333):::INT8::INT8 // ((-6448369151594613222):::INT8::INT8 * NULL::INT8)::INT8::INT8)::INT8::INT8)::STRING)) AND false)
    changefeed_test.go:932: Skipping predicate (((true OR false) OR ((e'\x11':::STRING::STRING || (B'10001010101000111101110010000110011011000101011011111100010110011110101111010101011111'::VARBIT | B'00101101100011010001100011010110110101011011010'::VARBIT)::VARBIT::VARBIT)::STRING NOT ILIKE (st_asgeojson(e'&8\x07':::STRING::STRING, NULL::INT8, (-5254389604683114372):::INT8::INT8)::STRING::STRING || ((-1035320416713543333):::INT8::INT8 // ((-6448369151594613222):::INT8::INT8 * NULL::INT8)::INT8::INT8)::INT8::INT8)::STRING)) AND false) because it returned no rows
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE EXISTS (SELECT B'011010101100110100001101110111011101000000' AS "coL4356" FROM d.public.seed@[0] AS tab1515)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE EXISTS (SELECT B'011010101100110100001101110111011101000000' AS "coL4356" FROM d.public.seed@[0] AS tab1515) to emit 100 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:939: Test tolerating create changefeed error: failed to start feed for job 0: pq: sub-query expressions not supported by CDC
    helpers_test.go:703: runtime error: invalid memory address or nil pointer dereference
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE 'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID::UUID NOT IN (SELECT seed._uuid::UUID AS col4381 FROM d.public.seed@[0] AS tab😎1522 WHERE crdb_internal.pretty_key(seed._bytes::BYTES, seed._int2::INT8)::STRING ILIKE seed._string ORDER BY seed._interval DESC NULLS LAST, seed._timestamptz NULLS LAST, seed._bytes ASC NULLS LAST)
    changefeed_test.go:927: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE 'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID::UUID NOT IN (SELECT seed._uuid::UUID AS col4381 FROM d.public.seed@[0] AS tab😎1522 WHERE crdb_internal.pretty_key(seed._bytes::BYTES, seed._int2::INT8)::STRING ILIKE seed._string ORDER BY seed._interval DESC NULLS LAST, seed._timestamptz NULLS LAST, seed._bytes ASC NULLS LAST) because error pq: LIKE regexp compilation failed: LIKE pattern must not end with escape character
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE seed._string NOT LIKE seed._string
    changefeed_test.go:927: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE seed._string NOT LIKE seed._string because error pq: LIKE regexp compilation failed: LIKE pattern must not end with escape character
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE EXISTS (SELECT B'000000011010110000011000000011101' AS col4477 FROM d.public.seed@[0] AS "ta b1548" WHERE true)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE EXISTS (SELECT B'000000011010110000011000000011101' AS col4477 FROM d.public.seed@[0] AS "ta b1548" WHERE true) to emit 100 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:939: Test tolerating create changefeed error: failed to start feed for job 0: pq: sub-query expressions not supported by CDC
    helpers_test.go:703: runtime error: invalid memory address or nil pointer dereference
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE NULL::greeting NOT IN (SELECT 'hi':::greeting::greeting AS 🙃col4480 FROM d.public.seed@[0] AS tab1550 WHERE false ORDER BY seed.tableoid NULLS LAST, seed._timestamp DESC NULLS LAST)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE NULL::greeting NOT IN (SELECT 'hi':::greeting::greeting AS 🙃col4480 FROM d.public.seed@[0] AS tab1550 WHERE false ORDER BY seed.tableoid NULLS LAST, seed._timestamp DESC NULLS LAST) to emit 100 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:939: Test tolerating create changefeed error: failed to start feed for job 0: pq: sub-query expressions not supported by CDC
    helpers_test.go:703: runtime error: invalid memory address or nil pointer dereference
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE ((NULL AND ((-5652715609055749711):::INT8::INT8 NOT IN ((-1468674142098467087):::INT8, st_numinteriorring(st_multipointfromtext(e'\'':::STRING::STRING, 173716407849516012:::INT8::INT8)::GEOMETRY::GEOMETRY)::INT8, 1178245315123507576:::INT8))) AND true)
    changefeed_test.go:932: Skipping predicate ((NULL AND ((-5652715609055749711):::INT8::INT8 NOT IN ((-1468674142098467087):::INT8, st_numinteriorring(st_multipointfromtext(e'\'':::STRING::STRING, 173716407849516012:::INT8::INT8)::GEOMETRY::GEOMETRY)::INT8, 1178245315123507576:::INT8))) AND true) because it returned no rows
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (seed._bool AND NULL)
    changefeed_test.go:932: Skipping predicate (seed._bool AND NULL) because it returned no rows
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (seed._timestamptz < (seed._timestamptz::TIMESTAMPTZ - '00:00:00':::INTERVAL::INTERVAL)::TIMESTAMPTZ)
    changefeed_test.go:932: Skipping predicate (seed._timestamptz < (seed._timestamptz::TIMESTAMPTZ - '00:00:00':::INTERVAL::INTERVAL)::TIMESTAMPTZ) because it returned no rows
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (seed._string::STRING % seed._string::STRING)::BOOL
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE (seed._string::STRING % seed._string::STRING)::BOOL to emit 48 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE EXISTS (SELECT seed.tableoid AS col4600)
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE EXISTS (SELECT seed.tableoid AS col4600) to emit 100 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:939: Test tolerating create changefeed error: failed to start feed for job 0: pq: sub-query expressions not supported by CDC
    helpers_test.go:703: runtime error: invalid memory address or nil pointer dereference
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE EXISTS (SELECT CASE WHEN seed._bool THEN seed._date ELSE seed._date END AS col4613 FROM d.public.seed AS tab1595 JOIN d.public.seed AS tab1596 ON (tab1595._jsonb) = (tab1596._jsonb))
    changefeed_test.go:936: Expecting statement CREATE CHANGEFEED WITH schema_change_policy='stop' AS SELECT rowid FROM seed WHERE EXISTS (SELECT CASE WHEN seed._bool THEN seed._date ELSE seed._date END AS col4613 FROM d.public.seed AS tab1595 JOIN d.public.seed AS tab1596 ON (tab1595._jsonb) = (tab1596._jsonb)) to emit 100 events
    testfeed_test.go:278: creating external connection
    testfeed_test.go:281: ran create external connection
    changefeed_test.go:939: Test tolerating create changefeed error: failed to start feed for job 0: pq: sub-query expressions not supported by CDC
    helpers_test.go:703: runtime error: invalid memory address or nil pointer dereference
    changefeed_test.go:923: SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (inet('':::STRING::STRING)::INET::INET >> set_masklen('1315:f15e:350a:92bf:a958:dcce:bc2e:3778/90':::INET::INET, 15441268322837485:::INT8::INT8)::INET::INET)::BOOL
    changefeed_test.go:927: Skipping query SELECT array_to_string(IFNULL(array_agg(distinct rowid),'{}'),'|') FROM seed WHERE (inet('':::STRING::STRING)::INET::INET >> set_masklen('1315:f15e:350a:92bf:a958:dcce:bc2e:3778/90':::INET::INET, 15441268322837485:::INT8::INT8)::INET::INET)::BOOL because error pq: inet(): could not parse "" as inet. invalid IP
    changefeed_test.go:955: 30 predicates checked: all had the same result in SELECT and CHANGEFEED
    --- FAIL: TestChangefeedRandomExpressions/kafka (13.48s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

mgartner commented 1 year ago

Failed again here: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/8525721?showRootCauses=true&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

andrewbaptist commented 1 year ago

Failed here also: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/8530993?showRootCauses=true&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

ajwerner commented 1 year ago

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/8545559?showRootCauses=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true on a bors run