search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.1k stars 3.81k forks source link

sql/logictest: TestLogic failed #60824

Closed cockroach-teamcity closed 3 years ago

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@557346d5fc755db5f64d3e198e47be1f53a4d390:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic977178260
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3343: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic977178260
--- FAIL: TestLogic (958.82s)
=== RUN   TestLogic/5node-metadata/distsql_numtables
=== PAUSE TestLogic/5node-metadata/distsql_numtables
=== CONT  TestLogic/5node-metadata/distsql_numtables
=== CONT  TestLogic/5node-metadata/distsql_numtables
    logic.go:2559: 

        testdata/logic_test/distsql_numtables:21: 
        expected success, but found
        (XXUUU) cannot up-replicate to s2; missing gossiped StoreDescriptor
--- done: testdata/logic_test/distsql_numtables with config 5node-metadata: 4 tests, 1 failures
E210220 00:01:04.366080 12116468 sql/logictest/logic.go:3575  [-] 1008  pq: cannot up-replicate to s2; missing gossiped StoreDescriptor
    logic.go:3118: 
        testdata/logic_test/distsql_numtables:21: error while processing
    logic.go:3118: pq: cannot up-replicate to s2; missing gossiped StoreDescriptor
--- done: testdata/logic_test/merge_join_dist with config 5node-metadata: 12 tests, 0 failures
        --- FAIL: TestLogic/5node-metadata/distsql_numtables (1.66s)
=== RUN   TestLogic/5node-metadata
    --- FAIL: TestLogic/5node-metadata (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@ab07ae1da6fe154eebc6c6847cca58b30184e14f:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic233649622
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3343: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic233649622
--- FAIL: TestLogic (740.75s)
=== RUN   TestLogic/5node-metadata/distsql_subquery
=== PAUSE TestLogic/5node-metadata/distsql_subquery
=== CONT  TestLogic/5node-metadata/distsql_subquery
=== CONT  TestLogic/5node-metadata/distsql_subquery
    logic.go:2559: 

        testdata/logic_test/distsql_subquery:18: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:43885: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/distsql_subquery with config 5node-metadata: 4 tests, 1 failures
E210222 19:54:48.567346 10789584 sql/logictest/logic.go:3575  [-] 1058  pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:43885: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3118: 
        testdata/logic_test/distsql_subquery:18: error while processing
    logic.go:3118: pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:43885: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/5node-metadata/distsql_subquery (1.70s)
=== RUN   TestLogic/5node-metadata
    --- FAIL: TestLogic/5node-metadata (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

yuzefovich commented 3 years ago

The first report seems possibly related to #59340.

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@bc98209ed36ec47675fe3d7a7d8201b4b77f89ba:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic516258408
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3343: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic516258408
--- FAIL: TestLogic (1031.64s)
=== RUN   TestLogic/fakedist-spec-planning/errors
=== PAUSE TestLogic/fakedist-spec-planning/errors
=== CONT  TestLogic/fakedist-spec-planning/errors
E210222 20:27:49.956148 10856322 jobs/adopt.go:117  [-] 7525  could not run claimed job 635552627370852353: job 635552627370852353: status changed to succeeded which is not resumable`
E210222 20:27:49.986725 10857012 jobs/adopt.go:117  [-] 7526  could not run claimed job 635552627323830273: job 635552627323830273: status changed to succeeded which is not resumable`
E210222 20:27:50.008918 10857561 jobs/adopt.go:117  [-] 7527  could not run claimed job 635552627463684097: job 635552627463684097: status changed to succeeded which is not resumable`
--- progress: testdata/logic_test/show_create_all_tables: 9 statements/queries
E210222 20:27:50.032915 10858483 jobs/adopt.go:117  [-] 7528  could not run claimed job 635552627347259393: job 635552627347259393: status changed to succeeded which is not resumable`
E210222 20:27:50.034966 10858482 jobs/adopt.go:117  [-] 7529  could not run claimed job 635552627423838209: job 635552627423838209: status changed to succeeded which is not resumable`
=== CONT  TestLogic/fakedist-spec-planning/errors
    logic.go:3122: 
        testdata/logic_test/errors:19: error while processing
    logic.go:3122: test was successful but validation upon completion failed: failed to test for descriptor JSON round-trip: pq: failed to connect to n3 at 127.0.0.1:34735: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/system with config fakedist-spec-planning: 65 tests, 0 failures
        --- FAIL: TestLogic/fakedist-spec-planning/errors (1.85s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.72s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@5cd07e02c291d017bedd2a237956a62eb63dbcc9:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic237413412
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic237413412
--- FAIL: TestLogic (1294.75s)
=== RUN   TestLogic/fakedist-spec-planning/show_indexes
=== PAUSE TestLogic/fakedist-spec-planning/show_indexes
=== CONT  TestLogic/fakedist-spec-planning/show_indexes
E210225 01:17:46.717279 11401034 jobs/adopt.go:117  [-] 7559  could not run claimed job 636175865142411265: job 636175865142411265: status changed to succeeded which is not resumable`
E210225 01:17:46.734043 11401736 jobs/adopt.go:117  [-] 7560  could not run claimed job 636175865086050305: job 636175865086050305: status changed to succeeded which is not resumable`
E210225 01:17:46.739681 11402071 jobs/adopt.go:117  [-] 7561  could not run claimed job 636175865136676865: job 636175865136676865: status changed to succeeded which is not resumable`
E210225 01:17:46.756434 11402182 kv/kvclient/kvcoord/txn_interceptor_committer.go:449  [-] 7562  making txn commit explicit failed for "unnamed" meta={id=39096c6e pri=0.05947514 epo=0 ts=1614215866.731093780,0 min=1614215866.731093780,0 seq=2} lock=true stat=STAGING rts=1614215866.731093780,0 wto=false gul=1614215867.231093780,0 int=1 ifw=1: result is ambiguous (server shutdown)
=== CONT  TestLogic/fakedist-spec-planning/show_indexes
    testcluster.go:1439: failed to create a status client because of initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/fakedist-spec-planning/show_indexes (1.54s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.66s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

RaduBerinde commented 3 years ago

Same failure like the one just above: https://teamcity.cockroachdb.com//viewLog.html?buildId=2716122&buildTypeId=Cockroach_MergeToMaster

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@ac25d9f3511435c4e6962f71eb169fd6d7d0ef28:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic596100037
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic596100037
--- FAIL: TestLogic (752.70s)
=== RUN   TestLogic/5node/distsql_distinct_on
=== PAUSE TestLogic/5node/distsql_distinct_on
=== CONT  TestLogic/5node/distsql_distinct_on
=== CONT  TestLogic/5node/distsql_distinct_on
    logic.go:2567: 

        testdata/logic_test/distsql_distinct_on:40: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:44201: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/distsql_distinct_on with config 5node: 5 tests, 1 failures
E210226 02:36:17.364026 10710610 sql/logictest/logic.go:3583  [-] 7053  pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:44201: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3126: 
        testdata/logic_test/distsql_distinct_on:40: error while processing
    logic.go:3126: pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:44201: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/5node/distsql_distinct_on (2.08s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@dcc8ab04251aa954f086cd2b347cd1d1b976a312:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic228119198
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic228119198
--- FAIL: TestLogic (971.83s)
=== RUN   TestLogic/fakedist-spec-planning/txn_stats
=== PAUSE TestLogic/fakedist-spec-planning/txn_stats
=== CONT  TestLogic/fakedist-spec-planning/txn_stats
=== CONT  TestLogic/fakedist-spec-planning/txn_stats
    logic.go:3130: 
        testdata/logic_test/txn_stats:44: error while processing
    logic.go:3130: test was successful but validation upon completion failed: failed to test for descriptor JSON round-trip: pq: failed to connect to n2 at 127.0.0.1:33081: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/fakedist-spec-planning/txn_stats (1.39s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.58s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@6521a8e138c9838853f9502303faf99da69aad2d:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic409701549
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic409701549
--- FAIL: TestLogic (998.67s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.56s)
=== RUN   TestLogic/fakedist-spec-planning/alter_sequence
=== PAUSE TestLogic/fakedist-spec-planning/alter_sequence
=== CONT  TestLogic/fakedist-spec-planning/alter_sequence
--- done: testdata/logic_test/save_table with config fakedist-spec-planning: 21 tests, 0 failures
=== CONT  TestLogic/fakedist-spec-planning/alter_sequence
    testcluster.go:1439: failed to create a status client because of initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/fakedist-spec-planning/alter_sequence (1.43s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@ae1e417e061cc1b869927959e8830e0e5cde2622:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic044184690
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic044184690
--- FAIL: TestLogic (988.19s)
=== RUN   TestLogic/fakedist-spec-planning/ordinality
=== PAUSE TestLogic/fakedist-spec-planning/ordinality
=== CONT  TestLogic/fakedist-spec-planning/ordinality
--- done: testdata/logic_test/partial_index with config fakedist-spec-planning: 316 tests, 0 failures
=== CONT  TestLogic/fakedist-spec-planning/ordinality
    logic.go:3130: 
        testdata/logic_test/ordinality:102: error while processing
    logic.go:3130: test was successful but validation upon completion failed: failed to test for descriptor JSON round-trip: pq: failed to connect to n2 at 127.0.0.1:43669: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
E210302 05:25:10.426834 11625673 sql/stats/automatic_stats.go:302  [-] 1469  failed to refresh stats: node unavailable; try another peer
        --- FAIL: TestLogic/fakedist-spec-planning/ordinality (1.50s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.55s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@f0c2d8a2f762ee33ca044ae86f46d06fc35682db:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic976886314
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic976886314
--- FAIL: TestLogic (949.79s)
=== RUN   TestLogic/fakedist-vec-off/inverted_join_geospatial
=== PAUSE TestLogic/fakedist-vec-off/inverted_join_geospatial
=== CONT  TestLogic/fakedist-vec-off/inverted_join_geospatial
--- done: testdata/logic_test/jobs with config fakedist-vec-off: 21 tests, 0 failures
--- done: testdata/logic_test/txn_retry with config fakedist-vec-off: 6 tests, 0 failures
--- done: testdata/logic_test/txn_as_of with config fakedist-vec-off: 72 tests, 0 failures
E210305 14:21:54.181701 5921621 jobs/adopt.go:260  [n1] 856  job 638594954554540033: adoption completed with error job 638594954554540033: could not mark as reverting: log-job: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer: log-job: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer
--- total progress: 75201 statements/queries
=== CONT  TestLogic/fakedist-vec-off/inverted_join_geospatial
    logic.go:2567: 

        testdata/logic_test/inverted_join_geospatial:33: SELECT lk, rk FROM ltable JOIN rtable@geom_index ON ST_Intersects(ltable.geom1, rtable.geom) ORDER BY (lk, rk)
        expected success, but found
        (XXUUU) failed to connect to n2 at 127.0.0.1:40659: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210305 14:21:55.397713 5324437 sql/logictest/logic.go:3558  [-] 858  
E210305 14:21:55.397713 5324437 sql/logictest/logic.go:3558  [-] 858 +pq: failed to connect to n2 at 127.0.0.1:40659: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2300: 
         pq: failed to connect to n2 at 127.0.0.1:40659: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/inverted_join_geospatial with config fakedist-vec-off: 5 tests, 2 failures
E210305 14:21:55.398118 5324437 sql/logictest/logic.go:3583  [-] 859  testdata/logic_test/inverted_join_geospatial:45: too many errors encountered, skipping the rest of the input
    logic.go:3126: 
        testdata/logic_test/inverted_join_geospatial:45: error while processing
    logic.go:3126: testdata/logic_test/inverted_join_geospatial:45: too many errors encountered, skipping the rest of the input
--- progress: testdata/logic_test/join: 113 statements/queries
        --- FAIL: TestLogic/fakedist-vec-off/inverted_join_geospatial (1.41s)
=== RUN   TestLogic/fakedist-vec-off
    --- FAIL: TestLogic/fakedist-vec-off (0.54s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@a69e6549a71f5a0e83eb13509001f4d7351050fb:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic150275333
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic150275333
--- FAIL: TestLogic (1873.11s)
=== RUN   TestLogic/fakedist-metadata/alter_primary_key/create_table_change_pk
=== CONT  TestLogic/fakedist-metadata/alter_primary_key/create_table_change_pk
    logic.go:2567: 

        testdata/logic_test/alter_primary_key:1332: 
        expected success, but found
        (42703) column "crdb_internal_id_shard_10" does not exist
        column_resolver.go:206: in NewUndefinedColumnError()
E210307 08:35:42.725685 7777196 sql/logictest/logic.go:3558  [-] 821  
E210307 08:35:42.725685 7777196 sql/logictest/logic.go:3558  [-] 821 +pq: column "crdb_internal_id_shard_10" does not exist
    logic.go:1844: 
         pq: column "crdb_internal_id_shard_10" does not exist
--- done: testdata/logic_test/alter_primary_key with config fakedist-metadata: 227 tests, 2 failures
--- progress: testdata/logic_test/alter_table: 300 statements/queries
--- total progress: 102482 statements/queries
            --- FAIL: TestLogic/fakedist-metadata/alter_primary_key/create_table_change_pk (2.55s)
=== RUN   TestLogic/fakedist-metadata/alter_primary_key
=== PAUSE TestLogic/fakedist-metadata/alter_primary_key
=== CONT  TestLogic/fakedist-metadata/alter_primary_key
--- progress: testdata/logic_test/alter_table: 68 statements/queries
E210307 08:35:34.903290 7753854 jobs/adopt.go:260  [n1] 806  job 639093095472267265: adoption completed with error validation of CHECK "i < 0:::INT8" failed on row: a=1, x=NULL, y=1.3, z=1.4, d=NULL, f=NULL, g=1, h=NULL, i=1
E210307 08:35:35.153586 7754777 jobs/adopt.go:260  [n1] 807  job 639093096392097793: adoption completed with error validation of CHECK "i < g" failed on row: a=1, x=NULL, y=1.3, z=1.4, d=NULL, f=NULL, g=1, h=NULL, i=1
        --- FAIL: TestLogic/fakedist-metadata/alter_primary_key (9.63s)
=== RUN   TestLogic/fakedist-metadata
    --- FAIL: TestLogic/fakedist-metadata (0.59s)
More

Parameters: - TAGS= - GOFLAGS=-parallel=2 ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

jordanlewis commented 3 years ago 
     pq: column "crdb_internal_id_shard_10" does not exist

Hmmm... @ajwerner

ajwerner commented 3 years ago

Hmm indeed

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@89c379950cb4ca77c5d25e45aac4008b1f1758d9:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic914265095
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic914265095
--- FAIL: TestLogic (752.85s)
=== RUN   TestLogic/fakedist-spec-planning/alter_primary_key/create_table_change_pk
=== CONT  TestLogic/fakedist-spec-planning/alter_primary_key/create_table_change_pk
    logic.go:2567: 

        testdata/logic_test/alter_primary_key:1332: 
        expected success, but found
        (42703) column "crdb_internal_id_shard_10" does not exist
        column_resolver.go:206: in NewUndefinedColumnError()
E210309 05:57:07.925464 10761106 sql/logictest/logic.go:3558  [-] 1191  
E210309 05:57:07.925464 10761106 sql/logictest/logic.go:3558  [-] 1191 +pq: column "crdb_internal_id_shard_10" does not exist
    logic.go:1844: 
         pq: column "crdb_internal_id_shard_10" does not exist
--- done: testdata/logic_test/alter_primary_key with config fakedist-spec-planning: 227 tests, 2 failures
--- done: testdata/logic_test/interleaved with config fakedist-spec-planning: 99 tests, 0 failures
            --- FAIL: TestLogic/fakedist-spec-planning/alter_primary_key/create_table_change_pk (4.71s)
=== RUN   TestLogic/fakedist-spec-planning/alter_primary_key
=== PAUSE TestLogic/fakedist-spec-planning/alter_primary_key
=== CONT  TestLogic/fakedist-spec-planning/alter_primary_key
--- progress: testdata/logic_test/alter_type: 86 statements/queries
        --- FAIL: TestLogic/fakedist-spec-planning/alter_primary_key (18.49s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.48s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

andreimatei commented 3 years ago

seems to keep happening, and also on release-21.1 https://github.com/cockroachdb/cockroach/issues/61657

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@dd87d114508b8b34230ae7da560ab053dd4f58ce:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic927282679
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3341: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic927282679
--- FAIL: TestLogic (754.47s)
=== RUN   TestLogic/5node/distsql_datetime
=== PAUSE TestLogic/5node/distsql_datetime
=== CONT  TestLogic/5node/distsql_datetime
=== CONT  TestLogic/5node/distsql_datetime
    logic.go:2557: 

        testdata/logic_test/distsql_datetime:3: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:44901: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/distsql_datetime with config 5node: 0 tests, 1 failures
E210309 21:22:34.947860 11118492 sql/logictest/logic.go:3573  [-] 1209  pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:44901: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3116: 
        testdata/logic_test/distsql_datetime:3: error while processing
    logic.go:3116: pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:44901: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/5node/distsql_datetime (2.54s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@3bae381e45397225ff926f3f8f60d52b70fdc661:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic961305549
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3341: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic961305549
--- FAIL: TestLogic (1554.07s)
=== RUN   TestLogic/local-vec-off/alter_primary_key/create_table_change_pk
E210310 09:07:38.128535 1719914 jobs/adopt.go:260  [n1] 190  job 639948744322187265: adoption completed with error could not parse "hello" as type int: strconv.ParseInt: parsing "hello": invalid syntax
E210310 09:07:38.542246 1721757 jobs/adopt.go:260  [n1] 191  job 639948745763160065: adoption completed with error could not parse "a\\01" as type bytes: bytea encoded value ends with incomplete escape sequence
--- progress: testdata/logic_test/alter_column_type: 115 statements/queries
E210310 09:07:38.790098 1722614 jobs/adopt.go:260  [n1] 192  job 639948746472423425: adoption completed with error integer out of range for type int2
--- done: testdata/logic_test/alter_column_type with config local-vec-off: 122 tests, 0 failures
--- total progress: 29593 statements/queries
=== CONT  TestLogic/local-vec-off/alter_primary_key/create_table_change_pk
    logic.go:2557: 

        testdata/logic_test/alter_primary_key:1332: 
        expected success, but found
        (42703) column "crdb_internal_id_shard_10" does not exist
        column_resolver.go:206: in NewUndefinedColumnError()
E210310 09:07:40.213479 1719775 sql/logictest/logic.go:3548  [-] 196  
E210310 09:07:40.213479 1719775 sql/logictest/logic.go:3548  [-] 196 +pq: column "crdb_internal_id_shard_10" does not exist
    logic.go:1834: 
         pq: column "crdb_internal_id_shard_10" does not exist
--- done: testdata/logic_test/alter_primary_key with config local-vec-off: 227 tests, 2 failures
            --- FAIL: TestLogic/local-vec-off/alter_primary_key/create_table_change_pk (2.22s)
=== RUN   TestLogic/local-vec-off/alter_primary_key
=== PAUSE TestLogic/local-vec-off/alter_primary_key
=== CONT  TestLogic/local-vec-off/alter_primary_key
        --- FAIL: TestLogic/local-vec-off/alter_primary_key (8.17s)
=== RUN   TestLogic/local-vec-off
    --- FAIL: TestLogic/local-vec-off (0.33s)
More

Parameters: - TAGS= - GOFLAGS=-parallel=2 ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

adityamaru commented 3 years ago

@jordanlewis test-infra is seeing this and #61657 flake often on master and 21.1. Just wondering if someone is actively fixing this, or you recommend skipping some of these tests?

jordanlewis commented 3 years ago

@ajwerner please advise. I think we could skip the create_table_change_pk test for now. I think we need to get to the bottom of this - it feels pretty new.

ajwerner commented 3 years ago

Ack, turning my attention to it now. My best guess is it is https://github.com/cockroachdb/cockroach/pull/61345 as that's the only real thing that's touched this code that I can think of.

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@cd96b8663d48c02633a4e43dda83e6dbf41362d3:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic765069083
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3341: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic765069083
--- FAIL: TestLogic (971.35s)
=== RUN   TestLogic/5node-metadata
    --- FAIL: TestLogic/5node-metadata (0.00s)
=== RUN   TestLogic/5node-metadata/distsql_datetime
=== PAUSE TestLogic/5node-metadata/distsql_datetime
=== CONT  TestLogic/5node-metadata/distsql_datetime
=== CONT  TestLogic/5node-metadata/distsql_datetime
    logic.go:2557: 

        testdata/logic_test/distsql_datetime:3: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n5,s5}]: failed to connect to n5 at 127.0.0.1:39631: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/distsql_datetime with config 5node-metadata: 0 tests, 1 failures
E210310 21:48:59.038509 12911254 sql/logictest/logic.go:3573  [-] 1267  pq: while carrying out changes [{ADD_VOTER n5,s5}]: failed to connect to n5 at 127.0.0.1:39631: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3116: 
        testdata/logic_test/distsql_datetime:3: error while processing
    logic.go:3116: pq: while carrying out changes [{ADD_VOTER n5,s5}]: failed to connect to n5 at 127.0.0.1:39631: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- total progress: 137174 statements/queries
        --- FAIL: TestLogic/5node-metadata/distsql_datetime (2.03s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@e7f33b7ef323bd29ba203c362408776bd77eac52:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic451763382
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3341: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic451763382
--- FAIL: TestLogic (1085.75s)
=== RUN   TestLogic/5node-disk
    --- FAIL: TestLogic/5node-disk (0.00s)
=== RUN   TestLogic/5node-disk/distsql_merge_join
=== PAUSE TestLogic/5node-disk/distsql_merge_join
=== CONT  TestLogic/5node-disk/distsql_merge_join
=== CONT  TestLogic/5node-disk/distsql_merge_join
    testcluster.go:1442: failed to create a status client because of initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/distsql_subquery with config 5node-disk: 7 tests, 0 failures
--- done: testdata/logic_test/contention_event with config 5node-disk: 11 tests, 0 failures
--- total progress: 137658 statements/queries
        --- FAIL: TestLogic/5node-disk/distsql_merge_join (2.06s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@e7f33b7ef323bd29ba203c362408776bd77eac52:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic161582333
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3341: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic161582333
--- FAIL: TestLogic (800.95s)
=== RUN   TestLogic/5node-spec-planning/distsql_union
=== PAUSE TestLogic/5node-spec-planning/distsql_union
=== CONT  TestLogic/5node-spec-planning/distsql_union
=== CONT  TestLogic/5node-spec-planning/distsql_union
    logic.go:2557: 

        testdata/logic_test/distsql_union:22: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n5,s5}]: failed to connect to n5 at 127.0.0.1:35645: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/distsql_union with config 5node-spec-planning: 3 tests, 1 failures
E210312 17:31:56.937224 11643012 sql/logictest/logic.go:3573  [-] 1173  pq: while carrying out changes [{ADD_VOTER n5,s5}]: failed to connect to n5 at 127.0.0.1:35645: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3116: 
        testdata/logic_test/distsql_union:22: error while processing
    logic.go:3116: pq: while carrying out changes [{ADD_VOTER n5,s5}]: failed to connect to n5 at 127.0.0.1:35645: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- total progress: 138082 statements/queries
--- done: testdata/logic_test/inverted_join_geospatial_dist with config 5node-spec-planning: 29 tests, 0 failures
--- done: testdata/logic_test/merge_join_dist with config 5node-spec-planning: 12 tests, 0 failures
--- done: testdata/logic_test/distsql_datetime with config 5node-spec-planning: 2 tests, 0 failures
--- done: testdata/logic_test/inverted_join_multi_column_dist with config 5node-spec-planning: 16 tests, 0 failures
--- done: testdata/logic_test/distsql_numtables with config 5node-spec-planning: 27 tests, 0 failures
E210312 17:31:57.863771 11676958 sql/stats/automatic_stats.go:302  [-] 1174  failed to refresh stats: node unavailable; try another peer
--- done: testdata/logic_test/distsql_tighten_spans with config 5node-spec-planning: 40 tests, 0 failures
E210312 17:31:58.042815 11740492 jobs/adopt.go:260  [n1] 1175  job 640614131390775297: adoption completed with error job 640614131390775297: could not mark as reverting: load-job-query: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer: log-job: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer
--- done: testdata/logic_test/inverted_join_json_array_dist with config 5node-spec-planning: 26 tests, 0 failures
--- done: testdata/logic_test/distsql_distinct_on with config 5node-spec-planning: 15 tests, 0 failures
E210312 17:31:58.431058 11745362 jobs/adopt.go:260  [n1] 1176  job 640614132657651713: adoption completed with error job-update: collecting referenced descriptors: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer
--- progress: testdata/logic_test/distsql_agg: 19 statements/queries
--- progress: testdata/logic_test/hash_join_dist: 14 statements/queries
--- progress: testdata/logic_test/distsql_merge_join: 27 statements/queries
--- done: testdata/logic_test/hash_join_dist with config 5node-spec-planning: 18 tests, 0 failures
E210312 17:31:59.608843 11756550 kv/kvclient/kvcoord/txn_interceptor_committer.go:452  [-] 1177  making txn commit explicit failed for "unnamed" meta={id=340f3101 pri=0.03129016 epo=0 ts=1615570319.602523681,0 min=1615570319.602523681,0 seq=2} lock=true stat=STAGING rts=1615570319.602523681,0 wto=false gul=1615570320.102523681,0 int=1 ifw=1: result is ambiguous (server shutdown)
--- total progress: 138278 statements/queries
        --- FAIL: TestLogic/5node-spec-planning/distsql_union (2.20s)
=== RUN   TestLogic/5node-spec-planning
    --- FAIL: TestLogic/5node-spec-planning (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

sumeerbhola commented 3 years ago

There have been multiple recent failures due to "failed to create a status client because of initial connection heartbeat failed". Any ideas on how to improve this?

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@283a21e3e84037b03ea73ca7d8cba6a9b6c11f62:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic047338459
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3341: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic047338459
--- FAIL: TestLogic (972.29s)
=== RUN   TestLogic/5node-spec-planning/inverted_join_multi_column_dist
=== PAUSE TestLogic/5node-spec-planning/inverted_join_multi_column_dist
=== CONT  TestLogic/5node-spec-planning/inverted_join_multi_column_dist
=== CONT  TestLogic/5node-spec-planning/inverted_join_multi_column_dist
    logic.go:2557: 

        testdata/logic_test/inverted_join_multi_column_dist:76: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:45933: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/inverted_join_multi_column_dist with config 5node-spec-planning: 5 tests, 1 failures
E210313 02:41:10.689646 13253608 sql/logictest/logic.go:3573  [-] 1274  pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:45933: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3116: 
        testdata/logic_test/inverted_join_multi_column_dist:76: error while processing
    logic.go:3116: pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:45933: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/distsql_subquery with config 5node-spec-planning: 7 tests, 0 failures
--- done: testdata/logic_test/contention_event with config 5node-spec-planning: 12 tests, 0 failures
--- done: testdata/logic_test/inverted_join_geospatial_dist with config 5node-spec-planning: 29 tests, 0 failures
        --- FAIL: TestLogic/5node-spec-planning/inverted_join_multi_column_dist (2.22s)
=== RUN   TestLogic/5node-spec-planning
    --- FAIL: TestLogic/5node-spec-planning (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@283a21e3e84037b03ea73ca7d8cba6a9b6c11f62:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic296668288
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3341: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic296668288
--- FAIL: TestLogic (957.38s)
=== RUN   TestLogic/fakedist-spec-planning/inflight_trace_spans
=== PAUSE TestLogic/fakedist-spec-planning/inflight_trace_spans
=== CONT  TestLogic/fakedist-spec-planning/inflight_trace_spans
=== CONT  TestLogic/fakedist-spec-planning/inflight_trace_spans
    logic.go:2331: let $curr_trace_id = 1967622793753616883
=== CONT  TestLogic/fakedist-spec-planning/inflight_trace_spans
    logic.go:2290: 

        testdata/logic_test/inflight_trace_spans:44: SELECT count(*) = 0
          FROM current_trace_spans
        expected:
            true
        but found (query options: "") :
            false
--- done: testdata/logic_test/inflight_trace_spans with config fakedist-spec-planning: 9 tests, 1 failures
        --- FAIL: TestLogic/fakedist-spec-planning/inflight_trace_spans (0.98s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.68s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@7899744d137334e8337e3bc3122f816c141ed7b5:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic479913299
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic479913299
--- FAIL: TestLogic (1016.61s)
=== RUN   TestLogic/5node/merge_join_dist_vec
=== PAUSE TestLogic/5node/merge_join_dist_vec
=== CONT  TestLogic/5node/merge_join_dist_vec
=== CONT  TestLogic/5node/merge_join_dist_vec
    logic.go:2549: 

        testdata/logic_test/merge_join_dist_vec:23: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:43635: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/merge_join_dist_vec with config 5node: 6 tests, 1 failures
E210322 22:49:09.664515 12843841 sql/logictest/logic.go:3565  [-] 1334  pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:43635: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
    logic.go:3108: 
        testdata/logic_test/merge_join_dist_vec:23: error while processing
    logic.go:3108: pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:43635: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        --- FAIL: TestLogic/5node/merge_join_dist_vec (2.08s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@7826ab1dabcfaa2bad8e84025351387e17ec0e5d:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic195117160
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic195117160
--- FAIL: TestLogic (1054.53s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.58s)
=== RUN   TestLogic/5node-spec-planning/merge_join_dist
=== PAUSE TestLogic/5node-spec-planning/merge_join_dist
=== CONT  TestLogic/5node-spec-planning/merge_join_dist
=== CONT  TestLogic/5node-spec-planning/merge_join_dist
    logic.go:2549: 

        testdata/logic_test/merge_join_dist:23: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:36979: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/merge_join_dist with config 5node-spec-planning: 6 tests, 1 failures
E210323 02:19:41.121166 13370297 sql/logictest/logic.go:3565  [-] 1329  pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:36979: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3108: 
        testdata/logic_test/merge_join_dist:23: error while processing
    logic.go:3108: pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:36979: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/contention_event with config 5node-spec-planning: 12 tests, 0 failures
--- total progress: 139486 statements/queries
        --- FAIL: TestLogic/5node-spec-planning/merge_join_dist (1.92s)
=== RUN   TestLogic/5node-spec-planning
    --- FAIL: TestLogic/5node-spec-planning (0.00s)
=== RUN   TestLogic/fakedist-spec-planning/collatedstring_uniqueindex2
=== PAUSE TestLogic/fakedist-spec-planning/collatedstring_uniqueindex2
=== CONT  TestLogic/fakedist-spec-planning/collatedstring_uniqueindex2
=== CONT  TestLogic/fakedist-spec-planning/collatedstring_uniqueindex2
    logic.go:2549: 

        testdata/logic_test/collatedstring_uniqueindex2:29: SELECT a, b FROM t ORDER BY a, b
        expected success, but found
        (XXUUU) failed to connect to n3 at 127.0.0.1:38647: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210323 02:18:07.492930 11089452 sql/logictest/logic.go:3540  [-] 1198  
E210323 02:18:07.492930 11089452 sql/logictest/logic.go:3540  [-] 1198 +pq: failed to connect to n3 at 127.0.0.1:38647: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2282: 
         pq: failed to connect to n3 at 127.0.0.1:38647: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/collatedstring_uniqueindex2 with config fakedist-spec-planning: 4 tests, 2 failures
E210323 02:18:07.493435 11089452 sql/logictest/logic.go:3565  [-] 1199  testdata/logic_test/collatedstring_uniqueindex2:42: too many errors encountered, skipping the rest of the input
    logic.go:3108: 
        testdata/logic_test/collatedstring_uniqueindex2:42: error while processing
    logic.go:3108: testdata/logic_test/collatedstring_uniqueindex2:42: too many errors encountered, skipping the rest of the input
        --- FAIL: TestLogic/fakedist-spec-planning/collatedstring_uniqueindex2 (1.50s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@ab7362161d5e3b88f4730db94b95c5926b5eb3af:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic726572008
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic726572008
--- FAIL: TestLogic (802.79s)
=== RUN   TestLogic/5node/hash_join_dist
=== PAUSE TestLogic/5node/hash_join_dist
=== CONT  TestLogic/5node/hash_join_dist
E210323 22:21:40.427406 11109734 sql/stats/automatic_stats.go:302  [-] 1235  failed to refresh stats: node unavailable; try another peer
E210323 22:21:40.427758 11140263 sql/stats/automatic_stats.go:302  [-] 1236  failed to refresh stats: node unavailable; try another peer
=== CONT  TestLogic/5node/hash_join_dist
    logic.go:2549: 

        testdata/logic_test/hash_join_dist:9: 
        expected success, but found
        (XXUUU) cannot up-replicate to s3; missing gossiped StoreDescriptor
--- done: testdata/logic_test/hash_join_dist with config 5node: 2 tests, 1 failures
E210323 22:21:42.141639 11100659 sql/logictest/logic.go:3565  [-] 1241  pq: cannot up-replicate to s3; missing gossiped StoreDescriptor
    logic.go:3108: 
        testdata/logic_test/hash_join_dist:9: error while processing
    logic.go:3108: pq: cannot up-replicate to s3; missing gossiped StoreDescriptor
--- progress: testdata/logic_test/ranges: 20 statements/queries
--- done: testdata/logic_test/merge_join_dist with config 5node: 12 tests, 0 failures
--- done: testdata/logic_test/inverted_filter_geospatial_dist with config 5node: 25 tests, 0 failures
        --- FAIL: TestLogic/5node/hash_join_dist (1.77s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@9fa4b125bfb07552b43ba4fd52c9301afd7a937b:

        expected success, but found
        (XX000) internal error: unexpected error from the vectorized engine: runtime error: index out of range [1] with length 1
        error.go:88: in func1()
        DETAIL: stack trace:
        github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:88: func1()
        runtime/panic.go:969: gopanic()
        runtime/panic.go:88: goPanicIndex()
        github.com/cockroachdb/cockroach/pkg/col/coldata/nulls.go:181: NullAt()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/crossjoiner.eg.go:740: func1()
        github.com/cockroachdb/cockroach/pkg/sql/colmem/allocator.go:297: PerformOperation()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/crossjoiner.eg.go:51: buildFromLeftInput()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin/crossjoiner.go:110: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:145: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/const.eg.go:212: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:78: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:290: next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:315: nextAdapter()
        github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91: CatchVectorizedRuntimeError()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:321: Next()
        github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:171: Run()
        github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:774: Run()
        github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:382: Run()
        github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:359: Run()
        github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:968: PlanAndRun()
        github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1040: execWithDistSQLEngine()
        github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:912: dispatchToExecutionEngine()
        github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:656: execStmtInOpenState()
        github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:120: execStmt()
        github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1515: func1()
        github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1517: execCmd()
        github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1438: run()
        github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:482: ServeConn()
        github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:626: func1()
        runtime/asm_amd64.s:1374: goexit()

        NOTE: internal errors may have more details in logs. Use -show-logs.
E210324 07:58:01.762364 8097184 sql/logictest/logic.go:3540  [-] 931  
E210324 07:58:01.762364 8097184 sql/logictest/logic.go:3540  [-] 931 +pq: internal error: unexpected error from the vectorized engine: runtime error: index out of range [1] with length 1
    logic.go:2282: 
         pq: internal error: unexpected error from the vectorized engine: runtime error: index out of range [1] with length 1
--- done: testdata/logic_test/postgresjoin with config fakedist-disk: 31 tests, 2 failures
E210324 07:58:01.762578 8097184 sql/logictest/logic.go:3565  [-] 932  testdata/logic_test/postgresjoin:454: too many errors encountered, skipping the rest of the input
    logic.go:3108: 
        testdata/logic_test/postgresjoin:454: error while processing
    logic.go:3108: testdata/logic_test/postgresjoin:454: too many errors encountered, skipping the rest of the input
E210324 07:58:01.768962 9170999 sql/stats/automatic_stats.go:302  [-] 933  failed to refresh stats: node unavailable; try another peer
        --- FAIL: TestLogic/fakedist-disk/postgresjoin (0.51s)
=== RUN   TestLogic/fakedist-disk
    --- FAIL: TestLogic/fakedist-disk (0.59s)
More

Parameters: - TAGS= - GOFLAGS=-parallel=2 ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@f84a9da9e62bad67c6a72d6c0717cd87e2372357:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic698830742
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic698830742
--- FAIL: TestLogic (957.45s)
=== RUN   TestLogic/fakedist-spec-planning/geospatial_zm
=== PAUSE TestLogic/fakedist-spec-planning/geospatial_zm
=== CONT  TestLogic/fakedist-spec-planning/geospatial_zm
=== CONT  TestLogic/fakedist-spec-planning/geospatial_zm
    testcluster.go:1442: failed to create a status client because of initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/fakedist-spec-planning/geospatial_zm (1.49s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.51s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

RaduBerinde commented 3 years ago

CC @yuzefovich for new "unexpected error from the vectorized engine: runtime error: index out of range" error. I just hit it in a CI.

yuzefovich commented 3 years ago

Thanks, I'll take a look

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@aaac3aeae88c5864643881c7d74e24074ffb2c86:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic459801672
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic459801672
--- FAIL: TestLogic (1002.68s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.66s)
=== RUN   TestLogic/fakedist-spec-planning/pg_extension
=== PAUSE TestLogic/fakedist-spec-planning/pg_extension
=== CONT  TestLogic/fakedist-spec-planning/pg_extension
--- done: testdata/logic_test/postgresjoin with config fakedist-spec-planning: 101 tests, 0 failures
=== CONT  TestLogic/fakedist-spec-planning/pg_extension
    logic.go:3112: 
        testdata/logic_test/pg_extension:55: error while processing
    logic.go:3112: test was successful but validation upon completion failed: failed to test for descriptor JSON round-trip: pq: failed to connect to n3 at 127.0.0.1:40399: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
E210325 19:15:05.441231 11861640 sql/stats/automatic_stats.go:302  [-] 1202  failed to refresh stats: node unavailable; try another peer
        --- FAIL: TestLogic/fakedist-spec-planning/pg_extension (1.53s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@1c9289e0589bf46aa5205290036b672d9afc525e:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic134629110
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic134629110
--- FAIL: TestLogic (1020.11s)
=== RUN   TestLogic/5node-spec-planning
    --- FAIL: TestLogic/5node-spec-planning (0.00s)
=== RUN   TestLogic/5node-spec-planning/inverted_join_json_array_dist
=== PAUSE TestLogic/5node-spec-planning/inverted_join_json_array_dist
=== CONT  TestLogic/5node-spec-planning/inverted_join_json_array_dist
--- done: testdata/logic_test/distsql_subquery with config 5node-spec-planning: 7 tests, 0 failures
E210326 02:44:36.453923 13252575 sql/logictest/logic.go:3553  [-] 1321  
E210326 02:44:36.453923 13252575 sql/logictest/logic.go:3553  [-] 1321 +testdata/logic_test/inverted_join_json_array_dist:60: 
E210326 02:44:36.453923 13252575 sql/logictest/logic.go:3553  [-] 1321 +expected success, but found
E210326 02:44:36.453923 13252575 sql/logictest/logic.go:3553  [-] 1321 +(XXUUU) while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:39745: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
E210326 02:44:36.453923 13252575 sql/logictest/logic.go:3553  [-] 1321 +nodedialer.go:169: in dial()
    logic.go:2549: 

        testdata/logic_test/inverted_join_json_array_dist:60: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:39745: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/inverted_join_json_array_dist with config 5node-spec-planning: 6 tests, 1 failures
E210326 02:44:36.454376 13252575 sql/logictest/logic.go:3565  [-] 1322  pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:39745: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3108: 
        testdata/logic_test/inverted_join_json_array_dist:60: error while processing
    logic.go:3108: pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:39745: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/inverted_join_geospatial_dist with config 5node-spec-planning: 29 tests, 0 failures
--- done: testdata/logic_test/merge_join_dist with config 5node-spec-planning: 12 tests, 0 failures
--- total progress: 139643 statements/queries
--- done: testdata/logic_test/contention_event with config 5node-spec-planning: 12 tests, 0 failures
E210326 02:44:36.612539 13260411 sql/stats/automatic_stats.go:302  [-] 1323  failed to refresh stats: node unavailable; try another peer
        --- FAIL: TestLogic/5node-spec-planning/inverted_join_json_array_dist (2.13s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

yuzefovich commented 3 years ago

Quick update that I can reproduce the problem with the vectorized cross joiner and filed #62636 to track it separately.

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@4f0629742ff665f8f6a60be60a5ecc5e4c248085:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic493037186
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic493037186
--- FAIL: TestLogic (1044.14s)
=== RUN   TestLogic/local/virtual_columns
=== PAUSE TestLogic/local/virtual_columns
=== CONT  TestLogic/local/virtual_columns
=== CONT  TestLogic/local/virtual_columns
    testcluster.go:1442: failed to create a status client because of initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        --- FAIL: TestLogic/local/virtual_columns (2.05s)
=== RUN   TestLogic/local
    --- FAIL: TestLogic/local (0.72s)
=== RUN   TestLogic/fakedist-spec-planning/values
=== PAUSE TestLogic/fakedist-spec-planning/values
=== CONT  TestLogic/fakedist-spec-planning/values
=== CONT  TestLogic/fakedist-spec-planning/values
    logic.go:3112: 
        testdata/logic_test/values:60: error while processing
    logic.go:3112: test was successful but validation upon completion failed: failed to test for descriptor JSON round-trip: pq: failed to connect to n2 at 127.0.0.1:40969: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- progress: testdata/logic_test/alter_primary_key: 46 statements/queries
        --- FAIL: TestLogic/fakedist-spec-planning/values (1.37s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.52s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@fe14895b103aae7f969ebf90763802629ad209c5:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic204041013
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic204041013
--- FAIL: TestLogic (1159.87s)
=== RUN   TestLogic/5node/merge_join_dist
=== PAUSE TestLogic/5node/merge_join_dist
=== CONT  TestLogic/5node/merge_join_dist
--- done: testdata/logic_test/distsql_datetime with config 5node: 2 tests, 0 failures
E210331 06:10:42.002065 13234838 jobs/adopt.go:260  [n1] 1340  job 645859383651368961: adoption completed with error job-update: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer
=== CONT  TestLogic/5node/merge_join_dist
    logic.go:2549: 

        testdata/logic_test/merge_join_dist:23: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:41973: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/merge_join_dist with config 5node: 6 tests, 1 failures
E210331 06:10:44.563527 13138693 sql/logictest/logic.go:3565  [-] 1347  pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:41973: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
    logic.go:3108: 
        testdata/logic_test/merge_join_dist:23: error while processing
    logic.go:3108: pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:41973: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
--- done: testdata/logic_test/explain_analyze_plans with config 5node: 20 tests, 0 failures
E210331 06:10:44.864557 13319459 jobs/adopt.go:260  [n1] 1348  job 645859393018658817: adoption completed with error job 645859393018658817: could not mark as reverting: load-job-query: collecting referenced descriptors: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer: log-job: node unavailable; try another peer
--- done: testdata/logic_test/merge_join_dist_vec with config 5node: 18 tests, 0 failures
--- done: testdata/logic_test/inverted_filter_json_array_dist with config 5node: 30 tests, 0 failures
--- done: testdata/logic_test/inverted_join_multi_column_dist with config 5node: 16 tests, 0 failures
--- progress: testdata/logic_test/distsql_stats: 36 statements/queries
E210331 06:10:45.879016 13346534 kv/kvclient/kvcoord/txn_interceptor_committer.go:452  [-] 1349  making txn commit explicit failed for "unnamed" meta={id=94981804 pri=0.01052059 epo=0 ts=1617171045.866589131,0 min=1617171045.866589131,0 seq=2} lock=true stat=STAGING rts=1617171045.866589131,0 wto=false gul=1617171046.366589131,0 int=1 ifw=1: result is ambiguous (server shutdown)
--- progress: testdata/logic_test/distsql_agg: 13 statements/queries
--- done: testdata/logic_test/distsql_distinct_on with config 5node: 15 tests, 0 failures
--- progress: testdata/logic_test/ranges: 44 statements/queries
E210331 06:10:46.368722 13296163 sql/stats/automatic_stats.go:302  [-] 1350  failed to refresh stats: node unavailable; try another peer
--- progress: testdata/logic_test/distsql_merge_join: 21 statements/queries
        --- FAIL: TestLogic/5node/merge_join_dist (2.62s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

ajwerner commented 3 years ago

These flakes seem bad. https://teamcity.cockroachdb.com/viewLog.html?buildId=2840106&buildTypeId=Cockroach_UnitTests is the same pathology: failure to start a testserver.

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@0966b275e63816fa62c1c754cde87917b8703f3c:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic327959758
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic327959758
--- FAIL: TestLogic (1002.65s)
=== RUN   TestLogic/5node/distsql_distinct_on
=== PAUSE TestLogic/5node/distsql_distinct_on
=== CONT  TestLogic/5node/distsql_distinct_on
E210402 00:37:51.203104 12852922 kv/kvclient/kvcoord/txn_interceptor_committer.go:452  [-] 1291  making txn commit explicit failed for "unnamed" meta={id=2e492b4e pri=0.01943498 epo=0 ts=1617323871.197022144,0 min=1617323871.197022144,0 seq=2} lock=true stat=STAGING rts=1617323871.197022144,0 wto=false gul=1617323871.697022144,0 int=1 ifw=1: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer
E210402 00:37:51.203257 12814387 sql/stats/automatic_stats.go:302  [-] 1292  failed to refresh stats: node unavailable; try another peer
=== CONT  TestLogic/5node/distsql_distinct_on
    logic.go:2549: 

        testdata/logic_test/distsql_distinct_on:40: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:37697: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/distsql_distinct_on with config 5node: 5 tests, 1 failures
E210402 00:37:53.473743 12764848 sql/logictest/logic.go:3565  [-] 1298  pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:37697: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
    logic.go:3108: 
        testdata/logic_test/distsql_distinct_on:40: error while processing
    logic.go:3108: pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:37697: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
E210402 00:37:53.482002 12900054 sql/stats/automatic_stats.go:302  [-] 1299  failed to refresh stats: node unavailable; try another peer
--- done: testdata/logic_test/explain_analyze_plans with config 5node: 20 tests, 0 failures
E210402 00:37:53.641656 12930890 jobs/adopt.go:260  [n1] 1300  job 646360182333669377: adoption completed with error result is ambiguous (server shutdown)
--- total progress: 138557 statements/queries
--- done: testdata/logic_test/distsql_datetime with config 5node: 2 tests, 0 failures
E210402 00:37:54.128419 12943965 jobs/adopt.go:260  [n1] 1301  job 646360183931830273: adoption completed with error job 646360183931830273: could not mark as reverting: load-job-query: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer: log-job: node unavailable; try another peer
--- done: testdata/logic_test/merge_join_dist_vec with config 5node: 18 tests, 0 failures
E210402 00:37:54.452850 12955137 jobs/adopt.go:260  [n1] 1302  job 646360184983977985: adoption completed with error job 646360184983977985: could not mark as reverting: node unavailable; try another peer: log-job: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer
--- done: testdata/logic_test/inverted_join_multi_column_dist with config 5node: 16 tests, 0 failures
--- done: testdata/logic_test/dist_vectorize with config 5node: 23 tests, 0 failures
E210402 00:37:54.761377 12963032 kv/kvclient/kvcoord/txn_interceptor_committer.go:452  [-] 1303  making txn commit explicit failed for "unnamed" meta={id=5b85e50a pri=0.00187382 epo=0 ts=1617323874.751517273,0 min=1617323874.751517273,0 seq=2} lock=true stat=STAGING rts=1617323874.751517273,0 wto=false gul=1617323875.251517273,0 int=1 ifw=1: result is ambiguous (server shutdown)
E210402 00:37:54.837396 12964543 jobs/adopt.go:260  [n1] 1304  job 646360186273169409: adoption completed with error job-update: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer
--- progress: testdata/logic_test/distsql_stats: 43 statements/queries
--- progress: testdata/logic_test/distsql_numtables: 19 statements/queries
--- progress: testdata/logic_test/ranges: 60 statements/queries
        --- FAIL: TestLogic/5node/distsql_distinct_on (2.32s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@0e3006351557d1d6dc4e2179a9cca1fd702b09f6:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic075015043
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic075015043
--- FAIL: TestLogic (1069.83s)
=== RUN   TestLogic/5node-metadata/merge_join_dist
=== PAUSE TestLogic/5node-metadata/merge_join_dist
=== CONT  TestLogic/5node-metadata/merge_join_dist
=== CONT  TestLogic/5node-metadata/merge_join_dist
    logic.go:2549: 

        testdata/logic_test/merge_join_dist:23: 
        expected success, but found
        (XXUUU) cannot up-replicate to s2; missing gossiped StoreDescriptor
--- done: testdata/logic_test/merge_join_dist with config 5node-metadata: 6 tests, 1 failures
E210403 01:31:48.006562 13515067 sql/logictest/logic.go:3565  [-] 1284  pq: cannot up-replicate to s2; missing gossiped StoreDescriptor
    logic.go:3108: 
        testdata/logic_test/merge_join_dist:23: error while processing
    logic.go:3108: pq: cannot up-replicate to s2; missing gossiped StoreDescriptor
E210403 01:31:48.010720 13544797 sql/stats/automatic_stats.go:302  [-] 1285  failed to refresh stats: node unavailable; try another peer
--- done: testdata/logic_test/distsql_subquery with config 5node-metadata: 7 tests, 0 failures
        --- FAIL: TestLogic/5node-metadata/merge_join_dist (1.71s)
=== RUN   TestLogic/5node-metadata
    --- FAIL: TestLogic/5node-metadata (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@df07592e26735be7362ef2b815ee163ead00739e:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic754400025
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic754400025
--- FAIL: TestLogic (1000.24s)
=== RUN   TestLogic/5node-metadata/hash_join_dist
=== PAUSE TestLogic/5node-metadata/hash_join_dist
=== CONT  TestLogic/5node-metadata/hash_join_dist
=== CONT  TestLogic/5node-metadata/hash_join_dist
    logic.go:2549: 

        testdata/logic_test/hash_join_dist:21: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:42913: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/hash_join_dist with config 5node-metadata: 6 tests, 1 failures
E210403 06:33:10.065656 13134939 sql/logictest/logic.go:3565  [-] 1334  pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:42913: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3108: 
        testdata/logic_test/hash_join_dist:21: error while processing
    logic.go:3108: pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:42913: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/distsql_tighten_spans with config 5node-metadata: 40 tests, 0 failures
        --- FAIL: TestLogic/5node-metadata/hash_join_dist (2.22s)
=== RUN   TestLogic/5node-metadata
    --- FAIL: TestLogic/5node-metadata (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@fe1327d0c2fe35a572586a8f8ee961a2bf19ffda:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic292273562
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic292273562
--- FAIL: TestLogic (878.98s)
=== RUN   TestLogic/local/set
=== PAUSE TestLogic/local/set
=== CONT  TestLogic/local/set
E210407 18:34:37.818555 138782 jobs/adopt.go:260  [n1] 14  job 647987454466621441: adoption completed with error job 647987454466621441: could not mark as reverting: job-update: collecting referenced descriptors: failed to send RPC: sending to all replicas failed; last error: node unavailable; try another peer: log-job: node unavailable; try another peer
=== CONT  TestLogic/local/set
    logic.go:2549: 

        testdata/logic_test/set:297: SHOW statement_timeout
        expected success, but found
        (57014) query execution canceled due to statement timeout
        errors.go:218: in init()
E210407 18:34:39.605075 2706 sql/logictest/logic.go:3540  [-] 16  
E210407 18:34:39.605075 2706 sql/logictest/logic.go:3540  [-] 16 +pq: query execution canceled due to statement timeout
    logic.go:2282: 
         pq: query execution canceled due to statement timeout
--- done: testdata/logic_test/set with config local: 79 tests, 2 failures
E210407 18:34:39.605764 2706 sql/logictest/logic.go:3565  [-] 17  testdata/logic_test/set:302: too many errors encountered, skipping the rest of the input
    logic.go:3108: 
        testdata/logic_test/set:302: error while processing
    logic.go:3108: testdata/logic_test/set:302: too many errors encountered, skipping the rest of the input
--- progress: testdata/logic_test/cascade: 84 statements/queries
        --- FAIL: TestLogic/local/set (2.02s)
=== RUN   TestLogic/local
    --- FAIL: TestLogic/local (1.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@f2eb62496fae3c6a72ad071b14e18a268af30ba5:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic375382853
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic375382853
--- FAIL: TestLogic (1072.86s)
=== RUN   TestLogic/fakedist-spec-planning/partial_index
=== PAUSE TestLogic/fakedist-spec-planning/partial_index
=== CONT  TestLogic/fakedist-spec-planning/partial_index
=== CONT  TestLogic/fakedist-spec-planning/partial_index
    testcluster.go:1468: failed to create a status client because of initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        --- FAIL: TestLogic/fakedist-spec-planning/partial_index (1.56s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.66s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #61657 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-21.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-21.1) - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ e1e10f99362e67771e00346fb0e006627eb67b09:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic891717717
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic891717717
--- FAIL: TestLogic (1059.19s)
=== RUN   TestLogic/fakedist-spec-planning/txn
=== PAUSE TestLogic/fakedist-spec-planning/txn
=== CONT  TestLogic/fakedist-spec-planning/txn
=== CONT  TestLogic/fakedist-spec-planning/txn
    logic.go:2549: 

        testdata/logic_test/txn:14: SELECT * FROM kv
        expected success, but found
        (XXUUU) failed to connect to n2 at 127.0.0.1:44407: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210409 05:33:27.769439 11155149 sql/logictest/logic.go:3540  [-] 1200  
E210409 05:33:27.769439 11155149 sql/logictest/logic.go:3540  [-] 1200 +pq: failed to connect to n2 at 127.0.0.1:44407: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2282: 
         pq: failed to connect to n2 at 127.0.0.1:44407: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/txn with config fakedist-spec-planning: 4 tests, 2 failures
E210409 05:33:27.769918 11155149 sql/logictest/logic.go:3565  [-] 1201  testdata/logic_test/txn:19: too many errors encountered, skipping the rest of the input
    logic.go:3108: 
        testdata/logic_test/txn:19: error while processing
    logic.go:3108: testdata/logic_test/txn:19: too many errors encountered, skipping the rest of the input
        --- FAIL: TestLogic/fakedist-spec-planning/txn (1.41s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.55s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

See this test on roachdash

Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 47e818cf8af59dcf886a987bc18307a3457f48a6:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic620710975
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic620710975
--- FAIL: TestLogic (858.10s)
=== RUN   TestLogic/fakedist-disk/inflight_trace_spans
=== PAUSE TestLogic/fakedist-disk/inflight_trace_spans
=== CONT  TestLogic/fakedist-disk/inflight_trace_spans
=== CONT  TestLogic/fakedist-disk/inflight_trace_spans
    logic.go:2323: let $curr_trace_id = 4911345170376068134
E210413 13:41:26.202572 8260816 sql/logictest/logic.go:3540  [-] 1039  
E210413 13:41:26.202572 8260816 sql/logictest/logic.go:3540  [-] 1039 +
E210413 13:41:26.202572 8260816 sql/logictest/logic.go:3540  [-] 1039 +testdata/logic_test/inflight_trace_spans:44: SELECT count(*) = 0
E210413 13:41:26.202572 8260816 sql/logictest/logic.go:3540  [-] 1039 +  FROM current_trace_spans
E210413 13:41:26.202572 8260816 sql/logictest/logic.go:3540  [-] 1039 +expected:
E210413 13:41:26.202572 8260816 sql/logictest/logic.go:3540  [-] 1039 +    true
E210413 13:41:26.202572 8260816 sql/logictest/logic.go:3540  [-] 1039 +but found (query options: "") :
E210413 13:41:26.202572 8260816 sql/logictest/logic.go:3540  [-] 1039 +    false
    logic.go:2282: 

        testdata/logic_test/inflight_trace_spans:44: SELECT count(*) = 0
          FROM current_trace_spans
        expected:
            true
        but found (query options: "") :
            false
--- done: testdata/logic_test/inflight_trace_spans with config fakedist-disk: 9 tests, 1 failures
        --- FAIL: TestLogic/fakedist-disk/inflight_trace_spans (1.41s)
=== RUN   TestLogic/fakedist-disk
    --- FAIL: TestLogic/fakedist-disk (0.71s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 1daf2df61aabf3e8d00d736fa4a843d05b596544:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic352869799
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3333: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic352869799
--- FAIL: TestLogic (781.29s)
=== RUN   TestLogic/fakedist-spec-planning/default
=== PAUSE TestLogic/fakedist-spec-planning/default
=== CONT  TestLogic/fakedist-spec-planning/default
=== CONT  TestLogic/fakedist-spec-planning/default
    testcluster.go:1468: failed to create a status client because of initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/fakedist-spec-planning/default (1.48s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.49s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ ea88b676d0c49b77375e32a290bd94753f93af48:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic623977100
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3312: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic623977100
--- FAIL: TestLogic (834.88s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
=== RUN   TestLogic/5node/experimental_distsql_planning_5node
=== PAUSE TestLogic/5node/experimental_distsql_planning_5node
=== CONT  TestLogic/5node/experimental_distsql_planning_5node
=== CONT  TestLogic/5node/experimental_distsql_planning_5node
    testcluster.go:1468: failed to create a status client because of initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/5node/experimental_distsql_planning_5node (2.26s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 4d987b3f0788c98fa358715b336a1c55e254ee50:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic879897068
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3312: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic879897068
--- FAIL: TestLogic (804.70s)
=== RUN   TestLogic/5node/merge_join_dist_vec
=== PAUSE TestLogic/5node/merge_join_dist_vec
=== CONT  TestLogic/5node/merge_join_dist_vec
=== CONT  TestLogic/5node/merge_join_dist_vec
    logic.go:2528: 

        testdata/logic_test/merge_join_dist_vec:23: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:39429: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/merge_join_dist_vec with config 5node: 6 tests, 1 failures
E210414 23:31:38.614575 11381253 sql/logictest/logic.go:3544  [-] 1242  pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:39429: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
    logic.go:3087: 
        testdata/logic_test/merge_join_dist_vec:23: error while processing
    logic.go:3087: pq: while carrying out changes [{ADD_VOTER n2,s2}]: failed to connect to n2 at 127.0.0.1:39429: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        --- FAIL: TestLogic/5node/merge_join_dist_vec (2.25s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 660df848fcd8569392791f35b27ef5e849b35e2c:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic182394997
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3312: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic182394997
--- FAIL: TestLogic (761.93s)
=== RUN   TestLogic/fakedist-spec-planning/aggregate
=== PAUSE TestLogic/fakedist-spec-planning/aggregate
=== CONT  TestLogic/fakedist-spec-planning/aggregate
=== CONT  TestLogic/fakedist-spec-planning/aggregate
    testcluster.go:296: listen tcp 127.0.0.1:0: listen: address already in use
        --- FAIL: TestLogic/fakedist-spec-planning/aggregate (0.24s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.48s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!