search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.85k stars 3.77k forks source link

sql/logictest: TestLogic failed #60824

Closed cockroach-teamcity closed 3 years ago

cockroach-teamcity commented 3 years ago

(sql/logictest).TestLogic failed on master@557346d5fc755db5f64d3e198e47be1f53a4d390:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic977178260
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3343: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic977178260
--- FAIL: TestLogic (958.82s)
=== RUN   TestLogic/5node-metadata/distsql_numtables
=== PAUSE TestLogic/5node-metadata/distsql_numtables
=== CONT  TestLogic/5node-metadata/distsql_numtables
=== CONT  TestLogic/5node-metadata/distsql_numtables
    logic.go:2559: 

        testdata/logic_test/distsql_numtables:21: 
        expected success, but found
        (XXUUU) cannot up-replicate to s2; missing gossiped StoreDescriptor
--- done: testdata/logic_test/distsql_numtables with config 5node-metadata: 4 tests, 1 failures
E210220 00:01:04.366080 12116468 sql/logictest/logic.go:3575  [-] 1008  pq: cannot up-replicate to s2; missing gossiped StoreDescriptor
    logic.go:3118: 
        testdata/logic_test/distsql_numtables:21: error while processing
    logic.go:3118: pq: cannot up-replicate to s2; missing gossiped StoreDescriptor
--- done: testdata/logic_test/merge_join_dist with config 5node-metadata: 12 tests, 0 failures
        --- FAIL: TestLogic/5node-metadata/distsql_numtables (1.66s)
=== RUN   TestLogic/5node-metadata
    --- FAIL: TestLogic/5node-metadata (0.00s)
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Related: - #58269 sql/logictest: TestLogic failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-20.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-20.1) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestLogic.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ bf6da8a40f6956bc88ee4a930ddc931f73dd5ce4:

=== CONT  TestLogic
    logic.go:3312: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic993746173
--- FAIL: TestLogic (1080.73s)
=== RUN   TestLogic/fakedist-spec-planning/target_names
=== PAUSE TestLogic/fakedist-spec-planning/target_names
=== CONT  TestLogic/fakedist-spec-planning/target_names
=== CONT  TestLogic/fakedist-spec-planning/target_names
    logic.go:2528: 

        testdata/logic_test/target_names:39: SELECT (pg_get_keywords()).word FROM t
        expected success, but found
        (XXUUU) failed to connect to n2 at 127.0.0.1:44747: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210419 22:25:18.149024 11276971 sql/logictest/logic.go:3519  [-] 1169  
E210419 22:25:18.149024 11276971 sql/logictest/logic.go:3519  [-] 1169 +pq: failed to connect to n2 at 127.0.0.1:44747: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2261: 
         pq: failed to connect to n2 at 127.0.0.1:44747: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/target_names with config fakedist-spec-planning: 6 tests, 2 failures
E210419 22:25:18.149481 11276971 sql/logictest/logic.go:3544  [-] 1170  testdata/logic_test/target_names:44: too many errors encountered, skipping the rest of the input
    logic.go:3087: 
        testdata/logic_test/target_names:44: error while processing
    logic.go:3087: testdata/logic_test/target_names:44: too many errors encountered, skipping the rest of the input
        --- FAIL: TestLogic/fakedist-spec-planning/target_names (1.41s)
=== RUN   TestLogic/fakedist-spec-planning/cascade/OptDriven
=== CONT  TestLogic/fakedist-spec-planning/cascade/OptDriven
    logic.go:2528: 

        testdata/logic_test/cascade:15: SELECT * FROM child
        expected success, but found
        (XXUUU) failed to connect to n3 at 127.0.0.1:46129: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210419 22:25:36.054415 12385639 sql/logictest/logic.go:3519  [-] 1198  
E210419 22:25:36.054415 12385639 sql/logictest/logic.go:3519  [-] 1198 +pq: failed to connect to n3 at 127.0.0.1:46129: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2261: 
         pq: failed to connect to n3 at 127.0.0.1:46129: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
E210419 22:25:36.054706 12385639 sql/logictest/logic.go:3519  [-] 1199  
E210419 22:25:36.054706 12385639 sql/logictest/logic.go:3519  [-] 1199 +testdata/logic_test/cascade:23: too many errors encountered, skipping the rest of the input
    logic.go:1805: 
         testdata/logic_test/cascade:23: too many errors encountered, skipping the rest of the input
--- done: testdata/logic_test/cascade with config fakedist-spec-planning: 2 tests, 3 failures
--- done: testdata/logic_test/window with config fakedist-spec-planning: 397 tests, 0 failures
--- total progress: 131332 statements/queries
            --- FAIL: TestLogic/fakedist-spec-planning/cascade/OptDriven (0.68s)
=== RUN   TestLogic/fakedist-spec-planning/cascade
=== PAUSE TestLogic/fakedist-spec-planning/cascade
=== CONT  TestLogic/fakedist-spec-planning/cascade
        --- FAIL: TestLogic/fakedist-spec-planning/cascade (1.91s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.72s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 1c103726de6994b26706244144d3d9f7dd6b1f58:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic039425781
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3312: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic039425781
--- FAIL: TestLogic (987.15s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.58s)
=== RUN   TestLogic/fakedist-spec-planning/timestamp/timestamp_precision
=== CONT  TestLogic/fakedist-spec-planning/timestamp/timestamp_precision
    logic.go:2528: 

        testdata/logic_test/timestamp:100: SELECT * FROM timestamp_test ORDER BY id ASC
        expected success, but found
        (XXUUU) failed to connect to n3 at 127.0.0.1:40241: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210419 22:59:32.858674 11450222 sql/logictest/logic.go:3519  [-] 1125  
E210419 22:59:32.858674 11450222 sql/logictest/logic.go:3519  [-] 1125 +pq: failed to connect to n3 at 127.0.0.1:40241: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2261: 
         pq: failed to connect to n3 at 127.0.0.1:40241: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
E210419 22:59:32.859058 11450222 sql/logictest/logic.go:3519  [-] 1126  
E210419 22:59:32.859058 11450222 sql/logictest/logic.go:3519  [-] 1126 +testdata/logic_test/timestamp:111: too many errors encountered, skipping the rest of the input
    logic.go:1805: 
         testdata/logic_test/timestamp:111: too many errors encountered, skipping the rest of the input
--- done: testdata/logic_test/timestamp with config fakedist-spec-planning: 19 tests, 3 failures
            --- FAIL: TestLogic/fakedist-spec-planning/timestamp/timestamp_precision (0.50s)
=== RUN   TestLogic/fakedist-spec-planning/timestamp
=== PAUSE TestLogic/fakedist-spec-planning/timestamp
=== CONT  TestLogic/fakedist-spec-planning/timestamp
        --- FAIL: TestLogic/fakedist-spec-planning/timestamp (1.81s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ bea4754bdf607ab050a72b7e2a3fe9dca0289d34:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic622773250
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3321: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic622773250
--- FAIL: TestLogic (1030.15s)
=== RUN   TestLogic/fakedist-spec-planning/computed
=== PAUSE TestLogic/fakedist-spec-planning/computed
=== CONT  TestLogic/fakedist-spec-planning/computed
--- done: testdata/logic_test/conditional with config fakedist-spec-planning: 9 tests, 0 failures
=== CONT  TestLogic/fakedist-spec-planning/computed
    logic.go:2537: 

        testdata/logic_test/computed:72: SELECT c FROM with_no_column_refs
        expected success, but found
        (XXUUU) failed to connect to n3 at 127.0.0.1:38231: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210420 12:38:46.870038 11122369 sql/logictest/logic.go:3528  [-] 1172  
E210420 12:38:46.870038 11122369 sql/logictest/logic.go:3528  [-] 1172 +pq: failed to connect to n3 at 127.0.0.1:38231: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2270: 
         pq: failed to connect to n3 at 127.0.0.1:38231: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/computed with config fakedist-spec-planning: 14 tests, 2 failures
E210420 12:38:46.870229 11122369 sql/logictest/logic.go:3553  [-] 1173  testdata/logic_test/computed:78: too many errors encountered, skipping the rest of the input
    logic.go:3096: 
        testdata/logic_test/computed:78: error while processing
    logic.go:3096: testdata/logic_test/computed:78: too many errors encountered, skipping the rest of the input
E210420 12:38:46.872919 11957392 sql/stats/automatic_stats.go:302  [-] 1174  failed to refresh stats: node unavailable; try another peer
        --- FAIL: TestLogic/fakedist-spec-planning/computed (1.60s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.66s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ b4766e0bdb491de6020d5d2acaa3d20f9c339176:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic692930075
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3350: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic692930075
--- FAIL: TestLogic (794.20s)
=== RUN   TestLogic/fakedist-vec-off/vectorize_unsupported
=== PAUSE TestLogic/fakedist-vec-off/vectorize_unsupported
=== CONT  TestLogic/fakedist-vec-off/vectorize_unsupported
--- done: testdata/logic_test/where with config fakedist-vec-off: 21 tests, 0 failures
--- done: testdata/logic_test/inverted_index with config fakedist-vec-off: 264 tests, 0 failures
=== CONT  TestLogic/fakedist-vec-off/vectorize_unsupported
    logic.go:2539: 

        testdata/logic_test/vectorize_unsupported:28: select (x/1) from t39417
        expected success, but found
        (XXUUU) failed to connect to n2 at 127.0.0.1:35759: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210422 20:37:40.337643 5095761 sql/logictest/logic.go:3557  [-] 587  
E210422 20:37:40.337643 5095761 sql/logictest/logic.go:3557  [-] 587 +pq: failed to connect to n2 at 127.0.0.1:35759: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2272: 
         pq: failed to connect to n2 at 127.0.0.1:35759: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/vectorize_unsupported with config fakedist-vec-off: 6 tests, 2 failures
E210422 20:37:40.346935 5095761 sql/logictest/logic.go:3582  [-] 588  testdata/logic_test/vectorize_unsupported:33: too many errors encountered, skipping the rest of the input
    logic.go:3125: 
        testdata/logic_test/vectorize_unsupported:33: error while processing
    logic.go:3125: testdata/logic_test/vectorize_unsupported:33: too many errors encountered, skipping the rest of the input
        --- FAIL: TestLogic/fakedist-vec-off/vectorize_unsupported (1.83s)
=== RUN   TestLogic/fakedist-vec-off
    --- FAIL: TestLogic/fakedist-vec-off (0.60s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 924935cc8ee14ff0c6d23aedd2a12673bf864808:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic599276416
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3350: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic599276416
--- FAIL: TestLogic (816.45s)
=== RUN   TestLogic/fakedist/serial/serial_rowid
=== CONT  TestLogic/fakedist/serial/serial_rowid
    logic.go:2539: 

        testdata/logic_test/serial:31: SELECT count(DISTINCT a), count(DISTINCT b), count(DISTINCT c) FROM serial
        expected success, but found
        (XXUUU) failed to connect to n2 at 127.0.0.1:35655: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
        nodedialer.go:169: in dial()
E210424 06:04:35.253726 4748781 sql/logictest/logic.go:3557  [-] 533  
E210424 06:04:35.253726 4748781 sql/logictest/logic.go:3557  [-] 533 +pq: failed to connect to n2 at 127.0.0.1:35655: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
    logic.go:2272: 
         pq: failed to connect to n2 at 127.0.0.1:35655: initial connection heartbeat failed: rpc error: code = Unavailable desc = timed out waiting for server handshake
E210424 06:04:35.254062 4748781 sql/logictest/logic.go:3557  [-] 534  
E210424 06:04:35.254062 4748781 sql/logictest/logic.go:3557  [-] 534 +testdata/logic_test/serial:36: too many errors encountered, skipping the rest of the input
    logic.go:1816: 
         testdata/logic_test/serial:36: too many errors encountered, skipping the rest of the input
--- done: testdata/logic_test/serial with config fakedist: 5 tests, 3 failures
            --- FAIL: TestLogic/fakedist/serial/serial_rowid (0.31s)
=== RUN   TestLogic/fakedist/serial
=== PAUSE TestLogic/fakedist/serial
=== CONT  TestLogic/fakedist/serial
        --- FAIL: TestLogic/fakedist/serial (1.64s)
=== RUN   TestLogic/fakedist
    --- FAIL: TestLogic/fakedist (0.58s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 05880958efb6fda9ab80ed16495d98922de3fbf6:

Fatal error:

panic: expected 2 components to be closed, but found that only 0 were

Stack: 

goroutine 9453985 [running]:
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError(...)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:192
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Cleanup(0xc146920aa0, 0x51d5e40, 0xc037fcf240)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:285 +0x505
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run.func8()
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:373 +0x82
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc0c811c600, 0x51f4140, 0xc05fff0318, 0xc0c811cae0, 0x3, 0x5243ee0, 0xc08e468c00, 0xc109496401, 0xc0b0f19078, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1055 +0x4cc
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc0c811c600, 0x51f4140, 0xc05fff0318, 0xc0c811cae0, 0x5243ee0, 0xc08e468c00, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:915 +0x75b
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc0c811c600, 0x51f4140, 0xc05fff0318, 0x5200700, 0xc02c975f90, 0xc0979d1bc0, 0x3c, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:659 +0xf86
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc0c811c600, 0x51d5e40, 0xc0ce4d52c0, 0x5200700, 0xc02c975f90, 0xc0979d1bc0, 0x3c, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:122 +0xb14
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1(0x5200700, 0xc02c975f90, 0xc0979d1bc0, 0x3c, 0x0, 0x0, 0xad5db37, 0xed8193acf, 0x0, 0xad5db37, ...)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1527 +0x225
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0c811c600, 0x51d5e40, 0xc0ce4d52c0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1529 +0x1457
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0c811c600, 0x51d5f00, 0xc08e468870, 0xc081bf7e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1450 +0x228
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1(0xc0c811c600, 0x51d5f00, 0xc08e468870, 0xc0b0f18960, 0xc08e4689c0, 0x5188200, 0xc0ce4d5140, 0xc02ad0a630, 0xc17f462d20)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:202 +0x65
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:201 +0x2f2
Log preceding fatal error

``` === RUN TestLogic/fakedist-disk/serial/serial_virtual_sequence E210426 22:50:55.106794 9444974 jobs/adopt.go:260 [n1] 954 job 653417037656260609: adoption completed with error failed to ingest index entries during backfill: duplicate key value violates unique constraint "bar" ```

Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 588d9d7ad7618784ee23a5e5f440a9d9d36bb0d5:

=== RUN   TestLogic/fakedist-disk/schema_change_in_txn/rollback_mutations
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 29305bfb08116f38e10ca090347eea170899b919:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic555034003
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic555034003
--- FAIL: TestLogic (817.89s)
=== RUN   TestLogic/fakedist/zigzag_join
=== PAUSE TestLogic/fakedist/zigzag_join
=== CONT  TestLogic/fakedist/zigzag_join
=== CONT  TestLogic/fakedist/zigzag_join
    logic.go:2540: 

        testdata/logic_test/zigzag_join:38: SELECT n,a,b FROM a WHERE a = 4 AND b = 1
        expected success, but found
        (XXUUU) failed to connect to n3 at 127.0.0.1:44899: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210427 21:45:54.071236 3454388 sql/logictest/logic.go:3558  [-] 479  
E210427 21:45:54.071236 3454388 sql/logictest/logic.go:3558  [-] 479 +pq: failed to connect to n3 at 127.0.0.1:44899: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2273: 
         pq: failed to connect to n3 at 127.0.0.1:44899: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/zigzag_join with config fakedist: 8 tests, 2 failures
E210427 21:45:54.072134 3454388 sql/logictest/logic.go:3583  [-] 480  testdata/logic_test/zigzag_join:43: too many errors encountered, skipping the rest of the input
    logic.go:3126: 
        testdata/logic_test/zigzag_join:43: error while processing
    logic.go:3126: testdata/logic_test/zigzag_join:43: too many errors encountered, skipping the rest of the input
        --- FAIL: TestLogic/fakedist/zigzag_join (1.81s)
=== RUN   TestLogic/fakedist
    --- FAIL: TestLogic/fakedist (0.75s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ f3286c1e1b8d2d44f0b5c92460c0d5acbfdd1610:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic036720330
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic036720330
--- FAIL: TestLogic (828.11s)
=== RUN   TestLogic/5node-metadata/distsql_numtables
=== PAUSE TestLogic/5node-metadata/distsql_numtables
=== CONT  TestLogic/5node-metadata/distsql_numtables
=== CONT  TestLogic/5node-metadata/distsql_numtables
    logic.go:2540: 

        testdata/logic_test/distsql_numtables:21: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n4,s4}]: failed to connect to n4 at 127.0.0.1:35873: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/distsql_numtables with config 5node-metadata: 4 tests, 1 failures
E210427 22:20:54.582791 12005955 sql/logictest/logic.go:3583  [-] 1219  pq: while carrying out changes [{ADD_VOTER n4,s4}]: failed to connect to n4 at 127.0.0.1:35873: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3126: 
        testdata/logic_test/distsql_numtables:21: error while processing
    logic.go:3126: pq: while carrying out changes [{ADD_VOTER n4,s4}]: failed to connect to n4 at 127.0.0.1:35873: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/contention_event with config 5node-metadata: 12 tests, 0 failures
--- total progress: 140737 statements/queries
        --- FAIL: TestLogic/5node-metadata/distsql_numtables (2.22s)
=== RUN   TestLogic/5node-metadata
    --- FAIL: TestLogic/5node-metadata (0.00s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 171b1bd0971171214c203ad615e40a651b71848a:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic231064638
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic231064638
--- FAIL: TestLogic (800.82s)
=== RUN   TestLogic/fakedist-disk/schema_change_in_txn
=== PAUSE TestLogic/fakedist-disk/schema_change_in_txn
=== CONT  TestLogic/fakedist-disk/schema_change_in_txn
=== CONT  TestLogic/fakedist-disk/schema_change_in_txn
    testcluster.go:135: condition failed to evaluate within 45s: unexpectedly found 1 active spans:
             0.000ms      0.000ms    === operation:colbatchscan
        goroutine 8364100 [running]:
        runtime/debug.Stack(0xc00356b860, 0x5152500, 0xc12816d700)
            /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x53233a0, 0xc008bd1080, 0xc00356b860)
            /go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:36 +0x6d
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers(0xc087166000, 0x51de520, 0xc000072128)
            /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:135 +0x23d
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start.func2()
            /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:335 +0x45
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc0792f12e0)
            /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:110 +0x25
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc06c0fb780, 0x51de520, 0xc000072128)
            /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:479 +0x22f
        github.com/cockroachdb/cockroach/pkg/sql/logictest.(*logicTest).shutdownCluster(0xc088faa800)
            /go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/logic.go:1602 +0x1da
        github.com/cockroachdb/cockroach/pkg/sql/logictest.(*logicTest).close(0xc088faa800)
            /go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/logic.go:1237 +0x3d
        github.com/cockroachdb/cockroach/pkg/sql/logictest.(*logicTest).runFile(0xc088faa800, 0xc000c8f200, 0x28, 0x450b498, 0xd, 0x3, 0x1, 0x44f2451, 0x2, 0x0, ...)
            /go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/logic.go:3132 +0x189
        github.com/cockroachdb/cockroach/pkg/sql/logictest.RunLogicTestWithDefaultConfig.func1.1(0xc008bd1080)
            /go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/logic.go:3324 +0x2de
        testing.tRunner(0xc008bd1080, 0xc01d91f440)
            /usr/local/go/src/testing/testing.go:1123 +0xef
        created by testing.(*T).Run
            /usr/local/go/src/testing/testing.go:1168 +0x2b3
--- progress: testdata/logic_test/tenant: 25 statements/queries
--- done: testdata/logic_test/tenant with config fakedist-disk: 27 tests, 0 failures
--- total progress: 122871 statements/queries
        --- FAIL: TestLogic/fakedist-disk/schema_change_in_txn (65.26s)
=== RUN   TestLogic/fakedist-disk
    --- FAIL: TestLogic/fakedist-disk (0.65s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 112d0c111c2e9bd54cd13d40ad353cc4b286016e:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic265796156
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic265796156
--- FAIL: TestLogic (1011.02s)
=== RUN   TestLogic/5node
    --- FAIL: TestLogic/5node (0.00s)
=== RUN   TestLogic/5node/distsql_distinct_on
=== PAUSE TestLogic/5node/distsql_distinct_on
=== CONT  TestLogic/5node/distsql_distinct_on
=== CONT  TestLogic/5node/distsql_distinct_on
    logic.go:2540: 

        testdata/logic_test/distsql_distinct_on:40: 
        expected success, but found
        (XXUUU) while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:36903: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
--- done: testdata/logic_test/distsql_distinct_on with config 5node: 5 tests, 1 failures
E210428 11:16:37.957233 13281632 sql/logictest/logic.go:3583  [-] 1274  pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:36903: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:3126: 
        testdata/logic_test/distsql_distinct_on:40: error while processing
    logic.go:3126: pq: while carrying out changes [{ADD_VOTER n3,s3}]: failed to connect to n3 at 127.0.0.1:36903: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        --- FAIL: TestLogic/5node/distsql_distinct_on (2.22s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 7910db27b63ad68d2b43f618976ef03da5451ed2:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic952021909
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic952021909
--- FAIL: TestLogic (931.44s)
=== RUN   TestLogic/fakedist-spec-planning/schema_change_in_txn
=== PAUSE TestLogic/fakedist-spec-planning/schema_change_in_txn
=== CONT  TestLogic/fakedist-spec-planning/schema_change_in_txn
--- progress: testdata/logic_test/join: 102 statements/queries
--- done: testdata/logic_test/select with config fakedist-spec-planning: 162 tests, 0 failures
=== CONT  TestLogic/fakedist-spec-planning/schema_change_in_txn
    testcluster.go:135: condition failed to evaluate within 45s: unexpectedly found 1 active spans:
             0.000ms      0.000ms    === operation:colbatchscan
        goroutine 10131596 [running]:
        runtime/debug.Stack(0xc07875b860, 0x5152860, 0xc3015f0fe0)
            /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x5323840, 0xc03e975c80, 0xc07875b860)
            /go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:36 +0x6d
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers(0xc26c7c2480, 0x51de8c0, 0xc00019e018)
            /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:135 +0x23d
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start.func2()
            /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:335 +0x45
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc2fdab6c50)
            /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:110 +0x25
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc0365ad280, 0x51de8c0, 0xc00019e018)
            /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:479 +0x22f
        github.com/cockroachdb/cockroach/pkg/sql/logictest.(*logicTest).shutdownCluster(0xc239594a00)
            /go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/logic.go:1602 +0x1da
        github.com/cockroachdb/cockroach/pkg/sql/logictest.(*logicTest).close(0xc239594a00)
            /go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/logic.go:1237 +0x3d
        github.com/cockroachdb/cockroach/pkg/sql/logictest.(*logicTest).runFile(0xc239594a00, 0xc000ea5230, 0x28, 0x452aca3, 0x16, 0x3, 0x1, 0x44f25d1, 0x2, 0x0, ...)
            /go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/logic.go:3132 +0x189
        github.com/cockroachdb/cockroach/pkg/sql/logictest.RunLogicTestWithDefaultConfig.func1.1(0xc03e975c80)
            /go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/logic.go:3324 +0x2de
        testing.tRunner(0xc03e975c80, 0xc1b4c981c0)
            /usr/local/go/src/testing/testing.go:1123 +0xef
        created by testing.(*T).Run
            /usr/local/go/src/testing/testing.go:1168 +0x2b3
        --- FAIL: TestLogic/fakedist-spec-planning/schema_change_in_txn (65.28s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.58s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 852e22a2ad78b1a19bd4f504f7079274c0c95fe5:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic570632428
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic570632428
--- FAIL: TestLogic (1167.38s)
=== RUN   TestLogic/fakedist-spec-planning/poison_after_push
=== PAUSE TestLogic/fakedist-spec-planning/poison_after_push
=== CONT  TestLogic/fakedist-spec-planning/poison_after_push
--- progress: testdata/logic_test/fk: 252 statements/queries
--- done: testdata/logic_test/rename_constraint with config fakedist-spec-planning: 15 tests, 0 failures
=== CONT  TestLogic/fakedist-spec-planning/poison_after_push
    logic.go:2540: 

        testdata/logic_test/poison_after_push:35: SELECT * FROM t
        expected success, but found
        (XXUUU) failed to connect to n2 at 127.0.0.1:33935: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210428 18:16:09.720477 11599440 sql/logictest/logic.go:3558  [-] 1206  
E210428 18:16:09.720477 11599440 sql/logictest/logic.go:3558  [-] 1206 +pq: failed to connect to n2 at 127.0.0.1:33935: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2273: 
         pq: failed to connect to n2 at 127.0.0.1:33935: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/poison_after_push with config fakedist-spec-planning: 7 tests, 2 failures
E210428 18:16:09.723213 11599440 sql/logictest/logic.go:3583  [-] 1207  testdata/logic_test/poison_after_push:40: too many errors encountered, skipping the rest of the input
    logic.go:3126: 
        testdata/logic_test/poison_after_push:40: error while processing
    logic.go:3126: testdata/logic_test/poison_after_push:40: too many errors encountered, skipping the rest of the input
        --- FAIL: TestLogic/fakedist-spec-planning/poison_after_push (1.92s)
=== RUN   TestLogic/fakedist-spec-planning
    --- FAIL: TestLogic/fakedist-spec-planning (0.67s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ ea6bcca8816a2705e46481eeb7543b16bb7c258c:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic134511204
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3351: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic134511204
--- FAIL: TestLogic (840.65s)
=== RUN   TestLogic/fakedist/poison_after_push
=== PAUSE TestLogic/fakedist/poison_after_push
=== CONT  TestLogic/fakedist/poison_after_push
=== CONT  TestLogic/fakedist/poison_after_push
    logic.go:2540: 

        testdata/logic_test/poison_after_push:35: SELECT * FROM t
        expected success, but found
        (XXUUU) failed to connect to n2 at 127.0.0.1:43475: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
        nodedialer.go:169: in dial()
E210429 07:43:04.114254 3494957 sql/logictest/logic.go:3558  [-] 511  
E210429 07:43:04.114254 3494957 sql/logictest/logic.go:3558  [-] 511 +pq: failed to connect to n2 at 127.0.0.1:43475: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
    logic.go:2273: 
         pq: failed to connect to n2 at 127.0.0.1:43475: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
--- done: testdata/logic_test/poison_after_push with config fakedist: 7 tests, 2 failures
E210429 07:43:04.115097 3494957 sql/logictest/logic.go:3583  [-] 512  testdata/logic_test/poison_after_push:40: too many errors encountered, skipping the rest of the input
    logic.go:3126: 
        testdata/logic_test/poison_after_push:40: error while processing
    logic.go:3126: testdata/logic_test/poison_after_push:40: too many errors encountered, skipping the rest of the input
E210429 07:43:04.118965 4588676 sql/stats/automatic_stats.go:302  [-] 513  failed to refresh stats: node unavailable; try another peer
        --- FAIL: TestLogic/fakedist/poison_after_push (1.42s)
=== RUN   TestLogic/fakedist/pg_extension
=== PAUSE TestLogic/fakedist/pg_extension
=== CONT  TestLogic/fakedist/pg_extension
E210429 07:43:02.638568 4568557 jobs/adopt.go:260  [n1] 500  job 654087889146118145: adoption completed with error could not remove enum value "hello" as it is being used by "use_greetings2" in row: k=1, v='hello'
--- done: testdata/logic_test/orms with config fakedist: 21 tests, 0 failures
=== CONT  TestLogic/fakedist/pg_extension
    logic.go:3130: 
        testdata/logic_test/pg_extension:55: error while processing
    logic.go:3130: test was successful but validation upon completion failed: failed to test for descriptor JSON round-trip: pq: failed to connect to n3 at 127.0.0.1:44877: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
E210429 07:43:04.106818 4588518 sql/stats/automatic_stats.go:302  [-] 509  failed to refresh stats: node unavailable; try another peer
E210429 07:43:04.113834 3494957 sql/logictest/logic.go:3571  [-] 510  
E210429 07:43:04.113834 3494957 sql/logictest/logic.go:3571  [-] 510 +testdata/logic_test/poison_after_push:35: SELECT * FROM t
E210429 07:43:04.113834 3494957 sql/logictest/logic.go:3571  [-] 510 +expected success, but found
E210429 07:43:04.113834 3494957 sql/logictest/logic.go:3571  [-] 510 +(XXUUU) failed to connect to n2 at 127.0.0.1:43475: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
E210429 07:43:04.113834 3494957 sql/logictest/logic.go:3571  [-] 510 +nodedialer.go:169: in dial()
        --- FAIL: TestLogic/fakedist/pg_extension (1.57s)
=== RUN   TestLogic/fakedist
    --- FAIL: TestLogic/fakedist (0.76s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.1]

/cc @jordanlewis

This test on roachdash | Improve this report!

ajwerner commented 3 years ago

Another https://teamcity.cockroachdb.com/viewLog.html?buildId=2941529&buildTypeId=Cockroach_UnitTests

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 0604c999494816236a1ed8660405144f071e655b:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic738526644
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3355: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic738526644
--- FAIL: TestLogic (1221.08s)
=== RUN   TestLogic/local-vec-off/inflight_trace_spans
=== PAUSE TestLogic/local-vec-off/inflight_trace_spans
=== CONT  TestLogic/local-vec-off/inflight_trace_spans
=== CONT  TestLogic/local-vec-off/inflight_trace_spans
    logic.go:2317: let $curr_trace_id = 1139691570243500368
E210624 19:44:03.657660 1390457 sql/logictest/logic.go:3563  [-] 170  
E210624 19:44:03.657660 1390457 sql/logictest/logic.go:3563  [-] 170 +
E210624 19:44:03.657660 1390457 sql/logictest/logic.go:3563  [-] 170 +testdata/logic_test/inflight_trace_spans:44: SELECT count(*) = 0
E210624 19:44:03.657660 1390457 sql/logictest/logic.go:3563  [-] 170 +  FROM current_trace_spans
E210624 19:44:03.657660 1390457 sql/logictest/logic.go:3563  [-] 170 +expected:
E210624 19:44:03.657660 1390457 sql/logictest/logic.go:3563  [-] 170 +    true
E210624 19:44:03.657660 1390457 sql/logictest/logic.go:3563  [-] 170 +but found (query options: "") :
E210624 19:44:03.657660 1390457 sql/logictest/logic.go:3563  [-] 170 +    false
    logic.go:2276: 

        testdata/logic_test/inflight_trace_spans:44: SELECT count(*) = 0
          FROM current_trace_spans
        expected:
            true
        but found (query options: "") :
            false
--- done: testdata/logic_test/inflight_trace_spans with config local-vec-off: 9 tests, 1 failures
        --- FAIL: TestLogic/local-vec-off/inflight_trace_spans (3.13s)
=== RUN   TestLogic/local-vec-off
    --- FAIL: TestLogic/local-vec-off (0.93s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot T-sql-schema branch-release-20.1]

/cc @cockroachdb/sql-queries jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ 5989de86b019c154076e42eb0cac1a55ac5c0be3:

=== RUN   TestLogic
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic769877379
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3362: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic769877379
--- FAIL: TestLogic (1369.21s)
=== RUN   TestLogic/local/inflight_trace_spans
=== PAUSE TestLogic/local/inflight_trace_spans
=== CONT  TestLogic/local/inflight_trace_spans
--- done: testdata/logic_test/int_size with config local: 32 tests, 0 failures
--- progress: testdata/logic_test/vectorize: 155 statements/queries
--- progress: testdata/logic_test/role: 263 statements/queries
=== CONT  TestLogic/local/inflight_trace_spans
    logic.go:2324: let $curr_trace_id = 9111394917085194647
=== CONT  TestLogic/local/inflight_trace_spans
    logic.go:2283: 

        testdata/logic_test/inflight_trace_spans:44: SELECT count(*) = 0
          FROM current_trace_spans
        expected:
            true
        but found (query options: "") :
            false
--- done: testdata/logic_test/inflight_trace_spans with config local: 9 tests, 1 failures
        --- FAIL: TestLogic/local/inflight_trace_spans (3.20s)
=== RUN   TestLogic/local
    --- FAIL: TestLogic/local (1.29s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ```

Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot T-sql-schema branch-release-20.1]

/cc @cockroachdb/sql-queries jordanlewis

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 years ago

sql/logictest.TestLogic failed with artifacts on master @ e723a85739a2e1b514f5de3221a2177a3a37d0e6:

            concat
        expected:
            count_1  count_lower  count_upper  concat
            3        3            3            aaa
            3        3            3            bbb

        but found (query options: "colnames") :
            count_1  count_lower  count_upper  concat
            3        3            3            
            3        3            3            bbb
E210812 05:33:45.323018 10152167 sql/logictest/logic.go:3566  [-] 939  
E210812 05:33:45.323018 10152167 sql/logictest/logic.go:3566  [-] 939 +testdata/logic_test/aggregate:3134: too many errors encountered, skipping the rest of the input
    logic.go:1818: 
         testdata/logic_test/aggregate:3134: too many errors encountered, skipping the rest of the input
--- done: testdata/logic_test/aggregate with config fakedist-disk: 499 tests, 2 failures
            --- FAIL: TestLogic/fakedist-disk/aggregate/string_agg (1.34s)
=== RUN   TestLogic/fakedist-disk/aggregate
=== PAUSE TestLogic/fakedist-disk/aggregate
=== CONT  TestLogic/fakedist-disk/aggregate
        --- FAIL: TestLogic/fakedist-disk/aggregate (12.51s)
=== RUN   TestLogic/fakedist-disk/window
=== PAUSE TestLogic/fakedist-disk/window
=== CONT  TestLogic/fakedist-disk/window
=== CONT  TestLogic/fakedist-disk/window
    logic.go:2274: 

        testdata/logic_test/window:385: SELECT k, concat_agg(s) OVER (PARTITION BY v ORDER BY w, k) FROM kv ORDER BY 1
        expected:
            1  ba
            3  Aa
            5  NULL
            6  bab
            7  b
            8  A

        but found (query options: "") :
            1  
            3  Aa
            5  NULL
            6  
            7  
            8  A
--- done: testdata/logic_test/window with config fakedist-disk: 56 tests, 1 failures
E210812 05:34:13.660944 9811824 sql/logictest/logic.go:3591  [-] 990  testdata/logic_test/window:395: too many errors encountered, skipping the rest of the input
    logic.go:3132: 
        testdata/logic_test/window:395: error while processing
    logic.go:3132: testdata/logic_test/window:395: too many errors encountered, skipping the rest of the input
        --- FAIL: TestLogic/fakedist-disk/window (2.27s)
=== RUN   TestLogic/fakedist-disk
    --- FAIL: TestLogic/fakedist-disk (0.63s)
Reproduce

To reproduce, try: ```bash make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` Parameters in this failure: - GOFLAGS=-json

Same failure on other branches

- #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2] - #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1] - #58269 sql/logictest: TestLogic failed [C-test-failure O-robot T-sql-schema branch-release-20.1]

/cc @cockroachdb/sql-queries jordanlewis

This test on roachdash | Improve this report!

yuzefovich commented 3 years ago

I was able to reproduce something related to the last hit here with fakedist-disk config after one and a half hours on the gceworker:

=== CONT  TestLogic/fakedist-disk/window
    logic.go:2541: 

        testdata/logic_test/window:2119: SELECT product_name, price, nth_value(product_name, 2) OVER w AS second FROM products WHERE price = 200 OR price = 700 WINDOW w as (PARTITION BY price ORDER BY product_name RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) ORDER BY price, product_name
        expected success, but found
        (XX000) internal error: unexpectedly found decreasing offsets: [0 7 0 9]
        bytes.go:80: in AssertOffsetsAreNonDecreasing()
        DETAIL: stack trace:
        github.com/cockroachdb/cockroach/pkg/col/coldata/bytes.go:80: AssertOffsetsAreNonDecreasing()
        github.com/cockroachdb/cockroach/pkg/col/coldata/bytes.go:547: AssertOffsetsAreNonDecreasing()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:87: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:146: spool()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/sort.go:283: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:197: func1()
        github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91: CatchVectorizedRuntimeError()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/disk_spiller.go:195: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:79: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:98: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/deselector.go:63: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:282: func1()
        github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:91: CatchVectorizedRuntimeError()
        github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:273: sendBatches()
        github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:407: runWithStream()
        github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:216: Run()
        github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:729: func1()
        github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1320: 1()

The expected output for the query is

Microsoft Lumia  200.00  Samsung
Samsung          200.00  NULL
Lenovo Thinkpad  700.00  Sony VAIO
Sony VAIO        700.00  iPad
iPad             700.00  NULL

It seems that the offsets are about the third column.

Guessing is that we're forgetting to call coldata.Batch.SetLength() somewhere, and it looks like the window functions might be to blame, but I didn't immediately find where. @DrewKimball curious if you have any insight. (Note that in the comment above both aggregate and window logic test files failed, but there are some window function in aggregate.)

I'm also not sure how/why fakedist-disk is the failing config.

DrewKimball commented 3 years ago

I don't see how we could be missing a SetLength, since it's always called immediately before outputting a batch. I think it might actually be a problem with Truncate - it can actually increase maxSetLength, meaning it increases the index from which we update offsets to be non-decreasing. I'll try and figure out exactly how that might happen.

DrewKimball commented 3 years ago

Hm, it looks like the one case where SpillingQueue.Enqueue doesn't call SetLength is when we spill immediately to disk and there is no selection vector. In that case, it should be possible to call Truncate with a larger index than maxSetIndex when there are nulls in the output. I'll see if I can write a query that reliably reproduces. We'd have to hit batch boundaries just right, so it might be tricky.