search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.2k stars 3.82k forks source link

roachtest: ycsb/B/nodes=3 failed #80049

Closed cockroach-teamcity closed 2 years ago

cockroach-teamcity commented 2 years ago

roachtest.ycsb/B/nodes=3 failed with artifacts on master @ 771432d1099e516dbc11827c5458886c176e73e3:

          |   212.0s        0         1471.6         1413.3     11.5     27.3     41.9     58.7 update
          |   213.0s        0        27239.1        26809.7      3.7     11.5     26.2     56.6 read
          |   213.0s        0         1471.3         1413.5     12.6     29.4     46.1     58.7 update
          |   214.0s        0        27217.1        26811.6      3.7     12.1     27.3     79.7 read
          |   214.0s        0         1456.9         1413.7     12.1     26.2     44.0     83.9 update
          |   215.0s        0        27483.6        26814.8      3.7     11.5     26.2     67.1 read
          |   215.0s        0         1496.7         1414.1     12.1     28.3     46.1     71.3 update
          |   216.0s        0         3397.4        26706.3      3.7     15.7     31.5     54.5 read
          |   216.0s        0          192.9         1408.5     12.1     33.6     46.1     56.6 update
          |   217.0s        0            0.0        26583.3      0.0      0.0      0.0      0.0 read
          |   217.0s        0            0.0         1402.0      0.0      0.0      0.0      0.0 update
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 4. Command with error:
          | ``````
          | ./workload run ycsb --init --insert-count=1000000 --workload=B --concurrency=144 --splits=3 --histograms=perf/stats.json --select-for-update=true --ramp=2m --duration=30m {pgurl:1-3}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,ycsb.go:84,ycsb.go:101,test_runner.go:875: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerYCSB.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/ycsb.go:84
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerYCSB.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/ycsb.go:101
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:875
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-15652

cockroach-teamcity commented 2 years ago

roachtest.ycsb/B/nodes=3 failed with artifacts on master @ 771432d1099e516dbc11827c5458886c176e73e3:

          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |   201.0s        0        36298.5        35999.8      2.5     10.5     24.1     67.1 read
          |   201.0s        0         1891.7         1897.4      8.1     19.9     32.5     65.0 update
          |   202.0s        0        36849.8        36003.9      2.5      9.4     25.2     83.9 read
          |   202.0s        0         1916.2         1897.5      7.9     21.0     33.6     75.5 update
          |   203.0s        0        35611.4        36002.0      2.5     10.0     27.3     67.1 read
          |   203.0s        0         1941.2         1897.7      7.9     21.0     33.6     75.5 update
          |   204.0s        0        24103.3        35943.6      2.6     10.5     23.1     71.3 read
          |   204.0s        0         1314.1         1894.8      8.1     21.0     35.7     71.3 update
          |   205.0s        0            0.0        35768.2      0.0      0.0      0.0      0.0 read
          |   205.0s        0            0.0         1885.6      0.0      0.0      0.0      0.0 update
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 4. Command with error:
          | ``````
          | ./workload run ycsb --init --insert-count=1000000 --workload=B --concurrency=144 --splits=3 --histograms=perf/stats.json --select-for-update=true --ramp=2m --duration=30m {pgurl:1-3}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,ycsb.go:84,ycsb.go:101,test_runner.go:875: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerYCSB.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/ycsb.go:84
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerYCSB.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/ycsb.go:101
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:875
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

kvoli commented 2 years ago

tracking issue #80072, caused by #76609. Resolved in #80071.