search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.99k stars 3.79k forks source link

roachtest: sysbench/oltp_read_only/nodes=3/cpu=32/conc=128 failed #71268

Closed cockroach-teamcity closed 3 years ago

cockroach-teamcity commented 3 years ago

roachtest.sysbench/oltp_read_only/nodes=3/cpu=32/conc=128 failed with artifacts on master @ cc6296c24ddb048215dabe5cc41339f306db4f41:

          | Wraps: (2) Node 4. Command with error:
          |   | ``````
          |   | sysbench \
          |   |         --db-driver=pgsql \
          |   |         --pgsql-host=127.0.0.1 \
          |   |         --pgsql-port=26257 \
          |   |         --pgsql-user=root \
          |   |         --pgsql-password= \
          |   |         --pgsql-db=sysbench \
          |   |         --report-interval=1 \
          |   |         --time=600 \
          |   |         --threads=128 \
          |   |         --tables=10 \
          |   |         --table_size=10000000 \
          |   |         --auto_inc=false \
          |   |         oltp_read_only run
          |   | ``````
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  tps: 4716.03 qps: 75707.51 (r/w/o: 66270.44/0.00/9437.06) lat (ms,95%): 47.47 err/s: 0.00 reconn/s: 0.00
          | [ 417s ] thds: 128 tps: 4734.33 qps: 75726.36 (r/w/o: 66259.69/0.00/9466.67) lat (ms,95%): 48.34 err/s: 0.00 reconn/s: 0.00
          | [ 418s ] thds: 128 tps: 4930.95 qps: 78812.15 (r/w/o: 68949.25/0.00/9862.90) lat (ms,95%): 46.63 err/s: 0.00 reconn/s: 0.00
          | [ 419s ] thds: 128 tps: 4607.13 qps: 73806.03 (r/w/o: 64590.78/0.00/9215.26) lat (ms,95%): 50.11 err/s: 0.00 reconn/s: 0.00
          | [ 420s ] thds: 128 tps: 4790.31 qps: 76594.89 (r/w/o: 67015.27/0.00/9579.61) lat (ms,95%): 48.34 err/s: 0.00 reconn/s: 0.00
          | [ 421s ] thds: 128 tps: 4903.46 qps: 78532.33 (r/w/o: 68725.41/0.00/9806.92) lat (ms,95%): 46.63 err/s: 0.00 reconn/s: 0.00
          | [ 422s ] thds: 128 tps: 4626.85 qps: 73959.63 (r/w/o: 64705.92/0.00/9253.71) lat (ms,95%): 49.21 err/s: 0.00 reconn/s: 0.00
          | [ 423s ] thds: 128 tps: 4865.60 qps: 77933.63 (r/w/o: 68201.43/0.00/9732.20) lat (ms,95%): 48.34 err/s: 0.00 reconn/s: 0.00
          | [ 424s ] thds: 128 tps: 4716.86 qps: 75463.77 (r/w/o: 66031.04/0.00/9432.72) lat (ms,95%): 49.21 err/s: 0.00 reconn/s: 0.00
          | [ 425s ] thds: 128 tps: 5025.37 qps: 80447.94 (r/w/o: 70396.20/0.00/10051.74) lat (ms,95%): 46.63 err/s: 0.00 reconn/s: 0.00
          | [ 426s ] thds: 128 tps: 4684.77 qps: 74670.28 (r/w/o: 65303.74/0.00/9366.54) lat (ms,95%): 49.21 err/s: 0.00 reconn/s: 0.00
          | [ 427s ] thds: 128 tps: 4389.89 qps: 70523.29 (r/w/o: 61744.50/0.00/8778.79) lat (ms,95%): 51.02 err/s: 0.00 reconn/s: 0.00
          | FATAL: PQexecPrepared() failed: 7 root: memory budget exceeded: 40960 bytes requested, 7571107081 currently allocated, 7571136512 bytes in budget
          | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = '53200': root: memory budget exceeded: 40960 bytes requested, 7571107081 currently allocated, 7571136512 bytes in budget
          | FATAL: PQexecPrepared() failed: 7 root: memory budget exceeded: 30720 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = '53200': root: memory budget exceeded: 30720 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: PQexecPrepared() failed: 7 root: memory budget exceeded: 40960 bytes requested, 7571117321 currently allocated, 7571136512 bytes in budget
          | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = '53200': root: memory budget exceeded: 40960 bytes requested, 7571117321 currently allocated, 7571136512 bytes in budget
          | FATAL: PQexecPrepared() failed: 7 root: memory budget exceeded: 10240 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = '53200': root: memory budget exceeded: 10240 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: PQexecPrepared() failed: 7 root: memory budget exceeded: 10240 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = '53200': root: memory budget exceeded: 10240 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: PQexecPrepared() failed: 7 root: memory budget exceeded: 20480 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = '53200': root: memory budget exceeded: 20480 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: PQexecPrepared() failed: 7 root: memory budget exceeded: 10240 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
          | FATAL: ``thread_run' function failed: /usr/share/sysbench/oltp_common.lua:432: SQL error, errno = 0, state = '53200': root: memory budget exceeded: 10240 bytes requested, 7571127561 currently allocated, 7571136512 bytes in budget
        Wraps: (8) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *exec.ExitError
Reproduce

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

tbg commented 3 years ago

https://github.com/cockroachdb/cockroach/pull/71280