search

chaincodelabs / bitcoinperf

High-level performance monitoring framework for Bitcoin Core
https://bitcoinperf.com
52 stars 16 forks source link

Big refactor: easier local use for comparisons #33

Closed jamesob closed 6 years ago

jamesob commented 6 years ago

Did a big 'ol refactor of the runner to make it much easier to do comparisons of different refs locally. This motivated moving the synced bitcoind into its own docker container for easier reuse over separate bench runner invocations.

While I was at it, I also added a flag called --run-counts (e.g. ibd:3,reindex:2) that allows the user to specify how many times a benchmark should be run. This should help to remove some of the noise of running benchmarks on a multi-use (vs. dedicated) system.

Separate git refs can now be specified for comparison with --commits a,b,c. Passing a single (or no) ref will result in the same behavior as before.

An example invocation with some of the new flags is below, which compares master to the v0.16.0 tag, showing a 24% improvement in IBD time!

$ docker-compose run --rm bench ./runner/run_bench.py --commits "v0.16.0,master" --no-clean=1 --run-counts "ibd:3" --benches-to-run "gitclone,build,ibd" --log-level INFO --bitcoind-stopatheight 200000

2018-08-17 17:16:24,541 __main__ [INFO] Running with configuration:
2018-08-17 17:16:24,544 __main__ [INFO]
2018-08-17 17:16:24,544 __main__ [INFO]   benches_to_run             ['gitclone', 'build', 'ibd']
2018-08-17 17:16:24,545 __main__ [INFO]   bitcoind_assumevalid       000000000000000000176c192f42ad13ab159fdb20198b87e7ba3c001e47b876
2018-08-17 17:16:24,545 __main__ [INFO]   bitcoind_dbcache           2048
2018-08-17 17:16:24,545 __main__ [INFO]   bitcoind_port              9003
2018-08-17 17:16:24,545 __main__ [INFO]   bitcoind_rpcport           9004
2018-08-17 17:16:24,545 __main__ [INFO]   bitcoind_stopatheight      200000
2018-08-17 17:16:24,545 __main__ [INFO]   codespeed_envname          ccl-bench-hdd-1
2018-08-17 17:16:24,545 __main__ [INFO]   codespeed_no_send          False
2018-08-17 17:16:24,545 __main__ [INFO]   codespeed_password         password
2018-08-17 17:16:24,545 __main__ [INFO]   codespeed_url              http://codespeed:8000
2018-08-17 17:16:24,545 __main__ [INFO]   codespeed_user             admin
2018-08-17 17:16:24,545 __main__ [INFO]   commits                    ['v0.16.0', 'master']
2018-08-17 17:16:24,545 __main__ [INFO]   compilers                  ['clang']
2018-08-17 17:16:24,545 __main__ [INFO]   ibd_peer_address           synced
2018-08-17 17:16:24,545 __main__ [INFO]   log_level                  INFO
2018-08-17 17:16:24,546 __main__ [INFO]   make_jobs                  5
2018-08-17 17:16:24,546 __main__ [INFO]   no_caution                 True
2018-08-17 17:16:24,546 __main__ [INFO]   no_clean                   True
2018-08-17 17:16:24,546 __main__ [INFO]   no_teardown                False
2018-08-17 17:16:24,546 __main__ [INFO]   nproc                      4
2018-08-17 17:16:24,546 __main__ [INFO]   repo_branch                master
2018-08-17 17:16:24,546 __main__ [INFO]   repo_location              https://github.com/bitcoin/bitcoin.git
2018-08-17 17:16:24,546 __main__ [INFO]   run_counts                 {'ibd': 3}
2018-08-17 17:16:24,546 __main__ [INFO]   slack_webhook_url
2018-08-17 17:16:24,546 __main__ [INFO]   synced_bitcoin_repo_dir    /bitcoin
2018-08-17 17:16:24,546 __main__ [INFO]   synced_bitcoind_args       -minimumchainwork=0x00
2018-08-17 17:16:24,546 __main__ [INFO]   synced_data_dir            /bitcoin/data
2018-08-17 17:16:24,546 __main__ [INFO]   workdir
2018-08-17 17:16:24,546 __main__ [INFO]

...
                                              master                   v0.16.0

build.make.5.clang                            0:04:41.262233 (x1)      0:04:18.015711 (x1)
ibd.local.200000.dbcache=2048                 0:03:16.758905 (x3)      0:04:04.669509 (x3)
ibd.local.200000.dbcache=2048.mem-usage       863.716MiB (x3)          856.366MiB (x3)
gitclone                                      0:00:11.970551 (x1)
build.make.5.clang.mem-usage                  612.424MiB (x1)          636.048MiB (x1)

Relative measurements:

build.make.5.clang                            1.090097311723555 (x1)   1.0 (x1)
ibd.local.200000.dbcache=2048                 1.0 (x3)                 1.2434990374618275 (x3)
ibd.local.200000.dbcache=2048.mem-usage       1.00858199369429 (x3)    1.0 (x3)
gitclone                                      1.0 (x1)
build.make.5.clang.mem-usage                  1.0 (x1)                 1.03857458231552 (x1)
jamesob commented 6 years ago

Post-merge review appreciated as always.