Benchmark the Tangle cache

jakubcech commented 5 years ago

Description

We should run a test (prefferably JMH) on the speed of loading a larger amount of transactions from:

With our custom cache
Without our custom cache

Motivation

Allow us to determine how slower the caching in RocksDB actually is and see what is the benfit of our cache

Compare concurrency for both implementations.

Requirements

Base test:

Write transactions to the DB w/o caching.
Read them in a random order.
Read them in a reverse order.
Read them in the same order.
Write transactions to the DB with caching.
Read them in a random order.
Read them in a reverse order.
Read them in the same order.

[ ] Tinker with the RocksDB cache settings.
[ ] Compare different sizes of the cache to one another (in tx):
- 500
- 2400
  - 12000
  - 24000
[ ] Test parallel reads using 3 threads and compare how fast you're able to chew through a set of data.
[ ] Create a test that reads the keys more than once to ensure RocksDB variance is accounted for.
[ ] Make the benchmark beautiful. Or use JMH.

If you're having a ton of fun:

Write the values to the RocksDB with cache off
Read them in a random order.
Read them in a reverse order.
Read them in the same order.

GalRogozinski commented 4 years ago

@kwek20 and @luca-moser I understood you are done with this?

Can you write a summary of what you have done in a comment so we can close this please?

GalRogozinski commented 4 years ago

The description of this test has been edited

acha-bill commented 4 years ago

Initial test using DefaultCacheConfiguration.

DEFAULT_MAX_SIZE = 1000;
DEFAULT_EVICTION_COUNT = 10;
DEFAULT_CONCURRENCY_LEVEL = 5;

Results

Benchmark                                                  (numTxsToTest)  Mode  Cnt   Score   Error  Units
CacheBenchmark.readTransactionsInOrderWithCacheOff                    500  avgt   10   1.634 ± 0.239  ms/op
CacheBenchmark.readTransactionsInOrderWithCacheOn                     500  avgt   10   0.114 ± 0.007  ms/op
CacheBenchmark.readTransactionsInOrderWithCacheOff                   2400  avgt   10   8.817 ± 1.774  ms/op
CacheBenchmark.readTransactionsInOrderWithCacheOn                    2400  avgt   10   1.329 ± 0.018  ms/op
CacheBenchmark.readTransactionsInOrderWithCacheOff                  12000  avgt   10  73.856 ± 4.906  ms/op
CacheBenchmark.readTransactionsInOrderWithCacheOn                   12000  avgt   10  14.838 ± 0.094  ms/op
CacheBenchmark.readTransactionsInRandomOrderWithCacheOff              500  avgt   10   1.508 ± 0.166  ms/op
CacheBenchmark.readTransactionsInRandomOrderWithCacheOn               500  avgt   10   0.123 ± 0.002  ms/op
CacheBenchmark.readTransactionsInRandomOrderWithCacheOff             2400  avgt   10   9.676 ± 2.164  ms/op
CacheBenchmark.readTransactionsInRandomOrderWithCacheOn              2400  avgt   10   1.566 ± 0.013  ms/op
CacheBenchmark.readTransactionsInRandomOrderWithCacheOff            12000  avgt   10  89.529 ± 5.068  ms/op
CacheBenchmark.readTransactionsInRandomOrderWithCacheOn             12000  avgt   10  17.269 ± 1.520  ms/op
CacheBenchmark.readTransactionsInReverseOrderWithCacheOff             500  avgt   10   1.535 ± 0.211  ms/op
CacheBenchmark.readTransactionsInReverseOrderWithCacheOn              500  avgt   10   0.118 ± 0.006  ms/op
CacheBenchmark.readTransactionsInReverseOrderWithCacheOff            2400  avgt   10   8.718 ± 1.404  ms/op
CacheBenchmark.readTransactionsInReverseOrderWithCacheOn             2400  avgt   10   1.611 ± 0.328  ms/op
CacheBenchmark.readTransactionsInReverseOrderWithCacheOff           12000  avgt   10  72.687 ± 4.658  ms/op
CacheBenchmark.readTransactionsInReverseOrderWithCacheOn            12000  avgt   10  15.637 ± 0.162  ms/op

acha-bill commented 4 years ago

The branch is available here: https://github.com/acha-bill/iri_benchmark

iotaledger / iri