Performance bash

[jarohen]

While we're happy that the underlying architecture of XT2 is still a good baseline for competitive performance in the future, we've deliberately not focused on precise performance numbers or benchmarking in the last few months, in order to get the early access version in front of users for their feedback. Now that we have that, we'd like to better understand where our performance is at, and what actions we can take to improve it.

We currently have TPC-H (OLAP) and AuctionMark (OLTP) benchmarks set up for XT 2 - the former is runnable at a REPL, the latter additionally through 'bench2'

Our current aims in this area:

• Get us on a good footing for future performance optimisation work - e.g. giving us the ability to easily run and compare statistically-sound benchmarks.
  • [x] One benchmarking framework
  • [x] #3135
  • [x] #3136
  • [x] Transaction/query Micrometer metrics moved to core engine so that we don't need a wrapper in bench #3149
  • [x] #3158
  • [x] #3172
  • [ ] Individual benchmarks able to submit their own metrics (e.g. Auctionmark/TPC-H query timings)
  • [x] Ability to visualise metrics locally - use off-the-shelf tooling that we can re-use in production clusters.
  • [x] docker-compose to start Prometheus/Grafana, connect one to t'other.
  • [ ] ~XT metrics configurable to post to a Prometheus server~
  • [x] Example Grafana dashboard
  • [x] Dev instructions to do all of the above.
  • [ ] AWS (nice-to-have)
  • [x] Ability to run the benchmark on AWS
  • [x] Metrics configurable to post to Cloudwatch
• Incrementing the index structure to cope with larger data-sets
  • [x] #2652
  • [x] #2796
  • [x] #3138
• Understand what our current numbers are - primarily relative to XT1 (for ease of comparison), but also what other databases are able to achieve.
• Understand where the current performance bottlenecks are (likely separately) for OLAP and OLTP queries.
  • [x] #3200
  • [x] #3210
  • [x] #3209
  • [x] #2045
  • [ ] #2830
  • [ ] #3166
  • [x] #2739
• Understand what the limits are in terms of how much data can currently be put into XT2 before the performance becomes prohibitive.

[jarohen]

Will close this - the remaining two cards can be prio'd in isolation