search

elastic / elasticsearch-net

This strongly-typed, client library enables working with Elasticsearch. It is the official client maintained and supported by Elastic.
https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/index.html
Apache License 2.0
10 stars 1.15k forks source link

Benchmarking/Profiling the clients #2040

Closed russcam closed 7 years ago

russcam commented 8 years ago

This issue aims to discuss our approach to benchmarking and profiling for the purpose of being able to track the performance of the clients as the source is iterated upon.

Definitions

Firstly, some clarfication on what we mean by Benchmarking and Profiling:

Benchmarking

Benchmarking is the process of measuring the performance of a program by getting it to process some task and measuring how long it takes.

The fundamental reason for benchmarking is to be able to identify when a change has made the program run slower. It also allows us to have some rough metrics for expected throughput for a given environment.

Profiling

Profiling is used to identify the parts of a program which take the most time and sometimes, the most memory.

Performance in the context of profiling is relative only to the components that make up the program; that is, the part of a program that takes the most time may not be the slowest part of a process in which the program is used. For example, in the case of NEST, the json serializer may be the slowest part of the client but the time taken by serializer could be insignificant in comparison to the latency of making a HTTP request.

Both Benchmarking and Profiling are important; the former to understand as a whole where performance has changed within the client and the latter, to understand where the time/memory bottlenecks within the client could be improved.

Approach

Benchmarking

There are a number of projects in the .NET space that have sprung up around Benchmarking; the two that look to be showing the most promise are

BenchmarkDotNet is a low level library for performance testing a piece of code; the code to be executed is placed within a [Benchmark] attributed method, optionally with two other methods to perform setup and teardown for the benchmark. BenchmarkDotNet takes the source of the method and generates a separate isolated executable in which to benchmark the running of the method. This is performed for every method to be benchmarked.

NBench is a higher level library for performing benchmarking that internally uses BenchmarkDotNet. The intention with NBench is to be able to write benchmarking unit tests to make assertions about program performance, in addition to being able to collecting metrics as per BenchmarkDotNet

Goals

What do we want to get out of benchmarking? Obviously we want to record performance metrics of the client, but there are a number of areas for discussion:

Do we want to benchmark all methods or a meaningful subset?

My initial thoughts here are that we should look at benchmarking the most travelled paths to begin:

  1. bulk indexing, using a configuration like the one used in the Elasticsearch index benchmarks
  2. searching, perhaps by generating queries to execute against a known datasource, similar to what Lucene benchmarks do with Wikipedia English export
    Elasticsearch cluster configuration

It might make sense to use the same configuration as is used for the Elasticsearch nightly benchmarks

Collecting other metrics

Both BenchmarkDotNet and NBench expose collection of GC and memory allocation stats. Are there other metrics that we might be interested in capturing through e.g. performance counters, event tracing?

Ability to run from the command line

So that we can set up our CI environment to run benchmarks nightly/on each checkin/on each release.

Use our existing Tests project

It would be ideal if we could use the existing suite of integration tests for benchmarking purposes, avoiding the overhead in maintaining another suite of tests.

Plot/Illustrate performance over time

Visualizing performance would provide an easy way to determine and investigate any issues that may occur.

Challenges

  1. BenchmarkDotNet runs against Core CLR and CLR using DNX tooling with the newer dotnet cli toolchain; NEST is currently still on the RC1 tooling of DNX using dnvm/dnu/dnx toolchain. The project used to run against this toolchain but maybe it a sign that we should update to the new dotnet cli.
  2. NBench does not run against Core CLR or DNX (dotnet cli or older) currently.
  3. Due to the way in which benchmarks run (extracting the method out and generating a separate executable), it looks like from initial research that it would be tricky to re-use integration tests as they are written for benchmarking purposes. This may be a non-issue to begin if we're only looking to benchmark a meaningful subset of operations.
    So far

The feature/profiling-benchmarkdotnet branch has a POC for benchmarking using BenchmarkDotNet. This is an alpha to get it running against the dnx build of the solution, which is incomplete for Core CLR due to challenge 1 above.

The feature/profiling-nbench branch has a POC for benchmarking using NBench. This builds the non-DNX solution using MSBuild to produce a .NET 4.5 compatible version of NEST.

Profiling

A number of tools are available for profiling

Only the last tool is free for commercial use.

Goals

Persist Profile snapshots over time

Record profile snapshots for components of the client to be able to determine

Ability to Diff snapshots

So we can quickly determine where performance may have got better (or worse)

Ability to run from the command line and IDE

So that we can set up our CI environment to run profiling nightly/on each checkin/on each release, as well as be able to profile ad-hoc within the IDE when the need arises.

Use our existing Tests project

It would be ideal if we could use the existing suite of integration tests for profiling purposes, avoiding the overhead in maintaining another suite of tests.

Since we would need some task to process in order to perform profiling, we could use the benchmarking components for profiling purposes.

Plot/Illustrate performance profiles over time

Same as for benchmarking

Challenges

I have experience with both DotTrace and ANTS which makes me lean towards using one of those for profiling purposes and, since I already have a JetBrains license, this makes using DotTrace more compelling.

So far

The feature/profiling branch has a POC for profiling using DotTrace's Self Profiling API. The intention with the setup in the branch have is to be able to attribute types in the Tests project with different attributes to run performance and timeline profiling; The Profiling project simply being a shell for running the profiler by

  1. pulling out test types and methods that are attributed with [Performance] and [Timeline] attributes in the Tests project via Reflection.
  2. creating an instance of the type and passing it the Elasticsearch cluster against which it will run
  3. Starting a profiling session by self attaching the profiler to the running Profiling process.
  4. Running the test method and saving a snapshot of the profile session for it, optionally running it a number of iterations.
  5. Repeating steps 2,3 and 4 for each attributed test method.

The profile snapshots would be saved into separate files as this should make it easier to understand where performance may have changed over time, based on knowing which method a snapshot relates to.

This setup has intermittently worked in the past but now when attaching the profiler, the profiler never becomes active; It looks like JetBrains are looking into this issue but thet have also just released a new version of DotTrace (2016.1) that should be tried.

adamsitnik commented 8 years ago

As a person that is responsible for .NET Core support in BechmarkDotNet project I am very happy to see people using it. Please do not hesitate to ask any questions, I will do my best to solve any upcoming issues.

@russcam I have commented your question in our issues list, please take a look. I think that we already have what you need, except that you will have to install dotnet cli toolchain on the machines that you would like to run your benchmarks.

Btw soon we should have a possibility to run single Benchmark for multiple runtimes so you could compare your performance for all supported frameworks in one place (Classic Clr, Dnx451, .NET Core)

mattwarren commented 8 years ago

Both BenchmarkDotNet and NBench expose collection of GC and memory allocation stats. Are there other metrics that we might be interested in capturing through e.g. performance counters, event tracing?

I've done most of the work on "diagnostics" in BenchmarkDotNet. For reference we already use ETW to capture memory allocations and we plan to have performance counter support soon (ish!). Please feel free to add a comment to an existing issue or create a new one for anything else that you would need.

russcam commented 8 years ago

Thanks @adamsitnik and @mattwarren! This was a very rough brain dump of thoughts on benchmarking and profiling for discussion.

I'll be sure to reach out if we come up with other ideas :+1: