search

tzaeschke / TinSpin

Testing in-memory spatial indexes
Apache License 2.0
26 stars 1 forks source link

How to add benchmark #1

Closed karussell closed 5 years ago

karussell commented 6 years ago

How would I add a benchmark e.g. for our LocationIndexTree (memory efficient quadtree)?

And I did not see any code to warm up the JVM is this considered? Or maybe you can have a look into JMH?

tzaeschke commented 6 years ago

How would I add a benchmark e.g. for our LocationIndexTree (memory efficient quadtree)?

Have a look at the classes in ch.ethz.glbis.tinspin.wrappers, they are the wrapper classes for different indexes. There are separate wrappers for Point data and Rectangle Data. Then, you need to register the new wrapper in TestStats.IDX. If you don't want to modify the Enum, you can use the generic 'IDX.CUSTOM' enum, see AbstractWrapperTest for an example (just need to call setCandidateClass()).

Let me know if you get stuck.

And I did not see any code to warm up the JVM is this considered?

There are two types of warmup:

  1. Each test is run twice, first with a dataset size of 100'000, then again with the full desired size. See TestRunnerLocal.test(...).
  2. Most loads (except insert and remove) are run twice, so the first run can be considered as warmup.

Or maybe you can have a look into JMH?

I think JMH can be very useful for low level routines. However, an index is a more complex beast whose performance appears to mostly depend on the dataset (size, dimensionality, distribution) and environment (type of machine, primary loads). I found that the bottleneck is usually the amount of data and operations that need to be processed (which both depend on the general algorithm), while low-level optimization tend to have negligible effect on overall performance, except in rare cases. Did you have something in mind related to JMH?

tzaeschke commented 6 years ago

Btw, if you don't want to run the full suite: I usually do preliminary testing by directly running the TestRunner.

karussell commented 6 years ago

Thanks, will try!

Did you have something in mind related to JMH?

No. Just wanted to point to a framework for benchmarking, but likely you coded something simple enough to replace it (like we do for our stuff :))

tzaeschke commented 5 years ago

Just in case this is still of interest, TinSpin has been refactored to simplify integration of custom indexes and datasources. See TinSpin/src/main/java/ch/ethz/globis/tinspin/Example.java for an example.