Add performance testing setup (close #37)

[matus-tomlein]

Issue: #37

Overview

This PR adds a new executable into the project that runs simple performance tests and logs their output. The goal is to be able to measure effect of changes in the tracker code on performance.

The tests only measure performance of tracking events without sending events over network. The idea is to measure the impact of the tracker on the "main" thread or the thread where events are tracked. The performance tests do not account for asynchronous operations (such as sending events).

The aim is to evaluate performance by comparison. There are several comparison that the tests enable:

1. Comparison with past performance. Output of each run is logged into a performance/logs.txt file and there is a script that extracts statistics from the historical and last performance.
2. Comparison of performance with and without emitter and session. We may add other variations in the future, but currently I think that the emitter and session have the biggest impact on performance. The tests compare the performance in 4 variations:
   1. mocked emitter and mocked session
   2. mocked emitter and real session
   3. mute emitter (standard with saving to DB, but without sending background loop) and mocked session
   4. mute emitter and real session

Each variation runs the same procedure – it tracks 10 000 timing, 10 000 screen view, 10 000 structured events in 5 threads in parallel (i.e., 150 000 events).

Running the performance tests

To run performance tests under macOS, you will need to build the project using make and run the tests:

 host> ./build/performance/tracker_performance

Performance logs

Each run of the tests appends a JSON to the performance/logs.txt file with desktop context describing the current machine and statistics extracted from the test. One measurement looks like this:

{
    "desktop_context": {
        "data": {
            "deviceManufacturer": "Apple Inc.",
            "deviceModel": "MacBookPro17,1",
            "deviceProcessorCount": 8,
            "osIs64Bit": true,
            "osServicePack": "",
            "osType": "macOS",
            "osVersion": "12.2.0"
        },
        "schema": "iglu:com.snowplowanalytics.snowplow/desktop_context/jsonschema/1-0-0"
    },
    "results": {
        "mocked_emitter_and_mocked_session": 5.273350042,
        "mocked_emitter_and_real_session": 5.072687333,
        "mute_emitter_and_mocked_session": 18.7668315,
        "mute_emitter_and_real_session": 22.612970792,
        "num_operations": 10000,
        "num_threads": 5
    },
    "timestamp": 1644589599181,
    "tracker_version": "cpp-0.1.0"
}

Comparing with historical performance

There is a simple Python script that compares the last run with historical performance. It outputs a table with several statistics.

To run the comparison script:

 host> ./performance/stats.py

This will generate a table something like this:

Metric	Max	Min	Mean	Last
mocked emitter and mocked session	5.27s	5.08s	5.18s	5.27s
mocked emitter and real session	5.08s	5.04s	5.06s	5.07s
mute emitter and mocked session	18.77s	17.29s	18.08s	18.77s
mute emitter and real session	28.02s	22.61s	25.06s	22.61s

[matus-tomlein]

@mscwilson Hey Miranda, I thought I'd tag you here in case this is interesting for you since you were working on similar performance testing tasks. No need to review the C++ code, probably more interesting is the general approach and procedure for performance testing. If you have any suggestions or would like to chat about it, let me know.

[mscwilson]

For the Python comparison table, can you add a column for n? Would give more context to the mean and other values.

[matus-tomlein]

Thanks for the comments, both!

Miranda, I changed the Python script to also show the number of operations and threads.

Alex, I don't have a clear idea yet how to integrate with CI. It would be ideal to run the tests on every tagged release, but not sure how to store the results of the test. One option would be to add it as asset to the release as Paul suggested yesterday. It requires some more thought, let's first see how useful the tests are as we change stuff in the tracker.