Large-scale performance tests like in the paper

[llarsson]

Please document how one would replicate the (distributed) load tests that are mentioned in Section IV ("Use-cases") of the paper. How many hosts were used for the TeaStore, and how many were used to subject it to load? Are there any scripts that you used that may be of value, should one like to independently verify the results presented in the paper (and build upon them)?

[SimonEismann]

There might be minor differences between the three experiments, but for the first one we used the following setup:

• The registry is placed on an isolated host, as we did not consider it as part of the system under test in this experiment.
• The other five services are deployed as described in figure 4 across five additional hosts (HPE ProLiant DL160 servers, each equipped with a Xeon E5-2640 v3 processor with 16 logical cores at 2.6 GHz and 32 GB RAM.)
• For the load generation, we used the LIMBO Load Generator with the default browse profile
• The number of machines for the load generation is unfortunately not documented in the paper, but based on the throughput numbers I'd guess it was 3-4 machines.
• The load level was 1000, 2000, 3000, 4000 and 5000 requests per second.

Off the top of my head, I can't think of any scripts that might be useful for you. In general, a performance test consists of 3 tasks:

• Deploying the system under test This depends a bit on the setup you are going for, but the simplest solution is probably docker-compose files on each host and a script that connects to the hosts via SSH to run docker-compose up/down.
• Generating Load Here, I'd recommend the LIMBO Load generator. It has a few weird quirks, but is quite powerful and scales well for large request rates.
• Metric collection For each performance experiment, we need to collect the performance metrics we are interested in. Common examples here are resource utilization, end-to-end response time and per-request tracing information. The LIMBO Load generator per default only reports end-to-end response time. For actual tracing information, I'd recommend using kieker traces. For the utilization measurements, we used a not production ready extension for the LIMBO Load generator (hit me up via email, in case you are interested), but its also pretty straight forward to build a custom solution here.

If you are interested, we can also discuss this in a quick skype call.

[llarsson]

A LIMBO beginner's two follow-up questions:

• To get an overall request per second rate of 5000, would I typically then use 5 load generator machines and provide them each with a load intensity profile of 1000 requests per second?
• How does the number of threads relate to the load generator's ability to function? What would be a reasonable number for e.g. 1000 requests per second?

[SimonEismann]

• To get an overall request per second rate of 5000, would I typically then use 5 load generator machines and provide them each with a load intensity profile of 1000 requests per second?

For large experiments that require more than a single loaddriver instance to generate the required load, the LIMBO loadgenerator supports using multiple loadgenerator instances with a single director instance. To do so, the --ip commands can parse multiple ips (seperated by a comma, without any spaces e.g. 10.0.0.0,10.0.0.1). So in your scenario, a single load intensity file with 5000 requests per second, a single director instance and five loadgenerator instances should work.

• How does the number of threads relate to the load generator's ability to function? What would be a reasonable number for e.g. 1000 requests per second?

Each thread sends out a request, waits for a reply and is then added back to the pool of available threads to send out new request. Therefore, the required number of threads depends on the response time of the application, for example if the application response within 0.5s, then 500 threads could generate a load of 500 requests. A too large number of threats combined with complex user sessions can generate some unintended behavior, however, this can be fixed using the flag --randomize-users.

[SimonEismann]

Closed as stale, feel free to reopen if the issue still persists.