Tests run long, which produces inaccurate RPS, is this intended?

[sparkida]

When a test being ran is given a high rate like this:

  phases:
    -
      duration: 300
      arrivalRate: 5500

I'd expect the test to end at the duration's end. For example, if the test has been running for 5 minutes but has any requests still processing, the test should end showing how many scenarios it could not complete in those 5 minutes.

This would at least help gauge what a realistic test looks like, whereas currently I have ran the exact same two tests concurrently, using the above phase. The first test completed in about 7 minutes (2+ minutes overrun), the second test is still running (23 minutes, 18+ minutes of overrun).

Additionally the RPS accuracy is being affected as it is slowly dropping while tasks complete the remaining requests, whereas, if each task was given 5 minutes time to run, they'd have an accurate RPS for a 5 minute period.

I'd like to hear your thoughts on this, please share :)

[sparkida]

I had to manually stop the test here as well.

[hassy]

This comes up frequently and we should probably update the FAQ / docs! What we set with duration here is the duration of arrival phases for our virtual users, which does not have a direct relationship with the total running time of the test. The total running time depends on how many requests need to be made in each scenario and server's response time (since each virtual users will wait for a request to complete before sending the next one in the scenario). Slower response times (e.g. when the server is getting overloaded) will increase total running time of the test.

Now being able to set a limit on the total running time of the test could be an interesting feature, especially when comparing two alternative configurations/implementations of the same service/API.

[sparkida]

Thanks for explaining @hassy, I had a feeling you had come across this question in the past, but couldn't find anything in the issues, I'll check the docs in the future.

I do think that this would be a nice feature to have, in the case where I'd like to stress our servers without having to run a large array of tests to find the sweetspot. Instead, I'd like to just throw an unreasonable rate at them for a certain time period and gauge how well they handled it. I'll keep thinking about this, but I've dealt with it in the past and ended up using my own system with the community Artillery and my own fleet of time-boxed servers.

[NivLipetz]

I had this exact same problem and a good solution to it is limiting the number of virtual users in the phase. For the merged PR and how to limit the virtual users you can look here: https://github.com/artilleryio/artillery/pull/615

[sparkida]

@NivLipetz thanks for offering this solution, can you help me understand how this solves the problem of having to guess the load? From what you have provided so far, I'd still run into the main issues: tests overrun duration set and affect RPS, and tests will not yield useful information into the load capabilities of a system.

[NivLipetz]

@sparkida sorry for the delayed reply, if you know your server can't handle more than X RPS, then you can limit the number of virtual users in your test that it won't exceed this max RPS that your server can handle. That way no matter what your arrival rate is during the test, it won't pass this max RPS by "throwing away" the requests after reaching this limit. This will keep your load test flowing smoothly without any queued requests keeping the final artillery report relevant and useful