artilleryio / artillery

The complete load testing platform. Everything you need for production-grade load tests. Serverless & distributed. Load test with Playwright. Load test HTTP APIs, GraphQL, WebSocket, and more. Use any Node.js module.
https://www.artillery.io
Mozilla Public License 2.0
8.03k stars 510 forks source link

Parallel causing incorrect request latency #742

Open kingjerod opened 5 years ago

kingjerod commented 5 years ago

Not sure if parallel is supported in the most recent version, but using it causes incorrect numbers in the summary. Maybe I'm just not understanding the numbers? I created a very basic server that just returns a string after waiting 1 second. So the min/max/average request time should always be nearly 1000 ms. If I do 3 requests in parallel, the numbers jump to 3000 ms. Test:

{
  "config": {
    "target": "http://localhost:3333",
    "phases": [
      {
        "duration": 5,
        "arrivalRate": 1
      }
    ]
  },
  "scenarios": [
    {
      "flow": [
        {
          "parallel": [
            {
              "get": {
                "url": "/"
              }
            },
            {
              "get": {
                "url": "/"
              }
            },
            {
              "get": {
                "url": "/"
              }
            }
          ]
        }
      ]
    }
  ]
}

Results:

Started phase 0, duration: 5s @ 12:20:41(-0700) 2019-09-11
Report @ 12:20:49(-0700) 2019-09-11
Elapsed time: 8 seconds
  Scenarios launched:  5
  Scenarios completed: 5
  Requests completed:  15
  RPS sent: 2.01
  Request latency:
    min: 1001
    max: 3009.7
    median: 2005.8
    p95: 3009.3
    p99: 3009.7
  Codes:
    200: 15

All virtual users finished
Summary report @ 12:20:49(-0700) 2019-09-11
  Scenarios launched:  5
  Scenarios completed: 5
  Requests completed:  15
  RPS sent: 2.01
  Request latency:
    min: 1001
    max: 3009.7
    median: 2005.8
    p95: 3009.3
    p99: 3009.7
  Scenario counts:
    0: 5 (100%)
  Codes:
    200: 15

My basic HAPI server looks like this:

'use strict';

import * as Hapi from "@hapi/hapi";
import * as Bluebird from "bluebird";

function log(msg) {
  console.log((new Date()).toISOString() + " - " + msg);
}

async function init() {
  const server = Hapi.server({
    port: 3333,
    host: 'localhost'
  });
  server.route({
    method: 'GET',
    path:'/',
    handler: async (request, h) => {
      await Bluebird.delay(1000);
      return 'Hello World!';
    }
  });
  await server.start();
  log("HAPI Server running on " + server.info.uri);
};
init();
hassy commented 5 years ago

Thanks for a detailed report @kingjerod! That does sound like a bug.