search

artilleryio / artillery

The complete load testing platform. Everything you need for production-grade load tests. Serverless & distributed. Load test with Playwright. Load test HTTP APIs, GraphQL, WebSocket, and more. Use any Node.js module.
https://www.artillery.io
Mozilla Public License 2.0
8.03k stars 510 forks source link

Metrics from "metrics-by-endpoint" plugin are not set to Artillery events.stats.customStats field #1431

Open savvagen opened 2 years ago

savvagen commented 2 years ago

I am using:

Artillery Core: 2.0.0-14
Node.js: v14.19.1

metrics-by-endpoint plugin is added to the config section.

plugins:
    metrics-by-endpoint:
        useOnlyRequestNames: false
    my-plugin: {}

I am receiving an output with all detailed metrcis:

--------------------------------------
Metrics for period to: 11:44:40(+0300) (width: 9.977s)
--------------------------------------

http.codes.200: ................................................................ 45
http.codes.201: ................................................................ 6
http.dns:
  min: ......................................................................... 0
  max: ......................................................................... 1
  median: ...................................................................... 0
  p95: ......................................................................... 1
  p99: ......................................................................... 1
http.request_rate: ............................................................. 5/sec
http.requests: ................................................................. 51
http.response_time:
  min: ......................................................................... 3
  max: ......................................................................... 22
  median: ...................................................................... 7
  p95: ......................................................................... 16.9
  p99: ......................................................................... 21.1
http.responses: ................................................................ 51
http.tcp:
  min: ......................................................................... 0
  max: ......................................................................... 1
  median: ...................................................................... 0
  p95: ......................................................................... 0
  p99: ......................................................................... 0
http.tls:
  min: ......................................................................... Infinity
  max: ......................................................................... -Infinity
  median: ...................................................................... 0
  p95: ......................................................................... 0
  p99: ......................................................................... 0
http.total:
  min: ......................................................................... 4
  max: ......................................................................... 27
  median: ...................................................................... 7
  p95: ......................................................................... 19.1
  p99: ......................................................................... 24.8
plugins.expect.ok: ............................................................. 97
plugins.expect.ok.hasProperty: ................................................. 46
plugins.expect.ok.statusCode: .................................................. 51
plugins.metrics-by-endpoint./comments (POST/comments).codes.201: ............... 4
plugins.metrics-by-endpoint./comments/28 (GET/posts/ID).codes.200: ............. 1
plugins.metrics-by-endpoint./comments/29 (GET/posts/ID).codes.200: ............. 2
plugins.metrics-by-endpoint./comments/3 (GET/posts/ID).codes.200: .............. 1
plugins.metrics-by-endpoint./comments/4 (GET/posts/ID).codes.200: .............. 2
plugins.metrics-by-endpoint./comments/6 (GET/posts/ID).codes.200: .............. 1
plugins.metrics-by-endpoint./comments/67 (GET/posts/ID).codes.200: ............. 2
plugins.metrics-by-endpoint./comments/8 (GET/posts/ID).codes.200: .............. 1
plugins.metrics-by-endpoint./comments/9 (GET/posts/ID).codes.200: .............. 1
plugins.metrics-by-endpoint./comments/97 (GET/posts/ID).codes.200: ............. 1
plugins.metrics-by-endpoint./get_token (GET/get_token).codes.200: .............. 1
plugins.metrics-by-endpoint./posts (GET/posts).codes.200: ...................... 4
plugins.metrics-by-endpoint./posts (POST/posts).codes.201: ..................... 1
plugins.metrics-by-endpoint./posts/1 (GET/posts/ID).codes.200: ................. 4
plugins.metrics-by-endpoint./posts/2 (GET/posts/ID).codes.200: ................. 4
plugins.metrics-by-endpoint./posts/3 (GET/posts/ID).codes.200: ................. 5
plugins.metrics-by-endpoint./posts/4 (GET/posts/ID).codes.200: ................. 6
plugins.metrics-by-endpoint./posts/5 (GET/posts/ID).codes.200: ................. 4
plugins.metrics-by-endpoint./posts/6 (GET/posts/ID).codes.200: ................. 5
plugins.metrics-by-endpoint./users (POST/users).codes.201: ..................... 1
plugins.metrics-by-endpoint.response_time./comments (POST/comments):
  min: ......................................................................... 9
  max: ......................................................................... 10
  median: ...................................................................... 10.1
  p95: ......................................................................... 10.1
  p99: ......................................................................... 10.1
plugins.metrics-by-endpoint.response_time./comments/28 (GET/posts/ID):
  min: ......................................................................... 4
  max: ......................................................................... 4
  median: ...................................................................... 4
  p95: ......................................................................... 4
  p99: ......................................................................... 4
plugins.metrics-by-endpoint.response_time./comments/29 (GET/posts/ID):
  min: ......................................................................... 6
  max: ......................................................................... 11
  median: ...................................................................... 6
  p95: ......................................................................... 6
  p99: ......................................................................... 6
plugins.metrics-by-endpoint.response_time./comments/3 (GET/posts/ID):
  min: ......................................................................... 17
  max: ......................................................................... 17
  median: ...................................................................... 16.9
  p95: ......................................................................... 16.9
  p99: ......................................................................... 16.9

When I am trying to print the stats from the plugin events customStats is empty, but I am expecting to get the detailed metrics like in artillery - v1.7.9

Plugin index.js


'use strict';

const debug = require('debug')('plugin:my-plugin');

module.exports = { Plugin: MyPlugin }

function MyPlugin(script, events) {
    this.script = script;
    this.events = events;
    debug("Initializing plugin:my-plugin")

    // Set event handlers
    debug('Setting event handlers...')
    events.on('stats', printStats)

    return this;
}

MyPlugin.prototype.cleanup = function(done) {
    debug('cleaning up');
    done(null);
};

function printStats(statsObject) {
    const stats = statsObject.report()
    debug("Stats\n" + JSON.stringify(stats, null, 2))
}

Finally I am receiving

{
  "customStats": {

  },
  "counters": {

  },
  "scenariosAvoided": 0,
  "timestamp": null,
  "scenariosCreated": 5,
  "scenariosCompleted": 5,
  "requestsCompleted": 51,
  "latency": {
    "min": 3,
    "max": 22,
    "count": 51,
    "p50": 7,
    "median": 7,
    "p75": 10.1,
    "p90": 15,
    "p95": 16.9,
    "p99": 21.1,
    "p999": 21.1
  },
  "rps": {
    "mean": 0,
    "count": 51
  },
  "scenarioDuration": {
    "min": 3114,
    "max": 6607.9,
    "count": 5,
    "p50": 3134.5,
    "median": 3134.5,
    "p75": 3134.5,
    "p90": 3134.5,
    "p95": 3134.5,
    "p99": 3134.5,
    "p999": 3134.5
  },
  "scenarioCounts": {
    "Post Reader Scenario": 3,
    "Post Writer Scenario": 2
  },
  "errors": {

  },
  "codes": {
    "200": 45,
    "201": 6
  }
}
hassy commented 2 years ago

hi @savvagen 👋

Artillery v2 removed customStats. All metrics (from Artillery itself and any custom metrics) are stored under counters/summaries/rates field of the stats object now.

So for example, this metric emitted by the metrics-by-endpoint plugin and is a counter:

plugins.metrics-by-endpoint./comments/3 (GET/posts/ID).codes.200

and will be available under statsObject.counters['plugins.metrics-by-endpoint./comments/3 (GET/posts/ID).codes.200']

hope that makes sense. lmk if that does not work or if you have any other questions!

savvagen commented 2 years ago

@hassy Hi. Solution like: console.log(JSON.stringify(statsObject.counters['plugins.metrics-by-endpoint./posts (GET/posts).codes.200'])) Is not working:

TypeError: Cannot read properties of undefined (reading 'plugins.metrics-by-endpoint./posts (GET/posts).codes.200')

Cause as I mentioned:

 "customStats": {

  },
  "counters": {

  },

these fields are returned as empty from the report() function