search

PragmaticFlow / NBomber

Modern and flexible load testing framework for Pull and Push scenarios, designed to test any system regardless a protocol (HTTP/WebSockets/AMQP etc) or a semantic model (Pull/Push).
https://nbomber.com
Other
2.08k stars 129 forks source link

Request: Add simple way to set failure and latency thresholds in the config.json #707

Closed ctissellb closed 1 month ago

ctissellb commented 5 months ago

It would be quite handy to be able to use the config to set global or per scenario latency and failure percent thresholds rather than MaxFailCount.

For instance:

MaxFailPercent: 10 <-- Scenario failure rate must not exceed 10% MaxHighLatencyPercent: [10, 500] <-- The number of scenarios that take longer than 500ms to complete must not exceed 10%

AntyaDev commented 5 months ago

This is actually a very good suggestion. I will add it to the next release v5.8.0

AntyaDev commented 2 months ago

Hi @ctissellb What do you think about this API:

"GlobalSettings": {

        "ScenariosSettings": [
            {
                "ScenarioName": "scenario",

                "ThresholdSettings": [
                    { "StepName":  "global-info", "StatusCode": ["200", "Count => 20"] },

                    { "StepName":  "login", "Errors": "Percent > 20" },

                    { "StepName":  "logout", "Latency": "P50 >= 30" },

                    { "StepName":  "logout", "Latency": "Max >= 900" },

                    { "StepName":  "logout", "Request": "RPS >= 900" }                    
                ]
            }
        ]
}
ctissellb commented 2 months ago

Works for me!

AntyaDev commented 2 months ago

Hey @ctissellb Could you please try the latest beta: https://www.nuget.org/packages/NBomber/5.8.0-beta.3 Here you can find more info about Threshold feature

AntyaDev commented 1 month ago

https://www.nuget.org/packages/NBomber/5.8.0-beta.5