Having the option to run scenarios in sequential order rather than parallel

[Knanoco]

Currently, NBomber only allows for scenarios to be run in parallel order.

However there are some case scenarios where it'd be preferable to run sequentially such as in instances where we have scenarios with mostly the same code sans some database changes.

E.g. Both scenarios are testing the same concept, except require different database sets. Scenario A: Seed 1,000 rows in the DB Scenario B: Seed 5,000 rows in the DB Each scenario must exist in a clean database with no pre-existing data.

[AntyaDev]

Hi @Knanoco

Are you going to run them on CI/CD pipeline?

[Knanoco]

Hi @AntyaDev,

Yes we're planning to run them in the pipeline. For our case specifically, we want to run multiple scenarios in sequence and then have the results output into the same report.

Currently we have multiple tests for each scenario but that produces a lot of reports each time the pipeline is run due to the amount of tests.

[jarlef]

This would be a nice feature to have. If you want to load test different endpoints to see the max throughput without being affected by the amount of parallell scenarios running (stress testing) https://www.geeksforgeeks.org/difference-between-load-testing-and-stress-testing/

The only way around this is to run the scenarios in different sessions, but this will result in multiple reports

[AntyaDev]

Hi @jarlef ,

It looks to me that the main difference here is that you can't get an HTML report that contains all scenarios in one report with the ability to compare them in one place. Is'n it?

[jarlef]

@AntyaDev This could also solve the issue. The main thought is to run load test in the CI each weekend to verify the throughput on each endpoint in the api and compare them over time. When scenarios are run in parallell it will affect the overall load (stress testing).

Having a tool / api to combine the run sessions into one report would be useful. Then we can e.g run the tests using xunit with [assembly: CollectionBehavior(DisableTestParallelization = true)]. And afterward publish the report as a single artifact on the build

[AntyaDev]

Regarding your request, why not just use JSON Configuration or simply pass TargetScenarios? With this option, you can specify what scenarios with what settings should be executed.

I mean this API: NBomberRunner.WithTargetScenarios("scenario1")

or just use JSON Config: https://nbomber.com/docs/nbomber/json-config#json-config

[jarlef]

Wont this generate multiple reports?

NBomberRunner.RegisterScenarios(scenarios).WithTargetScenarios("scenario1").Run() NBomberRunner.RegisterScenarios(scenarios).WithTargetScenarios("scenario2").Run()

[AntyaDev]

Yes, it will generate a report per each run. But it's a different problem. Because to make running your scenarios sequentially will not solve the UX problems of HTML reports. Some code should be written to display it correctly, for example, comparing several scenarios.

[AntyaDev]

If you use NBomber in the organization I think this task can be prioritized.

[ph1l1pF]

We also have this problem. Our workaround is to run each scenario like this:

scenarios.ForEach(scenario =>
    NBomberRunner
        .RegisterScenarios(scenario)
        .WithWorkerPlugins(new HttpMetricsPlugin([HttpVersion.Version1]))
        .LoadConfig(
            runAsCluster
                ? "Properties/default-cluster-config.json"
                : "Properties/default-single-config.json"
        )
        .Run()
);

But we would prefer an option to run all scenarios sequentially by calling NBomberRunner only once. The reports can then be multiple html reports.

[AntyaDev]

Hi @ph1l1pF We have planned this feature for version 5.8.0. It's the next release. https://github.com/PragmaticFlow/NBomber/issues/643