Provide a method of running only a subset of scenarios

[kkovaacs]

We have a use-case where a load-test contains a lot of scenarios. Occasionally we would like to run only some (one or more) of those scenarios and it would be great if this could be accomplished without commenting out the rest and recompiling the load test program.

One can already list the available scenarios/transactions with the -l command line argument. Could goose maybe provide a way to run only a subset of the scenarios? For example, it could accept an optional list of scenario names on the command line and just run those if specified?

[jeremyandrews]

I would also find this interesting/useful. It would be important that the ASCII and HTML reports properly reflect if a subset of Scenarios were enabled for a given load test.

So to enable a subset of the Scenarios from the Umami example (for example) perhaps you'd simply do something like:

---scenarios "Anonymous Spanish user, Admin user "

...which would effectively disable the "Anonymous English user" Scenario. Is this the sort of functionality you're looking for?

It would still respect weighting etc of any scenarios enabled, but in this way you could select a subset of scenarios.

[jeremyandrews]

Referring to the -l option you mentioned, here's the current output for the umami example:

% cargo run --release --example umami -- -l 
    Finished release [optimized] target(s) in 0.11s
     Running `target/release/examples/umami -l`
Available transactions:
 - Anonymous English user (weight: 40)
    o anon / (weight: 2)
    o anon /en/basicpage (weight: 1)
    o anon /en/articles/ (weight: 1)
    o anon /en/articles/% (weight: 2)
    o anon /en/recipes/ (weight: 1)
    o anon /en/recipes/% (weight: 4)
    o anon /node/%nid (weight: 1)
    o anon /en term (weight: 2)
    o anon /en/search (weight: 1)
    o anon /en/contact (weight: 1)
 - Anonymous Spanish user (weight: 9)
    o anon /es/ (weight: 2)
    o anon /es/basicpage (weight: 1)
    o anon /es/articles/ (weight: 1)
    o anon /es/articles/% (weight: 2)
    o anon /es/recipes/ (weight: 1)
    o anon /es/recipes/% (weight: 4)
    o anon /es term (weight: 2)
    o anon /es/search (weight: 1)
    o anon /es/contact (weight: 1)
 - Admin user (weight: 1)
    o auth /en/user/login (weight: 1)
    o auth / (weight: 2)
    o auth /en/articles/ (weight: 1)
    o auth /en/node/%/edit (weight: 2)

Another option may be to modify that output, and number the scenarios, so it may look something like this:

Available transactions:
 1) Anonymous English user (weight: 40)
    o anon / (weight: 2)
    o anon /en/basicpage (weight: 1)
    o anon /en/articles/ (weight: 1)
    o anon /en/articles/% (weight: 2)
    o anon /en/recipes/ (weight: 1)
    o anon /en/recipes/% (weight: 4)
    o anon /node/%nid (weight: 1)
    o anon /en term (weight: 2)
    o anon /en/search (weight: 1)
    o anon /en/contact (weight: 1)
 2) Anonymous Spanish user (weight: 9)
    o anon /es/ (weight: 2)
    o anon /es/basicpage (weight: 1)
    o anon /es/articles/ (weight: 1)
    o anon /es/articles/% (weight: 2)
    o anon /es/recipes/ (weight: 1)
    o anon /es/recipes/% (weight: 4)
    o anon /es term (weight: 2)
    o anon /es/search (weight: 1)
    o anon /es/contact (weight: 1)
 3) Admin user (weight: 1)
    o auth /en/user/login (weight: 1)
    o auth / (weight: 2)
    o auth /en/articles/ (weight: 1)
    o auth /en/node/%/edit (weight: 2)

Then the earlier command could be simplified to something like:

--scenarios 2,3

[kkovaacs]

Yes, something along the lines of --scenarios would be great for our use-case.

I'd prefer using "stable" names for that option though (likely the name of the scenario from the code?), so that adding a new scenario wouldn't break scripts using --scenarios.

[kkovaacs]

Although thinking more about this, I don't think simply comma-separating scenario names would work reliably (since scenario names themselves can contain commas).

Something like --scenario "Anonymous English user" --scenario "Anonymous Spanish user" would maybe work more reliably?

[visig9]

I think make selection in scenario level is a wrong approach. Because currently weight is associated with scenario directly:

For example, Maybe I have following Scenario :

Scenarios:
 a) Anonymous English user (weight: 4)
 b) Anonymous Spanish user (weight: 2)
 c) Admin user (weight: 1)

But weight may not always the same in all situation I want to test. Consider some time (due to timezone, for example), weight may change to:

Scenarios:
 a) Anonymous English user (weight: 2)
 b) Anonymous Spanish user (weight: 2)
 c) Admin user (weight: 0 = not used)

Select by --scenario can not fit this use case. I think we should add an extra level on top of scenarios which may call, say, Assumption? Let we pre-configure all assumptions in code, like:

Available Assumptions: 
1. 20:00 at UTC+2
   a) Anonymous English user (weight: 2)
   b) Anonymous Spanish user (weight: 2)
2. 20:00 at UTC-4
   a) Anonymous English user (weight: 4)
   b) Anonymous Spanish user (weight: 2)
   c) Admin user (weight: 1)

And choice one assumption at runtime:

--assumption "20:00 at UTC-4"
# or
--assumption 2