tag1consulting / goose

Load testing framework, inspired by Locust
https://tag1.com/goose
Apache License 2.0
791 stars 70 forks source link

How to parametrize? #479

Closed dpc closed 2 years ago

dpc commented 2 years ago

I want to write a load tester that will test very synthetic loads, and I need it to happen in different configurations. Eg. request A + request B, then request B + request C, then A, B, C combined. Stuff like that. Different combinations, different weights.

I might have missed something but it looks like by default goose will run all the defined scenarios mixed together (possibly by weight, etc.) executed by N users. So I don't really see a way to run workload X first, then workload Y, then Z.

I would be OK with running my binary multiple times, each time with some different arguments. However I'm not sure how to best glue it with command line argument parsing already handled by goose. Ideally I'd like to add --workload X command line argument or something like that.

I guess I could use environment variable, and goose will just ignore it, but it's not a best UX: discoverability will be rather poor.

jeremyandrews commented 2 years ago

Would the request in https://github.com/tag1consulting/goose/issues/472 solve what you're asking for? I believe it solves the same ask. (It's not yet implemented, but coming soon.)

dpc commented 2 years ago
fn get_opts(args: &[String]) -> color_eyre::Result<(Opts, GooseConfiguration)> {
    let opts = opts::Opts::parse_args_default(&args[1..])?;
    let goose_opts = GooseConfiguration::parse_args(&opts.free, ParsingStyle::default())?;

    if !opts.help_requested() && !opts.help_goose && !(opts.mainnet ^ opts.testnet) {
        bail!("{}: Must set one of: `--mainnet` or `--testnet`", args[0]);
    }

    Ok((opts, goose_opts))
}

#[tokio::main]
async fn main() -> color_eyre::Result<()> {
    let args = std::env::args().collect::<Vec<_>>();
    let (opts, goose_opts) = match get_opts(&args) {
        Err(e) => {
            eprintln!("{}: {}", args[0], e);
            std::process::exit(2);
        }
        Ok(opts) => opts,
    };

    if opts.help_requested() {
        eprintln!("Usage: {} [OPTIONS]", args[0]);
        eprintln!();
        eprintln!("{}", opts.self_usage());
        return Ok(());
    }

    if opts.help_goose {
        eprintln!("Usage: {} [OPTIONS]", args[0]);
        eprintln!();
        eprintln!("{}", goose_opts.self_usage());
        return Ok(());
    }
...
/// Load testing tool for some custom stuff
///
/// Built using https://github.com/tag1consulting/goose/
/// Make sure to check `--help-goose` for test runner options.
#[derive(Debug, Options)]
pub struct Opts {
    /// MY CUSTOM OPTIONS HERE

    #[options(free, help = "Options to the Goose load test runner")]
    pub free: Vec<String>,

    pub help: bool,

    #[options(help = "Show help for Goose runner")]
    pub help_goose: bool,
}

That's the pattern I used so far. Let me check the #472

dpc commented 2 years ago

I actually already added options to turn off and on certain scenarios, so having it somehow built in (e.g. via regex etc.) would save me some work. But I also have bunch of very weird custom options, so I will still need the extra stuff.

Hover I'm quite satisfied with my workaround.

dpc commented 2 years ago

This is a first time I'm using gumdrop and is irritating me with certain shortcomings (like that it doesn't handle --some-flag=false on bool arguments. I miss clap. :)

jeremyandrews commented 2 years ago

We used clap early on, but it had some issues that caused me to remove it (alphabetized help screen that weren't logically grouped, large dependency list, and slow compilation times) -- I believe all of these issues have since been resolved, I'd definitely consider a PR that replaces gumpdrop with Clap again. (I mostly miss the "did you mean" type features)