Benchmark filter: wildcards on *nix CLI

[morgan-kn]

We have such instructions in documentation for the console filter:

• Run all benchmarks from System.Memory namespace: -f System.Memory*
• Run all benchmarks: -f *
• Run all benchmarks from ClassA and ClassB -f ClassA ClassB

It works fine for Windows.

But when I tried to run all benchmarks on my mac, I saw this output:

✘ morgan_kn@Irinas-MBP  ~/Sandbox/BenchmarkFilterTest/BenchmarkFilterTest  dotnet run -c Release -- -f *
The filter that you have provided returned 0 benchmarks.
Please remember that the filter is applied to full benchmark name: `namespace.typeName.methodName`.
Some examples of full names:
BenchmarkFilterTest.A.DoNothing
BenchmarkFilterTest.B.CreateString
BenchmarkFilterTest.MyAwesomeNamespace.MyAwesomeBenchmark.GetDouble
BenchmarkFilterTest.MyAwesomeNamespace.MyAwesomeBenchmark.GetInt
To learn more about filtering use `--help`

So, when we offer this command for a *nix CLI, it sees that we've used wildcards and so, before running the command it replaces the pattern with every file or directory that matches that pattern. Because of this we have a list of file names as a user unput instead of a wildcard. That's why it didn't work on my mac.

So, as a workaround for nix we can do this trick (and it works): `dotnet run -c Release -- -f '' `

I suggest to introduce new command line options to get rid of the wildcard, i.e.

dotnet run -c Release  -- -all // to run all benchmarks that user has
dotnet run -c Release -- -n MyNamespace // to run all benchmars from particular namespace
dotnet run -c Release -- -c MyClass 

@adamsitnik what do you think?

[adamsitnik]

hi @morgan-kn

big thanks for the bug report, I had no idea about the behavior of *nix CLI

I really like the new glob filtering and I definitely don't want to get rid of it.

What I can do is to change the output of --help for non-Windows OSes to include the solution you have provided.

I am OK with adding new commands to our console arguments. However, we need to be carefull when we introduce a single letter aliases:

1. each letter can be used only once
2. some letters are already used by dotnet cli, an example: -c Release is a very commonly used thing

@morgan-kn @AndreyAkinshin what do you think?

[morgan-kn]

1. I agree, however we can use words as parameters if we need. Like you did for join option.

2. Yeah, these letters can be used by dotnet cli, but as we use EnableDashDash from CommanLineParser, I don't think we'll get in troubles here. I reckon, dotnet run -c Release -- -c MyClass is ok to understand.
   By the way, maybe we should mention this ("--" option) in documentation. I'm afraid, people can think we've misspelled here and just doubled “--“ accidentally: dotnet run -c Release -- --runtimes clr core

@adamsitnik

[AndreyAkinshin]

@adamsitnik, I like the idea of custom arguments for namespace/class/etc. The new glob filtering is a very powerful API and it can be super useful for advanced use cases, but in most simple cases you don't need it.

[adamsitnik]

I have changed the way we display help, now for the examples with an asterisk we use '*'

If you believe that the new glob filtering is missing something please create new issue. I am doing a pre-release issue cleanup ;p

[morgan-kn]

@adamsitnik I don't think that your solution works because an asterisk will be replaced with the current directory files list before any BDN method is executed.

[AndreyAkinshin]

@adamsitnik, what do you think about one additional line about the quotes requirement on unix? I guess it makes sense to add it in the console output with examples and documentation.