Add go.benchFlags to extract benchmark flags from testFlags

[cristaloleg]

Is your feature request related to a problem? Please describe. Go 1.17 provides a new flag -shuffle which changes order of the tests to a random one. But this flags is bothersome for benchmarks. In other words "I don't care about order of benchmarks, I want the results". Also using this flag complicates benchmark comparison (due to order, hah!). Same problem with a -race flag, not so useful in benchmarks for most of the cases.

Describe the solution you'd like Have go.benchFlags parameter where user can specify benchmark only flags.

Describe alternatives you've considered Disabling -shuffle or -race flag for tests or just use terminal for benchmarks. 1st solution disable good Go test features. 2nd is just less simpler than click "run benchmark" (or so).

Additional context Nothing to add.

[hyangah]

This proposes a new go.benchFlags:

• go.buildFlags -> apply to build. apply to test if go.testFlags is not set. apply to benchmark if go.benchFlags is not set.
• go.testFlags -> apply to test. apply to bench if go.benchFlags is not set.
• go.benchFlags -> apply to bench.

Note: when a Benchmark function runs as part of test runs, only go.buildFlags or go.testFlags are used.

Agree that disabling -shuffle/-race unconditionally for tests or benchmarks are not ideal. There are cases where these flags make sense in tests and benchmarks (e.g. compare performance with/without -race flag).

I think it's reasonable. Does anyone have an objection or a different approach we should consider?

[cristaloleg]

No objections, it's even better, after your edits users will be able to configure build flags too. Great! Probably title can be also updated now, 'cause feature request covers other cases.

Edit: didn't know that buildFlags already exists 👀