testing: Ambiguous error message for wrong `-benchtime` value

[umi0410]

What version of Go are you using (go version)?

$ go version
go version go1.16.4 linux/amd64

I also checked go1.14 and go1.15 by docker image.

Does this issue reproduce with the latest release?

I guess. I tested with go 1.16.4 docker image(which was built 15 hours ago) and the ambiguous error message was reproduced.

What operating system and processor architecture are you using (go env)?

I think go env doesn't matter so I skip this question.

What did you do?

I tried to repeat benchmark for specific times. I saw there is a flag to set the number of repetition ago but didn't remember how to use it exactly so I just tried -benchtime 1 to execute the benchmark only once.

# main_test.go
package main

import "testing"

func BenchmarkDummy(b *testing.B) {
    for i := 0; i < b.N; i++ {    }
}

$ go test -bench=. -benchtime 1
go test -bench=. -benchtime 1
invalid value "1" for flag -test.benchtime: invalid duration
Usage of /tmp/go-build191827203/b001/me.test:
  -test.bench regexp
        run only benchmarks matching regexp
  -test.benchmem
        print memory allocations for benchmarks
  -test.benchtime d
        run each benchmark for duration d (default 1s)
  -test.blockprofile file
        write a goroutine blocking profile to file
  -test.blockprofilerate rate
        set blocking profile rate (see runtime.SetBlockProfileRate) (default 1)
  -test.count n
        run tests and benchmarks n times (default 1)
 ... // I left out other logs.
exit status 2
FAIL    me      0.001s

What did you expect to see?

I think the error message should let users know the iteration count feature of benchtime flag.

invalid value "1" for flag -test.benchtime or -benchtime: invalid duration or iteration count.
benchtime should be the duration (e.g. ms, s) or the iteration count(e.g. 1x, 100x).
Usage of /tmp/go-build191827203/b001/me.test:
...
  -test.benchtime d
        d can be duration or iteration count(default 1s).
        if d is duration, run each benchmark for duration d.
        else if d is iteration count, run each benchmark iterating for d.
...

What did you see instead?

I pasted the log above.

My additional explanation

The error log said "invalid value "1" for flag -test.benchtime: invalid duration". But benchtime flag can be either duration or iteration count. This ambiguity of benchtime flag has also been introduced in #32051 and robpike hadn't been aware of the feature of benchtime flag as iteration count.

The iteration count feature of benchtime flag was introduced in 1.12 but I think the error message didn't reflect on the update.

[umi0410]

You can refer to this codes. (golang/go/src/testing/benchmark.go#L28)

[gopherbot]

Change https://golang.org/cl/317916 mentions this issue: testing: Improve ambiguous error message about wrong value for-benchtimeflag.