mmcloughlin / goperf

Continuous Benchmarking for the Go compiler
BSD 3-Clause "New" or "Revised" License
8 stars 0 forks source link

suites: standard library #37

Closed mmcloughlin closed 4 years ago

mmcloughlin commented 4 years ago

Add support for testing the special-case std module.

mmcloughlin commented 4 years ago

Test errors in the sandbox:

--- FAIL: TestImportDirNotExist (0.01s)
    --- FAIL: TestImportDirNotExist/GO111MODULE=on (0.00s)
        build_test.go:362: Import(full, 0) got error: "go/build: go list go/build/doesnotexist: exec: \"go\": executable file not found in $PATH\n\n", want "cannot find package" or "is not in GOROOT" error
        build_test.go:362: Import(full, FindOnly) got error: "go/build: go list go/build/doesnotexist: exec: \"go\": executable file not found in $PATH\n\n", want "cannot find package" or "is not in GOROOT" error
--- FAIL: TestImportPackageOutsideModule (0.00s)
    build_test.go:486: error when importing package when no go.mod is present: got "go/build: go list example.com/p: exec: \"go\": executable file not found in $PATH\n\n"; want "cannot find module providing package"
FAIL
exit status 1
FAIL    go/build        0.095s

Resolved by ensuring go binary is in the PATH for the benchmark workspace.

--- FAIL: TestCheck (0.17s)
    check_test.go:238: testdata/issues.src:9:6: no error expected: "could not import cmd/compile/internal/syntax (can't find import: \"cmd/compile/internal/syntax\")"
    check_test.go:293: --- issues: 2 source positions with expected (but not reported) errors:
    check_test.go:296: testdata/issues.src:350:20: "cannot convert 0 \\(untyped int constant\\) to \\*syntax.File"
    check_test.go:296: testdata/issues.src:325:4: "f.foo undefined \\(type \\*syntax.File has no field or method foo\\)"
FAIL
exit status 1
FAIL    go/types        0.619s

This one actually happens in my regular go environment: go test -short std. Seems highly unlikely that this is an actual bug in Go, but I can't yet see what I'm doing wrong.

mmcloughlin commented 4 years ago

Still unclear on how to fix the go/types.TestCheck tests, so I'm just going to skip tests for std. There is a failing benchmark though:

--- FAIL: BenchmarkExecHostname
    bench_test.go:15: could not find hostname: exec: "hostname": executable file not found in $PATH
FAIL
mmcloughlin commented 4 years ago

7c73c59c02fcab39da2520ab21d3c49115297a0f made various changes to support running standard library benchmarks and ingesting results.

mmcloughlin commented 4 years ago

It's working, on amd64 and arm64

Screenshot from 2020-05-02 19-41-07