cmd/go: support compiling all tests without running

bradfitz commented 8 years ago

I want this to work:

$ go test -c std cmd
cannot use -c flag with multiple packages

But I don't actually care about the binaries. I just want to test that all the tests can compile, and how fast. @randall77 also wants this for SSA coverage reasons.

cespare commented 8 years ago

Is go test -run xxxxx std cmd a reasonable workaround for at least some of those use cases?

josharian commented 8 years ago

And I want this, but I do want the test binary. I'd use it to compile tests, change the compiler, compile tests again, and then run before/after benchmarks. (All in a script, of course.) Like toolstash for std tests. I currently do something similar but with go list std and a lot of bookkeeping and go test -c invocations.

It seems to me we could generate a testmain that invokes all the tests in all the included packages. There are some questions: How do you know from the -v output which package this particular instance of TestReader is from? How does the -run filter work? And so on. My hand-waving answer is: Mimic the UI for subtests and sub benchmarks, which already have this nesting problem; the nesting is just up a level rather than down.

minux commented 8 years ago

we can't generate one binary for tests in multiple packages because there is no guarantee that package A's test doesn't interfere with package B's use of package A.

josharian commented 8 years ago

What if we leave that up to the user to diagnose? If package A's test interferes with package B's use of package A, then don't test them together. This sort of issue can already arise in a single package.

minux commented 8 years ago

how could the user know that one can't test package A and package B together?

consider "go test -c ./...", which could easily contain hundreds of packages.

josharian commented 8 years ago

They'll know when the test fails. If the tests pass when run independently (i.e. go test ./..., no -c), then there's clearly an interaction problem.

minux commented 8 years ago

Even if they could know the bad interaction between A & B (it's not that easy because given that B's tests failed, you still don't know which of B's dependent [both directly and indirectly] might be the cause. What's more, the behavior also depends on the order tests for different packages are run.) because cmd/go doesn't support negative patterns, there is no way to use ./... and exclude B.

Given a large enough GOPATH, it's very likely there are bad interactions between the package tests, and then basically the new feature is useless (in the worse case, the user has to name all the packages on the command line, and it's no better than status quo.)

We might be able to do this if multiple packages are compiled into a single package from the beginning, but I don't think we can add this feature now.

josharian commented 8 years ago

I can't imagine someone wanting to run go test -c ./... on their entire GOPATH unless they had a very good reason. I can, however, see this for a particular project or set of packages, in which case this becomes very useful. Think e.g. of cross-compiling all the tests for your project and having to only scp a single binary.

cmd/go doesn't support negative patterns

Yes, this is unfortunate.

I apologize, but I don't quite understand the resistance here. It seems like the resistance is "something could go wrong". But this doesn't really seem like a big footgun to me. And when this is helpful, like cross-compiling all tests for a big project or keeping around an old version for benchmarking or testing compilation speed, it is very helpful, and users will be motivated to make it work (or not use it). And it is entirely backwards compatible.

minux commented 8 years ago

I think this issue gives one reason for do go test -c ./... for the whole GOPATH: test the SSA compiler code generation.

The key point is most tests are not prepared for this. How many packages support go test -count=2? Even some of the std packages don't, until they're fixed not long ago.

And how do you propose to handle conflicting flags added by tests and multiple TestMains? Again, these features are just not designed to combine tests for multiple packages into a single executable.

Also note that tests for a package needs to run at the directory for the package, how to handle that when tests for multiple packages are combined into one binary?

(FTR, I have been thinking about this feature a long time ago, but my conclusion was and still is that the whole thing is not designed for this.)

josharian commented 8 years ago

How many packages support go test -count=2? Even some of the std packages don't, until they're fixed not long ago.

Exactly. And they got fixed because people cared. Same situation here, I'd say.

conflicting flags added by tests

Sorry, what do you mean by this?

multiple TestMains

Refuse to compile, with a lucid error message. Rare.

tests for a package needs to run at the directory for the package

os.Chdir? Note that the go command has the directory available at compilation time. This is a bit ugly for the cross-compilation case, but we could print a warning instead of failing when os.Chdir fails.

I have been thinking about this feature a long time ago

Yeah, I recall there being another issue about this but couldn't find it.

I agree that there are limitations, I just see them as limitations rather than showstoppers.

minux commented 8 years ago

By conflicting flags added by tests, consider two packages that both add -update flags to update golden output of their test data.

The code will compile, but will fail at runtime with a flag redefined panic.

There are just too many global states, hence I said the whole system is not designed for this.

mattklein123 commented 8 years ago

I'm new to Go, but FWIW, the lack of this feature is one of the most frustrating things that I have found so far about the tooling. I tend to write production code and test code at the same time, and during my normal dev cycle I want to see if things compile many more times than I want to run the tests. So right now I'm doing roughly the same thing as stated above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and have it be up to the user to deal with conflicts as suggested by @bradfitz

davecheney commented 8 years ago

If you want to compile but run not tests,

 go test -run=nope ./...

Replace nope with another pattern if you have test cases that match.

On Fri, 27 May 2016, 03:45 Matt Klein notifications@github.com wrote:

I'm new to Go, but FWIW, the lack of this feature is one of the most frustrating things that I have found so far about the tooling. I tend to write production code and test code at the same time, and during my normal dev cycle I want to see if things compile many more times than I want to run the tests. So right now I'm doing roughly the same thing as stated above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and have it be up to the user to deal with conflicts as suggested by @bradfitz https://github.com/bradfitz

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/15513#issuecomment-221943239

mattklein123 commented 8 years ago

@davecheney I'm probably doing something stupid but running that command still causes all the tests to execute. I tried:

go test -run nope ./...

Also still runs tests.

davecheney commented 8 years ago

Please try -run=nope

On Fri, 27 May 2016, 09:56 Matt Klein notifications@github.com wrote:

@davecheney https://github.com/davecheney I'm probably doing something stupid but running that command still causes all the tests to execute. I tried:

go test -run nope ./...

Also still runs tests.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/15513#issuecomment-222027132, or mute the thread https://github.com/notifications/unsubscribe/AAAcA8xNrwQtBDQEF3AeZEF-377nOLMUks5qFjMwgaJpZM4IV2e- .

mattklein123 commented 8 years ago

I tried both:

mklein@vm:~/Source/go/src/github.com/<redacted>$ go test -run=nope ./...
ok      github.com/<redacted>   0.025s

cespare commented 8 years ago

-run=nope and -run nope are equivalent.

@mattklein123 no tests are being run (probably). Use -v to confirm.

Please use the mailing list if you have further questions about go test.

davecheney commented 8 years ago

That is correct. The test binary is built and executed, but none of the test cases will be executed because they don't match the -run filter

On Fri, May 27, 2016 at 10:11 AM, Caleb Spare notifications@github.com wrote:

-run=nope and -run nope are equivalent.

@mattklein123 https://github.com/mattklein123 no tests are being run (probably). Use -v to confirm.

Please use the mailing list if you have further questions about go test.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/15513#issuecomment-222029023, or mute the thread https://github.com/notifications/unsubscribe/AAAcA8u95PM2MDjAGCytHBIVmp0py0RIks5qFjaxgaJpZM4IV2e- .

jurij commented 8 years ago

i prefer -run ^$

bradfitz commented 8 years ago

My motivation is for the builders, and test sharding.

The problem with go test -run=^$ std cmd is that it deletes the *.test binaries afterwards. I want the test binaries, to ship around the network and shard out isolated test execution. But I want to burn CPU as hard as possible where it's not latency sensitive. (That is, I can run compilation jobs at 100% CPU, but tests are often flaky at 100% if they sleep or depend on time.)

minux commented 8 years ago

@bradfitz, I think for your purpose, having multiple go test -c processes is better than figuring out where to store the compiled binaries when building multiple tests (the only safe way is to put the compiled test binary in their own package directories, but then you need to collect them. It's probably easier to just go test -c with explicitly output filename).

It won't do much more work either (assuming go install std cmd has passed.)

bradfitz commented 8 years ago

I can do it myself, but my concern is that I can't keep the CPU busy as well as the cmd/go binary could.

cmd/go can load the dependency graph once, come up with a plan, and keep a certain number of worker processes running at all times.

Any coordination I do trying to run multiple cmd/go binaries wouldn't be as good.

Definitely low priority, though.

rsc commented 7 years ago

There are at least three different things people might want here (and have asked for either here or other places):

Build all binaries and don't save them.
Build all binaries and do save them (maybe in a directory tree).
Compile but don't link (for quick turnaround in save hooks).

I propose we keep accumulating these kinds of go command issues and try to take a coherent look at them in the first half of next year.

I'll create a new GoCommand label. Feel free to label other (proposal or non-proposal) issues with that too.

Thanks.

gopherbot commented 7 years ago

CL https://golang.org/cl/42531 mentions this issue.

FlorianUekermann commented 6 years ago

It would be great if this enabled "buildall.bash" to also check if all tests compile. That would have saved me some embarrassment when submitting CLs.

shivakumargn commented 6 years ago

I have a large C based codebase which has Go wrappers. I would like to write Go test packages (only test *.go code) as test drivers for the C code, using the go wrappers, create go test binary and ship the binary with the installed software as a test suite for the environment. This is not a typical test suite in a dev machine. The structure of go source code and the godoc helps in organizing the test suite nicely out of the box.

titpetric commented 5 years ago

Just a note for future travellers:

if you're doing something like go test ./... it will continue with all package tests even if one of them fails to compile (they effectively run individually, as in O(N) not O(1)),
the flag -failfast doesn't seem to have any effect on this (1.12), all package tests will run,
note that the trick -run=^$ from above still runs ANY TestMain() which you may have - it's false to say that you are only compiling tests. This is especially dangerous with integration tests as you may populate an external service with data from TestMain,...

Overall: ouch

azr commented 5 years ago

Hi there, I think this could make multiplatform testing more simple/easy. (At least for me):

1/ have your ci in docker 2/ cross build your test binaries for the systems you care about. 3/ link & test

This should be fairly simply doable on circle-ci for example, since they recently added macos and windows support.

This way I don't have to install go for every OS I want to test upon.

I wouldn't mind that multiple binaries could be generated, if I can get a parsable list of generated binaries. In my case I think the argument of pkg.test A making pkg.test B fail is not an issue as it would be super easy to compartmentalise were these tests are being run.

And for now I will do this manually using that good old go list ./... 🙂

Cheers !

PS: Happy to contribute if need be !

earthelf commented 4 years ago

-run ^$ not works, -run=nope not perfect. I prefer -count=0

robert-zaremba commented 3 years ago

Any hope that we will be able to compile all tests and cache the results without running the tests?

donutloop commented 3 years ago

Any progress? I would like to have this feature. Thank you.

billinghamj commented 3 years ago

I would highlight again the desire to be able to build all tests into a single binary. If go test ./... works, it seems strange that it'd be impossible to make a single binary which does the same thing

ianlancetaylor commented 3 years ago

go test ./... works by building and running several different binaries. In particular, test binaries assume that they are run in the directory of the package being tested, so that they can refer directly to testdata files. That seems awkward if we build a single binary for multiple packages.

skipor commented 3 years ago

I have found pretty workaround. Idea is to use go test --exec flag. Here it's doc from go help test:

    -exec xprog
        Run the test binary using xprog. The behavior is the same as
        in 'go run'. See 'go help run' for details.

If pass --exec that does exit 0, than it works like build all binaries and don't save them.
If pass --exec that moves binary somewhere, than works like Build all binaries and do save them (maybe in a directory tree).

I have implement the workaround in simple script: https://github.com/skipor/go-test-compile It just works:

# Download some packages to test.
go get -d -t golang.org/x/tools/...
# Just check that all code building. Like `go build ./...` but also build test packages.
go-test-compile golang.org/x/tools/godoc/...  

# Compile all test binaries. # Like `go test -c ./...` it it were implemented.
go-test-compile -o /tmp/test-executables golang.org/x/tools/cmd/godoc golang.org/x/tools/godoc/...
tree /tmp/test-executables
#> /tmp/test-executables
#> └── golang.org
#>     └── x
#>         └── tools
#>             ├── cmd
#>             │   └── godoc.test
#>             ├── godoc
#>             │   ├── redirect.test
#>             │   ├── static.test
#>             │   ├── vfs
#>             │   │   ├── gatefs.test
#>             │   │   ├── mapfs.test
#>             │   │   └── zipfs.test
#>             │   └── vfs.test
#>             └── godoc.test

Of course, it is better to implement that features in go tooling, but workaround already works well.

P.S. To get compile but don't link (for quick turnaround in save hooks) same idea may be applied to go build --toolexec flag, that can be passed to go test too. Passing --toolexec, that writes exit 0 executable if link called and does exec else will give us desired behaviour.

benmoss commented 3 years ago

One problem (pointed out by @titpetric) with run=^$ and -count=0 is it still will run init and TestMain functions. Adapting @skipor's idea you can avoid that with:

go test -exec=echo ./...

gopherbot commented 3 years ago

Change https://golang.org/cl/319052 mentions this issue: cmd/go: allow go test -c on multiple packages

howardjohn commented 3 years ago

https://gist.github.com/howardjohn/c0f5d0bc293ef7d7fada533a2c9ffaf4 provides a small script to do this, inspired by some of the comments above. By skipping the linker we save a lot of time on warm builds (60s -> 3s)

ianlancetaylor commented 3 years ago

This has been on hold for 4 1/2 years for taking an overall look at the go command. I think that has happened, so taking off hold.

azr commented 3 years ago

The go embed feature and could be useful now. If tests were embedding things from testdata then a test binary would 'just work'. Now, I'm not sure if the solution should auto-embed things for tests.

leighmcculloch commented 3 years ago

Adding an ability to build all tests, similar to how we can build all code including commands with go build ./..., would be helpful in Go repos where building is fast and tests are slow, because it is beneficial to have CI separately build all code in a separate run than tests so that there's immediate feedback.

rsc commented 2 years ago

We already have go build -o \ writing multiple binaries to a directory. We also have go build -o /dev/null already special casing throwing the test binary away. And apparently that works for multiple binaries today already (see https://github.com/golang/go/issues/37378#issuecomment-591603974).

It seems like we should make go test -c -o /dev/null work for multiple tests, as well as go test -c -o \.

(On Windows /dev/null is spelled NUL in these commands; it's os.DevNull.)

/cc @bcmills

rsc commented 2 years ago

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

rsc commented 2 years ago

Does anyone object to https://github.com/golang/go/issues/15513#issuecomment-959775908 ?

AlekSi commented 2 years ago

I totally expected -count=0 to compile all tests without running them. If we can't change that behavior, can we at least document it better in go help testflag and other places? Maybe even issue a warning?

robert-zaremba commented 2 years ago

go build -o /dev/null

This is no where documented.

rsc commented 2 years ago

go build -o /dev/null doesn't really need documentation when writing a single file, since it's just what /dev/null always means on Unix. We should absolutely document that it works for multiple targets (once we make that true).

rsc commented 2 years ago

Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group

rsc commented 2 years ago

No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal. — rsc for the proposal review group

or-shachar commented 2 years ago

Regarding this:

We also have go build -o /dev/null already special casing throwing the test binary away. And apparently that works for multiple binaries today already (see #37378 (comment)).

It seems like we should make go test -c -o /dev/null work for multiple tests, as well as go test -c -o .

What if we are interested to take the test execution binaries and execute them in another process?

This can be beneficial if you have long running tests and very different resources footprint between compilation and test runtime. For instance - e2e tests.

It will allow us to compile all tests on a strong expensive machine and then run the tests on cheaper ones.

bcmills commented 1 year ago

@mknyszek and @prattmic pointed out that on Unix platforms go test -exec /bin/true does seem to support compiling the tests without running them today. 😅

golang / go

cmd/go: support compiling all tests without running #15513