runtime/coverage: coverage from other binaries is not collected in tests

FiloSottile commented 1 year ago

What version of Go are you using (`go version`)?

$ go version
go version go1.20.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes, with devel go1.21-91b8cc0d.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/filippo/Library/Caches/go-build"
GOENV="/Users/filippo/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/filippo/pkg/mod"
GONOPROXY="github.com/FiloSottile/*,filippo.io/*"
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/filippo"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/opt/homebrew/Cellar/go/1.20.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.4/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/Users/filippo/src/filippo.io/litetlog/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_j/hq4ytn1n4b94fhrpvvb9tktr0000gn/T/go-build1323402631=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Within a Go test, built a binary from a different package in the same module with -cover and executed it.

Ran the test with -coverpkg=./....

What did you expect to see?

Coverage of the command invocation included in the test summary.

What did you see instead?

coverage: 0.0% of statements in ./...

I expected this to work because there is explicit support for it in the testing package, which propagates the temporary GOCOVERDIR to the environment.

https://github.com/golang/go/blob/91b8cc0dfaae12af1a89e2b7ad3da10728883ee1/src/cmd/go/internal/test/test.go#L1326-L1330

Indeed, rogpeppe/go-internal#201 takes advantage of this to run the test binary as a subprocess and collect coverage from that invocation with the coverage of the main test run.

However, there is then code to exclude coverage from other binaries, introduced for #57924.

https://github.com/golang/go/blob/91b8cc0dfaae12af1a89e2b7ad3da10728883ee1/src/runtime/coverage/testsupport.go#L84-L96

I think running integration tests within a Go test is a more idiomatic way to do it than rigging a script with go tool covdata invocations, and it's nice that GOCOVERDIR gets already set up and propagated automatically, but the pattern then breaks when the coverage data is not collected.

As an aside, it should be documented somewhere how test coverage is propagated and collected.

seankhliao commented 1 year ago

cc @thanm

ankitjain1510 commented 1 year ago

@seankhliao @thanm My use case is that I need to generate the code-coverage from a containerised Go application running inside Kubernetes Pod after running the integration test. This application (server) is long running and doesn't terminate. I need a way to emit the coverage out after running my integration test.

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

FiloSottile commented 1 year ago

@ankitjain1510 That sounds unrelated to filtering of coverage collected by the testing package, so I suggest you open a separate issue for that, or ask on golang-nuts.

thanm commented 1 year ago

@ankitjain1510 collecting snapshots of coverage from server binaries is a bit different as @FiloSottile mentioned. We have APIs for doing this, but the documentation for them is not written/published (please bear with me on that). Thanks.

thanm commented 1 year ago

Hi @FiloSottile

Thanks for the report. It might be helpful for me to understand your high-level goals here in a bit more detail.

From your description it sounds like you have a module with that has a package P with tests, and in the test code you use "go build -cover" to build some sort of auxiliary binary (which might import P), then you would like to see the coverage effects of running the binary reflected in the coverage percentage numbers for P, something like that?

Is your main item of interest the coverage percentages or are you looking at text profiles as well?

It seems important to ensure that when a "-cover" test binary executes and writes out coverage stats or profiles, the raw material for that report comes only from running the test.

So for example, if you were to do something like

$ mkdir /tmp/xyz
$ go build -o myapp.exe -cover myapp
$ GOCOVERDIR=/tmp/xyz ./myapp.exe
$ GOCOVERDIR=/tmp/xyz go test -cover somePackage
...
$

it is important that the coverage numbers reported for "somePackage" don't incorporate any of the data files that might be in /tmp/xyz. These sorts of concerns are what motivated the testsupport.go code that you hlghlighted.

ankitjain1510 commented 1 year ago

@ankitjain1510 collecting snapshots of coverage from server binaries is a bit different as @FiloSottile mentioned. We have APIs for doing this, but the documentation for them is not written/published (please bear with me on that). Thanks.

Thanks for the response @thanm. Do we have any timeline available by when the documentation will be available?

enrichman commented 1 year ago

@seankhliao @thanm My use case is that I need to generate the code-coverage from a containerised Go application running inside Kubernetes Pod after running the integration test. This application (server) is long running and doesn't terminate. I need a way to emit the coverage out after running my integration test.

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

We have the same need, and at the moment we are using the same trick of pre-1.20 version. Our server is checking the GOCOVERDIR env var, if set we are adding a /exit route that will be used to quit the server.

Not sure if the new APIs will simplify this workflow but it will be neat.

FiloSottile commented 1 year ago

From your description it sounds like you have a module with that has a package P with tests, and in the test code you use "go build -cover" to build some sort of auxiliary binary (which might import P), then you would like to see the coverage effects of running the binary reflected in the coverage percentage numbers for P, something like that?

Is your main item of interest the coverage percentages or are you looking at text profiles as well?

Yep pretty much. I have testscript tests that build a set of commands from my module and use them all together in test scripts (for example, use foo-keygen to make a key and then run a server with that key and then use fooctl to inspec tthe database) and I would like to get aggregate coverage profiles.

I understand wanting to ignore spurious entries in GOCOVERDIR. Maybe that can be achieved by taking a snapshot of existing files before executing the test and then only aggregating new ones?

Anyway it's weirdly inconsistent that sub-invocations of the same test binary leads to profiles that get aggregated but sub-invocations of other binaries doesn't. Even invocations of the same test binary might be from separate test runs that shouldn't be aggregated, too.

sjuyal5 commented 1 year ago

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

@ankitjain1510 did you find solution to this problem?

ankitjain1510 commented 1 year ago

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

@ankitjain1510 did you find solution to this problem?

@sjuyal5 No. I am also stuck at the same place with the same issue. I started using the alternative tool i.e. https://github.com/qiniu/goc

sjuyal5 commented 1 year ago

Thanks for the prompt response Ankit.

Regards Savita Juyal

On Tue, 11 Jul 2023 at 2:41 PM, Ankit Jain @.***> wrote:

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

@ankitjain1510 https://github.com/ankitjain1510 did you find solution to this problem?

@sjuyal5 https://github.com/sjuyal5 No. I am also stuck at the same place with the same issue. I started using the alternative tool i.e. https://github.com/qiniu/goc

— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/60182#issuecomment-1630453047, or unsubscribe https://github.com/notifications/unsubscribe-auth/BADQKOZHISRONVQJCAOGVQTXPUKEPANCNFSM6AAAAAAYBBJITE . You are receiving this because you were mentioned.Message ID: @.***>

davidmdm commented 1 year ago

I have a very similar use case which led me to create an issue that was closed as duplicate https://github.com/golang/go/issues/63660

To sum it up, I wrote a package that handles context cancelation around signals. Signals cannot be tested as a unit test, and so I created a small testing acceptance binary in my tests that I can run, send signals to, and assert the correct output.

When I detect that a -test.gocoverdir flag is set because the go test command is being run in -cover , I build the acceptance binary with the -cover flag.

However, only the coverage of unit tests is reported using the go test command. If I however set the gocoverdir to some local non-temporary folder, I can see that all of the covdata has been emitted into it and that if I run the go tool covdata percent command that I have 100% coverage and everything is there.

This seems like a bug. I have noticed that go test -cover creates a new temporary directory for coverage on each run, and that the directory is removed before exiting. If that's the case, why aren't we taking into account all of the coverage that go test creates including the subprocesses that the tests execute? Why are we only reporting the covdata that go test -cover creates within its own process rather than all the coverage that was triggered by it?

As of today to get full coverage of you application you need to:

run go test with a local gocoverdir
- go test -cover -test.gocoverdir=./coverage ./...
view an incorrect report of your coverage
run covdata percentage yourself
- go tool covdata percent -i ./coverage
run cvdata to create your coverage profile
- go tool covdata textfmt -i ./coverage -o ./coverage.out
- go tool cover -html ./coverage.out

When really it should just work:

go test -coverprofile ./coverage.out
go tool cover -html ./coverage.out

mitar commented 12 months ago

I hit the same issue. In fact, I didn't know that GOCOVERDIR is magically set. So I was running my tests with:

GOCOVERDIR=coverage go test ./... -race -timeout 10m -cover -covermode atomic -args -test.gocoverdir=coverage

It took me quite some time to figure out why things do not work. Why when I then internally call go run from a test, coverage output does not end up in coverage dir. I checked and GOCOVERDIR was present. Only when I checked the value I realized something is strange.

So my proposal would be (and or or):

GOCOVERDIR should not be set automatically if it is already set.
All content from temporary gocovedir should be copied to original gocoverdir.

The issue is now that I even cannot manually set GOCOVERDIR because the value is lost because it is overridden with a temporary one.

hugelgupf commented 9 months ago

GOCOVERDIR should not be set automatically if it is already set.

All content from temporary gocovedir should be copied to original gocoverdir.

Or it should be merged into the cover profile provided with -coverprofile.

Anything but discarding my data would be acceptable... I just ran into this as well. Very frustrating as it took me a few days to figure out this is why my data was missing.

tmc commented 1 day ago

Can we get alignment on a preferred path here? I'd like to see a solution make it into 1.24.

golang / go