search

golang / mock

GoMock is a mocking framework for the Go programming language.
Apache License 2.0
9.3k stars 610 forks source link

race condition on go.mod with parallel mockgen calls #548

Closed sodul closed 3 years ago

sodul commented 3 years ago

We have a sizable monorepo and as part of our CI pipeline we run mockgen on all our targets.

Because this takes significant time we have been doing it in parallel and call 4 mockgen processes in parallel for over 100 targets. With mockgen 1.4.4 and Go 1.15 we never had any issues. When we upgraded to Go 1.16 we ran into this error almost consistently:

mkdir -p /go/src/project/test/mock/mylib
mockgen -destination /go/src/project/test/mock/mylib/mylibmock.go -package mylibmock project/mylib MyLib
[ ..snip.. ]
/go/src/project/test/mock/otherlib/otherlibmock.go:1:1: expected 'package', found 'EOF'
prog.go:14:2: package project/mylib is not in GOROOT (/usr/local/go/src/project/mylib)
prog.go:12:2: no required module provides package github.com/golang/mock/mockgen/model: go.mod file not found in current directory or any parent directory; see 'go help modules'
2021/04/03 01:20:52 Loading input failed: exit status 1
make: *** [mockgen.mk:250: /go/src/project/test/mock/mylib/mylibmock.go] Error 1

After upgrading to mockgen 1.5.0 the problem is 'almost' gone, but we are still seeing it several times per day.

We suspect that somewhere in the mockgen call go.mod gets rewritten, even if there is no change to it, and if one of the other parallel mockgen was about to read it, it does not find it and we end up with an error.

Note that for each mockgen call we also run goimports, so we are not 100% sure which process is making go.mod disappear, but it is always mockgen that breaks.

sodul commented 3 years ago

We were able to reproduce the go.mod file not found error while running mockgen sequentially, after only 50 calls.

In this case it turned out that our proto files had changed and I forgot to regenerate them in my local machine.

The error message was still:

prog.go:12:2: no required module provides package github.com/golang/mock/mockgen/model: go.mod file not found in current directory or any parent directory; see 'go help modules'

Maybe it is a red herring? I suspect that mockgen might be updated to give more relevant error messages to help troubleshoot the root cause.

sodul commented 3 years ago

A bit of an update. We confirmed that the issue was caused by calling mockgen in parallel where the sources and generated mock files overlap. Which mean that at some point one mockgen process is reading a mock file while it is being rewritten. The fix to not run mockgen within that same target group seems to work and we have not seen the issue since we made the changes.