Calling go/packages.Load during Bazel build

[amckinney]

I am writing a Go code generation tool that depends on go/packages for creating a types.Package from a particular import path.

I've written a custom Bazel rule that depends on a target go_library label so that I can pass the Go library's importpath to the executable. But given the default behavior of go/packages, the go list driver is selected and Bazel fails to resolve the import path (as expected):

couldn't exec 'go [list -e -json -compiled -test=false -export=true -deps=true
-find=false -- <$importpath>]': exec: "go": executable file not found in
$PATH *exec.Error

I've noticed that the go/packages.Load API defines a Config.Env parameter to override the Go packages driver, and looks for the GOPACKAGESDRIVER environment variable, but I'm not sure what implementation I need to provide in order to resolve the necessary package information that go/packages expects.

I realize that there are a couple of open issues that may or may not be related, most notably https://github.com/bazelbuild/rules_go/issues/1644 and recent conversations in https://github.com/bazelbuild/rules_go/issues/512. This issue is more targeted at seeking guidance on how to move forward in the absence of a sophisticated, upstream solution like that mentioned in https://github.com/bazelbuild/rules_go/issues/1644.

[jayconrod]

There are a couple of missing pieces needed to make this work correctly. Primarily, a proper GOPACKAGESDRIVER implementation is needed for Bazel actions. #1644 covers this.

In the absence of such a driver, you might be able to wrap the target in a go_path, and then run go list inside that. If you're defining your rule using go_rule and go_context, you can reference go.toolchain.go (the Go extectuable file) and go.toolchain.srcs (the standard library sources) as inputs. To set GOROOT, you can use go.toolchain.root_file.dirname.

Hope that helps.

[amckinney]

Thanks @jayconrod, I appreciate the advice! I'll give that a go and see if I can get something working in the interim. I realize that it's difficult to estimate these things, but is #1644 something the team is actively working on?

[jayconrod]

I'm afraid not. I'm currently working on rewriting a large portion of the execution logic, which will make #512 and #1644 more feasible. But I'm primarily focused on module support in the Go command for 1.13, and that's taking most of my time these days.

[amckinney]

Got it, I'll stay tuned and try to get the go_path approach working in the meantime. If there is any room for assistance following the execution logic rewrite, please let me know!

[simjesp]

@amckinney i'm stuck on this same obstacle trying to use go/packages in a custom tool through a bazel rule. Did you figure out a temporary workaround using go_path or otherwise that you can share?

[amckinney]

@simjesp We were able to get something working by first virtualizing a GOPATH with the go_path rule and adding that as an attribute to another go_rule. Our solution was very similar to the gomock implementation.

[robbertvanginkel]

Using a go_path rule as input to satisfy go/packges.Load was becoming a bottleneck for us doing codegen on large targets with a large set of dependencies. Although the packagesdriver work should enable generic go/packages based codegen in the future, we've also found another way to speedup codegen that's part of the build that works for us.

For some codegen, we now use golang.org/x/tools/go/gcexportdata to read go/types info. We provide the archive files based on the GoArchive providers of dependencies. Although the archives don't provide the exact same scope of data as packages.Load, our codegen tools were mostly types.Package based and that information is provided by both approaches. After modifying the codegenerators, we've seen significant improvements over the go_path+packages/Load approach. Some codegen steps went from 72 to 2s.