"Tools as dependencies": consider continuing to someone later using the repo and recreating the tool

[thepudds]

The "Tools as dependencies" example:

https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md

currently ends with committing and pushing the repo.

It could be helpful to continue to also show someone later using the repo to recreate the tool at the proper version. In other words, how to use a repo that has been set up with a tools.go.

[myitcv]

Yes, agreed. Many of the guides, this one included, end rather abruptly.

[atombender]

Is there a solution here that doesn't involve running go install on each individual dependency?

[myitcv]

@atombender - you can go run $main. Else you can use gobin -m -run $main

Is that the kind of thing you were after?

[atombender]

I didn't realize go run had that ability now — it used to be very limited about what you could do (e.g. go run cmd/main.go didn't see any other files the same package, so you typically had to do go run cmd/*.go). Thanks.

That suggests that installing binaries via $GOBIN isn't needed at all now. //go:generate, makefiles and so on can just use go run to run whatever tools they want. Right?

[atombender]

Well, other than performance. It appears that go run does not do any build caching:

$ time golangci-lint > /dev/null
golangci-lint > /dev/null  0.08s user 0.01s system 130% cpu 0.070 total
$ time go run github.com/golangci/golangci-lint/cmd/golangci-lint > /dev/null
go run github.com/golangci/golangci-lint/cmd/golangci-lint > /dev/null  2.09s user 1.01s system 179% cpu 1.731 total

[myitcv]

It appears that go run does not do any build caching:

Almost correct; the resulting binary is not cached. Hence it links every time (and that's the slow step).

Hence gobin (linked above) :)

https://github.com/myitcv/gobin/wiki/FAQ

FWIW, I use gobin -m -run everywhere, including in my //go:generate directives. No more worrying about PATH. (side note, if you do lots of code generation you might be interested in https://godoc.org/myitcv.io/cmd/gg; a dependency-aware, cached version of go generate)

There is a slight overhead for gobin -m -run (it effectively has to go list to find whether there is a binary for $mainpkg[@$version]). But this should reduce with the go list changes slated for Go 1.13.

FYI - the original issue where I asked about why go run does not cache the binary is https://github.com/golang/go/issues/25416

[atombender]

Looks like a nice tool, but my goal here is to reduce manual steps needed to work with a codebase. Having to tell people to go install extra stuff to work with the code is exactly what I'm trying to avoid. (You point out this problem in your FAQ, I see.)

We're currently using Docker for this, and it's actually working really well. The developer invokes a helper script and does something like ./run_in_docker golangci-lint or whatever, and the tool runs inside a Docker container that uses an image where Go and all the tools have been installed. No manual installation of anything, you don't even need Go, theoretically, just Docker. The nice thing about this is that it extends to other tools that aren't written in Go and can't be installed with go install.

Obviously this is an interim solution until something better comes along. I don't know if the Go team is working on something more permanent? For example, I'd love to be able to declare a tool dependency in a special section of go.mod and then run have a command, something like go mod tool golangci-lint, to run it.

[myitcv]

Looks like a nice tool, but my goal here is to reduce manual steps needed to work with a codebase.

Absolutely. The tool is, as the wiki explains, nothing more than an experiment to understand this space better.

I don't know if the Go team is working on something more permanent?

The current status is that https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md remains the best practice (reference https://github.com/golang/go/issues/25922#issuecomment-402918061). Working with that you either need to go run, go install or, and this is how the experiment came about, gobin -m -run.

In https://github.com/myitcv/gobin/issues/44 we explored other possibilities of tracking tool dependencies separately. But nothing has really stood out as "better".

On the most recent golang-tools (https://github.com/golang/go/wiki/golang-tools) call, @ianthehat mentioned he is going to put together a proposal on how go install should work with tools. @atombender, if I understand correctly you and I share the same preference for not go install-ing a tool, rather having a PATH-less way that it can be run without the link step.

I'll try to remember to post a link to Ian's proposal when it gets created, otherwise keep an eye out in the golang-tools call minutes.

[atombender]

Thanks for all that detail! It would be awesome if go run could actually cache the build output (I don't see why it couldn't), which would make it a general-purpose solution, but I'd also be happy to have a command to explicitly install tool deps.

[timbunce]

This topic came up recently in the Slack channel.

All that's needed was go install ./vendor/path/to/package

Another option noted in that thread was go build -mod=vendor -o ./bin/protoc-gen-go github.com/golang/protobuf/protoc-gen-go

[myitcv]

All that's needed was go install ./vendor/path/to/package

This is less than ideal because it's path-based and not package-based.

go build -mod=vendor -o ./bin/protoc-gen-go github.com/golang/protobuf/protoc-gen-go

This is better, but it uses go build and so will always be slower than go install.

What you probably want in this situation is:

GOBIN=$PWD/bin go install -mod=vendor github.com/golang/protobuf/protoc-gen-go

[tcard]

@atombender My somewhat hacky solution for this is is to put this in a file:

// +build generate

//go:generate bash -c "grep '^import _ ' tools.go | cut -b10- | GOBIN=$PWD/bin xargs go install"

package tools

Then you only need to run go generate, and your only implicit dependencies are standard Unix tools.

[samherrmann]

Another option for installing all tools defined in tools.go that appears to be working for me is the following:

go install $(go list -f "{{range .Imports}}{{.}} {{end}}" tools.go)