myitcv commented 6 years ago

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

go version go1.10.3 linux/amd64
vgo commit 22e23900224f03be49670113d5781e4d89090f45

Does this issue reproduce with the latest release?

Yes; and latest vgo commit (per above)

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN="/tmp/tmp.VQw1O3x8Wy/hello/bin"
GOCACHE="/home/myitcv/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/tmp.VQw1O3x8Wy"
GORACE=""
GOROOT="/home/myitcv/gos"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build414355570=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Off the back of https://github.com/golang/go/issues/25624#issuecomment-395556484, I'd like to confirm that the following represents the "best practice" advice when adding and installing tool dependencies:

cd `mktemp -d`
export GOPATH=$PWD

mkdir hello
cd hello
vgo mod -init -module example.com/hello

# this could be anywhere but for convenience...
export GOBIN=$PWD/bin

# add a dependency on golang.org/x/tools/cmd/stringer
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

vgo install golang.org/x/tools/cmd/stringer

The go.mod and .Target for stringer look fine:

$ cat go.mod
module example.com/hello

require golang.org/x/tools v0.0.0-20180615195736-465e6f399236
$ vgo list -f "{{.Target}}" golang.org/x/tools/cmd/stringer
/tmp/tmp.VQw1O3x8Wy/hello/bin/stringer

The issue however is that running vgo mod -sync then removes our module requirement on golang.org/x/tools - I suspect this is a bug:

$ vgo mod -json
{
        "Module": {
                "Path": "example.com/hello",
                "Version": ""
        },
        "Require": [
                {
                        "Path": "golang.org/x/tools",
                        "Version": "v0.0.0-20180615195736-465e6f399236"
                }
        ],
        "Exclude": null,
        "Replace": null
}
$ vgo mod -sync
warning: "ALL" matched no packages
$ vgo mod -json
{
        "Module": {
                "Path": "example.com/hello",
                "Version": ""
        },
        "Require": null,
        "Exclude": null,
        "Replace": null
}

If we assume this is a bug and ignore it for now, I also wonder whether we can improve this workflow for adding tool dependencies somehow. The following steps feel a bit "boilerplate" and unnecessary:

define a fake tools.go file
add a dependency on the tool (which will never compile because it's a main package, so I'm not sure we can ever safely verify tools.go is "good"?)
set GOBIN to an appropriate location
vgo install tool
ensure PATH includes GOBIN

I wonder whether we could in fact obviate all of this by having something like:

vgo install -tool golang.org/x/tools/cmd/stringer
vgo run golang.org/x/tools/cmd/stringer

Thoughts?

vgo run tool is possible as a result of https://github.com/golang/go/issues/22726, but because of https://github.com/golang/go/issues/25416 it effectively requires a link step each time.

What did you expect to see?

With respect to what I think is a bug with vgo mod -sync

go.mod unchanged by the vgo mod -sync

What did you see instead?

The golang.org/x/tools requirement removed.

/cc @rsc @bcmills

tschaub commented 4 years ago

Thanks, @thepudds. That is a nice example demonstrating what the original author might do. But unless I'm mistaken, it doesn't cover how someone else working on the same project might get the same.

That is, after the original author creates the module,go installs dependencies, and alters their path, what steps should someone follow after git clone if they wanted go generate to produce the same?

bcomnes commented 4 years ago

@tschaub there is no standard way to do that with go, however there is an additional tool you can introduce to sort of get that outcome.

Its called gobin and its use is described in https://github.com/go-modules-by-example/index/tree/master/017_using_gobin

This makes the minimum common set of dependencies for your team simply go + gobin, and gobin's installation can be automated with a makefile.

Here is an example I set up on the netlify open-api repo when I worked there:

https://github.com/netlify/open-api/blob/813b6ad88723e32e8c6760c0a28dfd7a254d2199/Makefile#L13-L14

and the accompanying generate file:

https://github.com/netlify/open-api/blob/813b6ad88723e32e8c6760c0a28dfd7a254d2199/generate.go#L3

This also takes advantage of the tools.go pattern.

I would love to see this practice standardized however it doesn't look like it will happen anytime soon.

coolaj86 commented 4 years ago

Since this is showing up in my searches when I'm trying to find the other reference that I made, I'm going to add it here as well:

If you get an error

I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:

tools/tools.go:6:5: import "git.rootprojects.org/root/go-gitver" is a program, not an importable package

(go-gitver is the thing I'm trying to add)

I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:

Using a "tools" package

I made a tools directory:

mkdir -p tools

I put the tools package inside of it (as mentioned above):

// +build tools

package tools

import (
    _ "git.rootprojects.org/root/go-gitver"
)

Note that the tag is mostly not important. You could use foo:

// +build foo

However, you cannot use ignore. That's a special predefined tag.

// +build ignore

// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies

Updating go.mod

The best way is probably to run go mod tidy:

go mod tidy

However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:

go install git.rootprojects.org/root/go-gitver # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor

Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.

vendoring

In order to actually take advantage of the modules cache in a project you need to copy-in the source code

go mod vendor

That will grab all dependencies from go.mod

You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.

go fmt -mod=vendor ./...  # this needs a fix that is scheduled for go1.15 I believe
go generate -mod=vendor ./...
go build -mod=vendor ./...

That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)

//go:generate go run -mod=vendor git.rootprojects.org/root/go-gitver --fail
package main

// ...

Reference

From https://stackoverflow.com/a/54028731/151312

tschaub commented 4 years ago

Here is a modification to the best-practices example that makes things reproducible (without additional installs or path modifications) on Go 1.13.8: https://gist.github.com/tschaub/66f5feb20ae1b5166e9fe928c5cba5e4

tl;dr - use go run golang.org/x/tools/cmd/stringer instead of stringer in your go:generate comment.

Update: I see now that this same suggestion appears above (https://github.com/golang/go/issues/25922#issuecomment-398792589)

coolaj86 commented 4 years ago

@tv42 What needs improvement in my example? As far as I can tell I've collected the best information from various threads and combined them into a single set of "just works" instructions, but I'd love to continue to improve it.

ags799 commented 4 years ago

Really like the

go run tool@version

idea.

But on Go 1.14, only go get supports the @version syntax.

Is there a way run or install a specific version of a Go executable in Go 1.14?

tooolbox commented 4 years ago

@ags799 use https://github.com/myitcv/gobin

bwplotka commented 4 years ago

Hi Awesome Go Community! :wave:

Thanks for this thread @myitcv and everyone else who contributed to this discussion. Maybe it's not properly appreciated but a huge amount of people traverse through those threads carefully every day to find answers, so big kudos to you all. Especially this thread, for us, was extremely insightful and gave us many pointers in the space of tools versioning.

This being said, we are maintaining several big(-ish) Go projects in the open-source (e.g Prometheus, Thanos, prometheus-operator and more) and we were looking for a solid solution to this problem for a few months already. While there are good experiments, we were looking for something we can rely on now and long-term.

This is why we created the open-source CLI called bingo, that (hopefully) makes it completely seamless. :muscle:

So, If you are looking for some smooth experience for versioning Go tools on top Go Modules for any project (even non Go projects that just use Go tools!) I would recommend taking a look and trying out! :hugs: Feel free to use and please contribute & give us feedback! We started to use it recently in many projects we maintain e.g Thanos and so far it works quite well... :rocket: Hopefully we can maintain it together and improve on the way.

After all, even if Go Team will have an answer to this problem someday (I am pretty sure they will!), I think it's only helpful if the community can try to solve this problems much earlier on top of existing tooling (Go Modules). I think it might be good experience for Go Team as well to tell what requirements matter here. :hugs:

I also wrote some detailed blog post on the way about this problem space and a little bit on how bingo was created: https://www.bwplotka.dev/2020/bingo/ Feedback welcome on this as well (e.g by creating issue here)

jpreese commented 4 years ago

@bwplotka we're just now starting to explore this space as mismatched versions of tooling is causing some pain points. Bingo looks interesting, but what prevents it from running on Windows? That would be a non-starter for us, unfortunately.

bwplotka commented 4 years ago

We just don't use windows, so not yet tackled, that's it (: There are huge differences in how the path, shell (PowerShell!) and $PATH works, etc. Otherwise, there is no blocker TBH (: I added an issue on bingo to track this: https://github.com/bwplotka/bingo/issues/26 it should be actually quite a quick fix to make it work if you guys want to help and contribute :muscle:

arvenil commented 4 years ago

I'm curious why tools.go with // +build tools? Wouldn't something like tools_test.go work as well?

AlekSi commented 4 years ago

Hi Kamil :) https://github.com/golang/go/issues/25922#issuecomment-414677877 To avoid init() side-effects

arvenil commented 4 years ago

@AlekSi Hi :) I was asking to use tools_test.go instead of tools.go. init() side-effects still apply but just for tests so 🤷 ... however I realized go test would pick it up and that would be much more annoying and not working:) Thanks for explanation.

gopherbot commented 4 years ago

Change https://golang.org/cl/261499 mentions this issue: internal/tools: add a dummy package that imports mkwinsyscall

cristaloleg commented 3 years ago

In one of the projects we had a separate go.tools.mod file where we've defined our tooling dependencies.

Installation is simple but with 1 additional param, ex: go get -modfile=go.tools.mod github.com/foo/bar.

Example from open source https://github.com/mattermost/mattermost-server/blob/master/Makefile#L573 and https://github.com/mattermost/mattermost-server/blob/master/go.tools.mod (kindly ping @agnivade as one of the authors and Go contributor)

agnivade commented 3 years ago

Hey there, I'm not exactly sure what is it you wanted me to do?

bwplotka commented 3 years ago

As mentioned above, @cristaloleg even separate go.tools.mod will not work for most of the cases.

I wrote about this in detail here but TL;DR on using go.tools.mod:

You don't fight with dependency conflicts between tools and your main project module, sure, but you still fight with dependency conflicts between multiple tools you pin in this go.tools.mod (unless you have one tool)
You cannot install automatically two tools with the same binary name.
You cannot pin two binaries from the same module under different version (go.mod does NOT support per package versioning)
In fact you cannot even pin just a single binary/tool for a huge module that has multiple of them.
If the tool you pin comes from the bigger module (e.g see prometheus mod) with multiple replace / exclude manual hackery (which is required for "broken" modules) - you are stuck. You can't pin such a tool or you need to manually fight a lot.
go.tools.mod requires go.mod to be in repo (!), so you cannot pin go tools in non-go projects.
Management is painful - you have to reference the tool via full path instead of just e.g tool name.

And there is more 🤗 but hope it's enough to motivate the bingo existence. Bingo solves all those painpoints. We recently released a new version v0.3.0 so you are more than welcome to try it out! I have a full-time job, and other much bigger open-source projects to maintain, so please use, contribute and help us to make this even better.

... and hopefully, this tool will inspire Go native tools for a similar user experience (happy to contribute anything!) 🤗

mewmew commented 3 years ago

re: @rsc in https://github.com/golang/go/issues/25922#issuecomment-413898264

Best practice remains creating the tools.go file. It's true that go mod init does not auto-create a tools.go from dep's config, but I think doing so is getting a bit beyond scope.

The best practice described currently triggers a go vet warning (see e.g. https://github.com/llir/llvm/issues/196)

From llir/llvm/ir/enum/tools.go:


//go:build tools

package enum

import (
    _ "golang.org/x/tools/cmd/stringer"
)

Running go vet -tags tools github.com/llir/llvm/ir/enum results in the following warning:

ir/enum/tools.go:6:2: import "golang.org/x/tools/cmd/stringer" is a program, not an importable package

Is using a tools.go file with a go:build tools build tag still the recommended best practice for handling tools dependencies? Or is there another approach that is now recommended?

Cheers, Robin

jayconrod commented 3 years ago

@mewmew In this case, don't run go vet with -tags tools. The tools.go file is not meant to be part of a buildable package, so vet won't provide useful feedback on it. Without -tags tools, everything except go mod tidy will ignore tools.go.

andig commented 2 years ago

I've moved to installing tools.go dependencies using make with the power of go list:

install:
    go install $$(go list -f '{{join .Imports " "}}' tools.go)

I've not looked into this, but I feel //go:build tools should become obsolete doing so.

SamWhited commented 2 years ago

This will actually install to the users home directory somewhere, polluting the machine outside the repo and is generally a bad practice regardless of the build system being used (and especially with Make where "install" is a phony target and the outputs can't be kept track of).

The benefit of using a tools.go and then running the deps with go run is that you always wind up using the correct version and don't have to do the install every single time (once the temporary files are downloaded and properly versioned/hashed Go is happy to skip fetching them). This should work fine with a properly written Makefile because tools.go will be a dependency of whatever your final build target is, eg. if you do something like this:

GOFILES!=find . -name '*.go' tool_generated_file: tools.go go run sometool mybinary: tool_generated_file $(GOFILES) go build -o $@

Then when the tools.go file changes the tool will be re-run and your files regenerated.

—Sam

On Thu, Oct 14, 2021, at 08:40, andig wrote:

I've moved to installing tools.go dependencies using make with the power of go list:
  install:        go install $$(go list -f '{{join .Imports " "}}'
  tools.go)
I've not looked into this, but I feel //go:build tools should become obsolete doing so.

-- Sam Whited

egonk commented 2 years ago

@rsc

Best practice remains creating the tools.go file. It's true that go mod init does not auto-create a tools.go from dep's config, but I think doing so is getting a bit beyond scope.

Another data point: I'm urgently porting an older large project from git submodules to go modules and tools.go approach breaks go list -deps, which I'm using as a precise audit step together with a custom GOPROXY with a more relaxed general allow list:

// +build tools

package x

import (
    _ "golang.org/x/tools/cmd/stringer"
)

C:\Users\egon\Desktop\x>go list -deps -tags tools -f "{{with .Module}}{{.}}{{end}}"
tools.go:6:2: import "golang.org/x/tools/cmd/stringer" is a program, not an importable package
golang.org/x/sys v0.0.0-20211019181941-9d821ace8654
golang.org/x/tools v0.1.8
...

I can get rid of the error by using -e, but I prefer hard errors. An alternative is to check the final binaries with go version -m or maybe complicate the GOPROXY infrastructure with more precise rules. Not a smooth experience...

@bcmills

antichris commented 2 years ago

I think I'll just summarize my findings down here; it didn't all seem obvious and took some googling and tinkering, as some sources are a bit ambiguous on some of the points.

All native: `tools.go`

The best practice as endorsed by the Go team (https://github.com/golang/go/issues/25922#issuecomment-413898264).

//go:build tools
// +build tools

package yourpackage

import (
    _ "golang.org/x/tools/cmd/stringer"
    // ...
)

Change yourpackage to the name of the package that you put this in, e.g., use qux if you put it at the root of your module foo/bar/qux. Only use tools when you actually put it in a tools package (directory).

If you're on Go 1.17+, you don't need the // +build tools line.

`go.mod`

To add the dependencies and the latest versions of these tools to your go.mod and freshen the module cache, run

go mod tidy

Configure `gopls`

If you use gopls, configure it to include the tools build tag, e.g., for VSCode, add the following to your settings.json:

"gopls": {
  "build.buildFlags": ["-tags=tools"],
}

Install globally

You can install the go.mod versions of all the tools to your $GOBIN directory (probably not the best idea, expand for details)

```shell go install $(go list -f '{{join .Imports " "}}' tools.go) ``` Note that this can break things for you well outside the module that you run this in, so you're likely going to be better off installing specific versions of tools manually, at your discretion. If you still feel like you want to have the tools built (they really do start faster that way), either of the following sections (on `Makefile` and `bingo`) might be right up your alley.

`Makefile`

Makefiles are very project-specific and everyone's setup is different, but I've found it useful to add this to my Makefile.

```makefile toolsGo := tools.go toolsDir := bin/tools toolPkgs := $(shell go list -f '{{join .Imports " "}}' ${toolsGo}) toolCmds := $(foreach tool,$(notdir ${toolPkgs}),${toolsDir}/${tool}) $(foreach cmd,${toolCmds},$(eval $(notdir ${cmd})Cmd := ${cmd})) go.mod: ${toolsGo} go mod tidy touch go.mod ${toolCmds}: go.mod go build -o $@ $(filter %/$(@F),${toolPkgs}) tools: ${toolCmds} .PHONY: tools ``` If you decided to put your `tools.go` in a `tools` package, you would have to change the `toolsGo` variable to `tools/tools.go`. It may also make sense to flip the `toolsDir` variable to `tools/bin`. And maybe put this snippet in a `tools/tools.mk`, that you could include to reduce the clutter in your main `Makefile`. You'd probably want to add the `toolsDir` directory to your `.gitignore`. The magic `$(foreach ... $(eval ...` line defines `Cmd` variables (e.g. `stringerCmd`, `mockgenCmd`, etc.) to be used in other recipes in your `Makefile`. For (a very rudimentary) example: ```makefile wire: ${wireCmd} ${wireCmd} ./... .PHONY: wire ``` Running `make wire` would now 1. run `go mod tidy` to ensure `go.mod` is up to date with `tools.go` (if the latter is modified more recently than the former) 2. build [`wire`] at the version that's defined in your `go.mod` (if it's not already built), and, finally, 3. recursively generate all your provider and injector wiring. You can also execute ```shell make tools ``` and, if `go.mod` is older than `tools.go`, it will run `go mod tidy`, after which all the tools that you have defined in `tools.go` will get built under the `toolsDir` (as `bin/tools/stringer`, for example).

3rd party: `bingo`

You can use bingo to install version-suffixed executables of your module's tools in the global $GOBIN directory. This is an entirely different approach that avoids conflicts among tools and tool dependencies in larger projects, but it doesn't integrate with your module's go.mod (which might also be a good thing).

The author introduced this tool in https://github.com/golang/go/issues/25922#issuecomment-639057308 and further expanded on it in https://github.com/golang/go/issues/25922#issuecomment-759774278

(gopls/doc/settings.md at master · golang/tools)

(google/wire: Compile-time Dependency Injection for Go)

(bwplotka/bingo: Like `go get` but for Go tools! CI Automating versioning of Go binaries in a nested, isolated Go modules.)

silverwind commented 2 years ago

I think golang needs to:

Provide a way to define tool dependencies in go.mod only.
Provide a way to run tool binaries, e.g. go execute <tool> <args>.

Edit: It appears go run package@version already provides point 2, so really only point 1 is missing for complete management of tool dependencies.

bluebrown commented 2 years ago

@antichris, I still dont understand how I get the binary from that tools file and can use it in go generate?

I have this in the main package.

//go:build tools

package main

import (
    _ "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/github"
)

I ran go mod tidy but how can I use the binary now?

package main

//go:generate migrate -source=db
func main() {

}

$ go generate
main.go:3: running "migrate": exec: "migrate": executable file not found in $PATH

seankhliao commented 2 years ago

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

antichris commented 2 years ago

@bluebrown In general, the common practice would be running the tool using full package path in a go:generate directive, like so:

//go:generate go run golang.org/x/tools/cmd/stringer -type Foo

This would work if you have imported the respective package in your tools.go, e.g.:

import _ "golang.org/x/tools/cmd/stringer"

In your specific case, you'd have to import github.com/golang-migrate/migrate/v4/cmd/migrate:

import _ "github.com/golang-migrate/migrate/v4/cmd/migrate"

And, according to their documentation, add the appropriate build tags when running the COMMAND that you need:

//go:generate go run -tags postgres github.com/golang-migrate/migrate/v4/cmd/migrate -source=db COMMAND

golang / go

cmd/go: clarify best practice for tool dependencies #25922

What version of Go are you using (`go version`)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (`go env`)?

What did you do?

What did you expect to see?

What did you see instead?

If you get an error

Using a "tools" package

Updating go.mod

vendoring

Reference

All native: `tools.go`

`go.mod`

Configure `gopls`

Install globally

`Makefile`

3rd party: `bingo`

golang / go

cmd/go: clarify best practice for tool dependencies #25922

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

If you get an error

Using a "tools" package

Updating go.mod

vendoring

Reference

All native: tools.go

go.mod

Configure gopls

Install globally

Makefile

3rd party: bingo

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

All native: `tools.go`

`go.mod`

Configure `gopls`

`Makefile`

3rd party: `bingo`