cmd/go: track tool dependencies in go.mod

mtibben commented 3 years ago

UPDATE: 2024-07-29: the latest proposal can be found here.

Background

The current best-practice to track tool dependencies for a module is to add a tools.go file to your module that includes import statements for the tools of interest. This has been extensively discussed in #25922 and is the recommended approach in the Modules FAQ

This approach works, but managing the tool dependencies still feels like a missing piece in the go mod toolchain. For example, the instructions for getting a user set up with a new project using gqlgen (a codegen tool) looks like this

# Initialise a new go module
mkdir example
cd example
go mod init example

# Add gqlgen as a tool
printf '// +build tools\npackage tools\nimport _ "github.com/99designs/gqlgen"' | gofmt > tools.go
go mod tidy

# Initialise gqlgen config and generate models
go run github.com/99designs/gqlgen init

The printf line above really stands out as an arbitrary command to "add a tool" and reflects a poor developer experience when managing tools. For example, an immediate problem is that the printf line will only work on unix systems and not windows. And what happens if tools.go already exists?

So while we have some excellent tools for managing dependencies within the go.mod file using go get and go mod edit, there is no such equivalent for managing tools in the tools.go file.

Proposed Solution

The go.mod file uses the // indirect comment to track some dependencies. An // indirect comment indicates that no package from the required module is directly imported by any package in the main module (source).

I propose that this same mechanism be used to add tool dependencies, using a // tool comment.

Users could add a tool with something like

go get -tool github.com/99designs/gqlgen@v0.14.0

or

go mod edit -require=github.com/99designs/gqlgen -tool

A go.mod would then look something like

module example

go 1.17

require (
    github.com/99designs/gqlgen v0.14.0 // tool
)

And would allow users to subsequently run the tool with go run github.com/99designs/gqlgen

This would mean a separate tools.go file would no longer be required as the tool dependency is tracked in the go.mod file.

Go modules would be "tool" aware. For example:

go mod tidy would not remove the // tool dependency, even though it is not referenced directly in the module
Perhaps if a module with a // tool dependency is imported by another module, Go modules understands that the // tool dependency is not required as an indirect dependency. Currently when using tools.go, go modules does not have that context and the tool is treated like any other indirect dependency
go get -tool [packages] would only add a dependency with a main package

fsouza commented 3 years ago

I like this, I find it annoying to use the tools.go solution, though I'll admit I don't have a better complaint than it being annoying/weird.

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

ianlancetaylor commented 3 years ago

CC @bcmills @jayconrod

mtibben commented 3 years ago

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

Good question! I'm not so familiar with the reasoning behind the multiple blocks... something to do with lazy loading? I'd defer to those with more experience in this area

mvdan commented 3 years ago

Personally, I think https://github.com/golang/go/issues/42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

Another big advantage is that you can pick specific versions of tools, and they won't interfere with your main go.mod module dependency graph. Perhaps I want to use a generator that pulls in an unstable master version of a library that my project also uses, and I don't want my project to be forced into using the same newer unstable version.

mvdan commented 3 years ago

The only downside to #42088 is that, if you repeat the same go run pkg@version commands across multiple files, it can get a bit repetitive. Luckily, you have multiple solutions at hand: sed scripts to keep the versions in sync, short script files to deduplicate the commands, or even a module-aware tool that could sync go run pkg@version strings with a go.mod file, if you wanted to do that.

seankhliao commented 3 years ago

Or GOBIN=local-dir go install pkg@version, always run from the local directory and not clobber whatever version the user may have globally installed. I think it would be a mistake for modules to implicitly rely on shared mutable global bin dir for a first class workflow

mtibben commented 3 years ago

Oh interesting, thanks @mvdan I wasn't aware of that solution. 🤔

A few concerns immediately come to mind...

You mean go run hack.me/now@v1.0.0 will just download and run some random go code 😱 That is slightly unexpected to me, equivalent to a curl | bash command. My assumption was always that go run ran local code or modules already specified in go.mod, but seems that assumption is incorrect
Should gqlgen instructions always be to specify version with go run github.com/99designs/gqlgen@0.14.0? That seems verbose
Repetition across multiple files, keeping version in sync, yep your comment above nails it

mtibben commented 3 years ago

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

mvdan commented 3 years ago

In module mode, go run can always download, build, and run arbitrary code. The difference between go run pkg relying on go.mod and go run pkg@version is how you specify the version and how it's verified. With a go.mod, you are forced into a specific version recorded in go.mod and go.sum. Without one, it's up to you what version you specify; @master is obviously risky, @full-commit-hash is safest, and @v1.2.3 is a middle ground that would probably be best for most people. Even if a malicious upstream rewrites a tag to inject code, GOPROXY and GOSUMDB should protect you from that.

mvdan commented 3 years ago

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

It certainly warrants a mention. I'm not sure we should bless it as the only best practice, though, because there can be legitimate reasons for versioning, downloading, and running tools some other way. Perhaps some of your tools aren't written in Go, such as protoc, so you use a "tool bundler" that's entirely separate to Go. Or perhaps you do need your tools to share the same MVS graph with your main module for proper compatibility, so you want them to share a go.mod file.

mtibben commented 3 years ago

Gotta say though... go run pkg@version seems like a massive security footgun to me.

go install I understand well that it can download code from a remote location and build a binary. It's not obvious at all that go run directly executes code from a remote location, and I wonder how widely that is understood.

mtibben commented 3 years ago

So even with the go run pkg@version approach, I still think this proposal has value for specifying tool dependency versions in the context of a module. This approach avoids requiring a tools.go file (as with the existing best-practice), and avoids specifying the tool version for every file that uses it (with the go run approach)

lwc commented 3 years ago

Also worth noting: codegen tools like gqlgen and protobuf are often comprised of a generator command and a runtime, both of which typically need to be versioned in lock-step.

This proposal solves that case rather neatly, allowing go.mod to manage both generator and runtime versions.

fsouza commented 3 years ago

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:
//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo
Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

We used to do that. Then people would have that replicated across different files and the version wouldn't always match, and we wanted to automate tool updating, so we figured that migrating to tools.go + having everything in go.mod would be better for compatibility with the ecosystem built around go modules (vs rolling our own tool to keep modules used directly in //go:generate up to date).

Again, tools.go works, but it's weird (not very scientific, I know 🙈). I think this proposal makes version management of tools better because it enables people to manage them using solely go commands (vs things like the bash oneliner shared by the OP).

bcmills commented 3 years ago

@jayconrod has previously suggested something similar, using a new directive (perhaps tool?) instead of a // tool comment.

Personally, I prefer the approach of adding a new directive — today we do treat requirements with // indirect comments a bit specially in terms of syntax, but they are semantically still just comments, and I would rather keep them that way at least to the extent possible.

A new tool directive, on the other hand, would allow us to preserve the existing semantics of go mod tidy without special treatment for // tool comments.

mvdan commented 3 years ago

@bcmills would such tool requirements be part of the same MVS module graph?

bcmills commented 3 years ago

The tool directive would list package paths (not module requirements), and the named packages would be treated as if imported in a .go source file in the main module.

In particular:

go mod tidy would ensure that the packages transitively imported by the named package (and its test) can be resolved from the module graph.
go mod vendor would copy the packages transitively imported by the name package into the vendor directory (but would omit its test code and dependencies as usual).
go list direct (#40364) would report the named packages as direct imports.

carldunham commented 3 years ago

Or go list tools

jayconrod commented 3 years ago

I like this proposal. I've had something similar in my drafts folder for a while. @bcmills touched on the main difference. go.mod would have a tool directive that would name the full package path for the tool. You'd still need a separate require directive for the containing module, and that would be treated like a normal require directive by MVS.

module example.com/use

go 1.18

require golang.org/x/tools v0.1.6

tool golang.org/x/tools/cmd/stringer

I don't think go run tool@version and go install tool@version completely replace go run tool and go install tool. When the @version suffix is used, it ignores the go.mod file for the current module. That's useful most of the time, but not if you want to track your tool dependencies together with other dependencies, or if you want to use a patched version of a tool (applying replace directives).

mtibben commented 3 years ago

Yeah I like the tool directive. There might be a couple of tradeoffs with compatibility with older go versions. A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

mtibben commented 3 years ago

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

jayconrod commented 3 years ago

A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

Right. The go command reports errors for unknown directives in the main module's go.mod file, but it ignores unknown directives in dependencies' go.mod files. So everyone working on a module that used this would need to upgrade to a version of Go that supports it (same as most other new features), but their users would be unaffected.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

My suggestion is to have tool act as a disembodied import declaration: it's just in go.mod instead of tools.go. You'd still need a require directive for the module providing the tool, and it would be treated as a regular requirement by go mod tidy and everything else.

If you don't want to mix tool and library dependencies in go.mod, it's probably better to either use go run tool@version or to have a separate tools.mod file, then go run -modfile=tools.mod tool.

mtibben commented 3 years ago

Yep that makes a lot of sense @jayconrod

rsc commented 3 years ago

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

mtibben commented 3 years ago

@jayconrod Did you want to write up the tool directive approach that we could incorporate as an option into this proposal? I'm happy to collaborate on it with you. Positive feedback on that approach so far in this thread, and it would be good to compare the options directly against each other, now that this proposal will be considered by the go-powers-that-be

jayconrod commented 3 years ago

Sure, I'll paste my draft proposal below. Unfortunately I won't be able to work on the implementation for this, but this is what I was thinking in terms of design.

tool directive

I propose adding a tool directive to the go.mod file format. Each tool directive names a package.

tool golang.org/x/tools/cmd/stringer

go mod tidy and go mod vendor would act as if each tool package is imported by a package in the main module. Tool packages would be matched by the all metapackage.

Modules providing tool packages must still be required with require directives. Requirements for tools would not be treated differently from other requirements. This means that if a command and a library are needed from the same module, they must be at the same version (related: #33926). Requirements on modules providing tools would also affect version selection in dependent modules if lazy loading is not enabled.

The tool directive itself would not affect version selection. go mod tidy, go mod vendor, and other commands would ignore tool directives outside the main module.

go get -tool

tool directives could be added or removed with go get, using the -tool flag. For example:

go get -tool golang.org/x/tools/cmd/stringer@v0.1.0

The command above would add a tool directive to go.mod if one is not already present. It would also add or update the requirement on golang.org/x/tools and any other modules implied by that update.

require golang.org/x/tools v0.1.0
tool golang.org/x/tools/cmd/stringer

A tool directive could be removed using the @none version suffix.

go get -tool golang.org/x/tools/cmd/stringer@none

-tool could be used with -u and -t.

tools metapackage

To simplify installation, go install and other commands would support a new metapackage, tools, which would match packages declared with tool dependencies.

# Install all tools in GOBIN
go install tools

# Install all tools in the bin/ directory
go build -o bin/ tools

# Update all tools to their latest versions.
go get tools

It would not be an error for a tool directive to refer to a package in the main module, so the tools metapackage could match a mix of local and external commands.

mtibben commented 3 years ago

Modules providing tool packages must still be required with require directives. Requirements for tools would not be treated differently from other requirements. This means that if a command and a library are needed from the same module, they must be at the same version (related: #33926). Requirements on modules providing tools would also affect version selection in dependent modules if lazy loading is not enabled.

Ah yes this is similar to the // tools approach which wouldn't allow a different version between the tool binary and the library to be specified.

Specifying the actual location of the binary isn't something the // tools approach solves - but do we need to? In the // tools approach, version is specified once as part of a normal require directive, but the the location binary to be run is left up to go run. e.g. the go.mod would look like

require golang.org/x/tools v0.1.0 // tool

and you'd call go run which would use the version defined by go.mod

go run golang.org/x/tools/cmd/stringer

@jayconrod Is the reason for specifying the exact tool location as you've described (e.g.tool golang.org/x/tools/cmd/stringer) just to allow go install tools? I feel that "installing" tools might just cause the same kind of version issues between projects if installing to a common location, and go run may be the superior approach.

jayconrod commented 3 years ago

@jayconrod Is the reason for specifying the exact tool location as you've described (e.g.tool golang.org/x/tools/cmd/stringer) just to allow go install tools? I feel that "installing" tools might just cause the same kind of version issues between projects if installing to a common location, and go run may be the superior approach.

I was suggesting tools would be a metapackage (like cmd or std), so you could use it with go install tools (hopefully setting GOBIN first), or go build -o bin/ tools or go list tools; it would work with any subcommand that accepts package arguments.

I think having a separate tool directive in go.mod is helpful for understanding why a requirement is needed. For example, suppose you stop using golang.org/x/tools/cmd/stringer in favor of a more advanced tool. At some point in the future, you might see:

require golang.org/x/tools v0.1.0 // tool

and wonder why it's there. It's not clear that it's safe to remove. But with:

tool golang.org/x/tools/cmd/stringer

you'd know that it's not used anymore, so it's safe to delete that line. The next go mod tidy would remove the corresponding requirements (assuming no other packages are needed).

bwplotka commented 3 years ago

2c from my side: Go tools are used not only by Go modules. This is why a separate go.mod for tools or something like what https://github.com/bwplotka/bingo automated for you might be preferred. (:

jayconrod commented 3 years ago

@bwplotka Separate go.mod files are already supported via the -modfile flag since Go 1.14.

rsc commented 3 years ago

@bcmills, @matloob, what's the status on this?

bcmills commented 3 years ago

I'm in favor of Jay's refinement to the proposal (posted in https://github.com/golang/go/issues/48429#issuecomment-938010150), using a new keyword (perhaps tool) to denote dependencies on specific packages that are not otherwise imported, for which dependencies would then be maintained by go mod tidy.

I think this would be a good ergonomic improvement, and it seems feasible to implement for Go 1.19 or so.

andig commented 3 years ago

I agree with @bcmills. Having go treat tools is easily achieved by using tools.go. It takes a learning curve but is a "one-time" problem on developer side. Being able to actually install tools using the enhanced tooling suggested in https://github.com/golang/go/issues/48429#issuecomment-938010150 is something that every user of a module is confronted with.

deefdragon commented 3 years ago

Just to confirm, but tools used by required modules or tools themselves will not be imported as indirect tools correct?

bcmills commented 3 years ago

Correct. It would be as if they were imported by an internal (or otherwise non-importable) package.

rsc commented 3 years ago

Putting on hold for Go 1.18 work.

mewmew commented 3 years ago

Firstly, I also support the proposed dedicated tool directive, as this reduces the use of "magic comments" with semantic meaning, of which there are already enough in the Go ecosystem. We don't need to introduce more :)

re: go install tools by @jayconrod in https://github.com/golang/go/issues/48429#issuecomment-938010150

# Install all tools in GOBIN
go install tools

# Install all tools in the bin/ directory
go build -o bin/ tools

# Update all tools to their latest versions.
go get tools

Just a quick comment regarding tool dependencies used by Go modules. Not all external tools will be developed in Go, some external tools may be developed in arbitrary languages, but we may still wish to pin specific Git commit revisions or tagged versions to ensure that all build requirements of our Go modules are satisfied and don't go out of sync with the required versions of external tools.

A real world example of this is the Textmapper tool (written in Java) used by github.com/llir/llvm to generate lexers and parses for LLVM IR from a BNF grammar. Since the Textmapper tool is not written in Go, it is currently tracked by a Git submodule: https://github.com/llir/ll/tree/master/tools

Just put this out here, to keep in consideration when working on dependency handling of tools (e.g. build tools) required by the Go module.

Should go.mod also pin versions/revisions of tools developed in other languages than Go? If not, simply disregard this commit.

Cheers, Robin

P.S. @inspirer is working on a Go version of Textmapper (https://github.com/inspirer/textmapper/issues/6), but it has yet to reach feature parity with the Java version. (The above still applies to other tool dependencies used by Go modules and developed in other languages than Go of course.)

deefdragon commented 2 years ago

@rsc This should be able to be taken off hold right?

Also, I am a bit confused why this went on hold in the first place. I understand that there was a lot of 1.18 work, but why did that necessitate this going on hold again?

ianlancetaylor commented 2 years ago

@bcmills Is there more work pending, or should this come off hold? Thanks.

bcmills commented 2 years ago

This should come off hold.

ConradIrwin commented 1 year ago

I like the proposal to add a tool directive to go.mod.

Instead of adding go get -tool, perhaps go install should update these lines when executed within a module (though this may have too many false-positives because I don't pay attention to my working directory when installing things, it would avoid the need for new syntax to learn – and any incidentally added tools could easily be removed).

Unlike the proposal by @jayconrod, I would not merge the tools' dependencies into the package's dependencies (and I'd build the tools ignoring the go modules require and replace directives). Each tool should be built in standalone mode, because it would be surprising for me that adding a tool could affect my main module's build.

As such I'd make the syntax inclusive of a version number:

tool golang.org/x/tools/cmd/stringer@v0.2.0

At this point running go run golang.org/x/tools/cmd/stringer or go install golang.org/x/tools/cmd/stringer in the module directory would always pick up v0.2.0.

ConradIrwin commented 1 year ago

Thinking further...

One problem with the current approach to bundling tools with repositories is that in order to manually run the tool I need to either type go run quite/a/lot/to/type every time (which is tedious),go install it (which means that I no longer can be sure I'm running the right version as I hop from project to project), or write a wrapper script (which is unnecessary).

Maybe a better way of avoiding this is to focus on fixing the ergonomics of running tools that have been bundled with the module, and making it easy to run the correct version (c.f. #57001 for go itself).

I propose the following:

go.mod gets a new directive run that lets you define a set of tools for use with the current project. The syntax is run [toolname] => [path-to-run]. The path-to-run must be in the current module (starting with ./) or in a module that is required by the current module.
```
require golang.org/x/tools v0.2.0
run stringer => golang.org/x/tools/cmd/stringer
run boop => ./cmd/boop.go
```
The reason that it is restricted to either the current module (or modules you depend on) is so that transitive dependencies appear in your go.mod and go.sum to give you reproducible builds, and it gives you the ability to replace tool dependencies if you wish. It is arguably a bit odd that the tool will use the same version of dependencies as your main module, but it is not likely a problem in practice, and it keeps the mental model simpler.

The reason to explicitly specify the toolname is to allow for the case where there are multiple commands with the same name. If this isn't a case we want to support, it would be reasonable for the syntax to be run path/to/X where the toolname is inferred from the last path segment.

Run lines may only contain one [path-to-run], so if you want to write a tool using multiple go files, you'll have to put them in a directory.
go run X would look for a run directive with toolname X and (if it exists) would act as though you'd run go run path-to-run in module mode.

When run in this mode, go run would cache the fully linked binaries so that future runs of go run X do not need to re-link if the built tool is up-to-date (just as go build does). This should mean that running go run X is relatively quick the second time.

The binary will be cached at $GOCACHE/tool/<current-module-path>/<toolname>, so there will be at most one version of each tool cached per module. Unlike go run in general, this should have a relatively good hit rate. go clean -cache would empty this directory. (We could also add a separate go clean -runcache if we think it's likely people will want to clear this directory without clearing the rest of the cache).
go get -run X@version would add a new run line, inferring the toolname from the last path segment; and (if necessary) add a require line for the module containing X at the given version. go get -run X@none would remove the line (and also the require if it's not otherwise needed).

Although I like @jayconrod's idea of the tools meta-package described above, I have removed it from this proposal because I think the use-case is not that clear. It would be somewhat nice to be able to "precompile all tools" so that go run X is fast the first time, but it's not essential (and it would be possible to write a script that did that). It could be a good thing to add later if there's demand.

Edit: this was updated to reflect a slightly tighter scope; and to rename the directive to "run" instead of "tool" to (maybe) reduce confusion (as go tool does something completely different from go run). (Though maybe go tool X should get this behaviour instead of go run X?)

Previous version

I propose the following: 1. `go.mod` gets a new directive `tool` that lets you define a set of tools for use with the current project. ``` // stringer is a standalone tool tool stringer => golang.org/x/tools/cmd/stringer v0.2.0 // boop is a tool in the current module tool boop => ./tools/boop // protoc-gen-go is provided by a required module tool protoc-gen-go => google.golang.org/protobuf/cmd/protoc-gen-go require google.golang.org/protobuf 1.28.1 ``` If a version number is specified in the `tool` line then that version of the tool is used, it is built in standalone mode (ignoring the require/replace directives of the current go.mod). If a version number is *not* specified, then it is built in "companion mode" (respecting the require/replace directives of the current go.mod), and it must either be in the current module or in a module that is required (copying the behavior of `go run` today). 5. `go run` learns to pick up tools from the `go.mod` file: `go run stringer` would work exactly as if I'd run `go run golang.org/x/tools/cmd/stringer@v0.2.0`. 6. `go get tools` would download tool dependencies, `go build tools` would compile them and cache the result so that `go run stringer` is fast. `go install tools` would install them globally (though this may be an anti-pattern). 7. `go get -tool golang.org/x/tools/cmd/stringer@latest` in module mode would add a tool line with the latest version, and with the toolname inferred from the path of the module in addition to doing the install. It is arguably possible to omit the first argument to the tool directive in the `go.mod` and infer it from the last path segment of the second argument (`tool golang.org/x/tools/cmd/stringer@v0.2.0` would be equivalent to `tool stringer golang.org/x/tools/cmd/stringer@v0.2.0`) but this feels a bit too magic to me... This would not support non-go tools, as I think specifying a way to version arbitrary binaries is probably out of scope for go's tooling. I am not sure whether `tool stringer golang.org/x/tools/cmd/stringer@v0.2.0` should change the behaviour of running `go run golang.org/x/tools/cmd/stringer`; it definitely could pick the version up from the `go.mod`, but it may not be clear why it would given the chosen syntax. This does require teaching people to use `go run stringer` instead of `stringer` or `go run golang.org/x/tools/cmd/stringer@v0.2.0`, but I think that's a much better tradeoff all around.

gopherbot commented 1 year ago

Change https://go.dev/cl/472755 mentions this issue: cmd/go: supportrundirective to go.mod

ConradIrwin commented 1 year ago

@rsc is there a way to ask the proposal committee to take a look at this when you next meet?

There are roughly two options proposed here: which seems like the right direction, and what are the remaining things to resolve before something like this could be accepted?

the one proposed by @jayconrod: https://github.com/golang/go/issues/48429#issuecomment-938010150 which replaces the existing hack with a line in go.mod
the version proposed by me: https://github.com/golang/go/issues/48429#issuecomment-1415058683 that attempts to solve the problem more holistically.

I'd be interested in trying to implement either of these approaches (or an alternate idea) for go 1.22; but I'd love some input from you all on what makes sense as a next step; and if it'd be helpful I'm happy to write a more detailed proposal doc.

rsc commented 1 year ago

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

joeblubaugh commented 1 year ago

I have some questions about @ConradIrwin 's proposal:

Should run directives really participate in MVS? One advantage of scripting go install X@v1.0.0 over tools.go is that the version of the installed tool doesn't affect the version of the modules selected for building the module's code.
go get -run X@none feels hacky - should the user just delete the run directive instead, and run go mod tidy? If not, perhaps a change to go mod? Something like go mod remove?
What happens if I call go get -run -u X?

I had a thought about a separate tool management system related to go install. A file go.tools that can interact with go install and go.run:

install ./cmd/boop.go
// or, with an alias:
install anyToolName => ./cmd/boop.go
install golang.org/x/tools/cmd/stringer v0.2.0

When run without arguments, in a module with a go.tools file, go install will:

build main packages that are local to the module like ./cmd/boop.go
build non-local packages in the same way the go install currently does.

These compiled binaries will be cached in the same way as @ConradIrwin's proposal. When calling go run within a module with a go.tools file, go run will select the cached version of the binary specified by go.tools

I like that this idea cleanly separates tools used to interact with a module (and may do nothing with the Go source at all), and dependency versions required to build and test the module's Go code.

leighmcculloch commented 1 year ago

I think @ConradIrwin's proposal (https://github.com/golang/go/issues/48429#issuecomment-1415058683) works for most of the Go repositories that I work in and I think I would use it, but it feels unnecessary given the go run command can run any Go tool already.

+1 @joeblubaugh's concern (https://github.com/golang/go/issues/48429#issuecomment-1465426311) about the tools affecting version selection. I think this would create some big surprises. There are times where it would be super useful if I'm using a tool that has a corresponding library that have to be aligned. But there are also times where it would be surprising if my transitive dependencies controlled which version of a tool was in use.

For the most part I've found the go run module/path@version to be a really effective way to run tools. I use that syntax in gogenerate directives and in Makefiles and it works great.

I think the main thing these proposals are adding is aliasing shorter names to paths. If paths are truly too long to be convenient maybe a general path aliasing system would be appropriate. Other systems have done this. For example, Deno added support for aliasing with their import map file (e.g. https://deno.land/manual@v1.31.0/basics/import_maps). I'm not advocating for aliasing, I don't think the go tool should adopt path aliasing as a feature, but that's what it feels like these proposals are adding to the go tool, narrowly for tools.

bcmills commented 1 year ago

I think the main thing these proposals are adding is aliasing shorter names to paths.

I think there are two separate concerns.

One concern is adding tools to the dependency graph, particularly for go mod vendor. It isn't feasible to separate those dependencies from the general dependency graph, because the vendor tree (intentionally) doesn't allow for more than one version of the same package import path, and I don't think it's worth adding more complexity in order to support that.

The other concern is making it easier to run tools; I think that's what the aliasing is getting at. I could see that being useful for, say, tests that run those tools. But I think there is a lot of complexity there that would need to be resolved — for example, if I run go test example.com/m, how would m_test identify the stringer selected by the module from which go test is run (which may in general be different from the one containing example.com/m)?

ConradIrwin commented 1 year ago

@joeblubaugh / @leighmcculloch I went back and forth on "should they contribute to MVS or not". Originally I thought maybe the module author should be able to chose, but the distinction is subtle. I landed on "yes, they should" to give module authors control over which dependencies are pulled in. You can of course still go run x@y (or go install x@y) if you need to not have them intermingle. For the projects I work on, it would make little difference because the dependencies of my tools are mostly distinct from the dependencies of the app (and as @bcmills points out, it would be a very big change to allow tool dependencies to differ from the main module dependencies).

I am not sure whether go get -run is needed, but I liked the idea from @jayconrod's proposal because it gives you a one-line command to run to make the change, which could be copy/pasted into documentation. The @none syntax is already supported by go get, so it made sense to me to support here too. go get -run -u would act just like go get -u (without the -run).

A separate go.tool file seems unnecessary make that much sense if the tools are participating in MVS as then any tool added would need changes in two files. It seems simpler to use the one file.

@joeblubaugh I think that change to go install you propose would be quite intrusive (I mostly use go install to "put the binary produced when building the current package on my path"). @jayconrod had proposed making go install tools do something like what you suggest – though putting them directly in the path – we could expand this proposal to support that (or do later) if it's a common desire.

@leighmcculloch glad to hear you would use this! I do hear your point around aliasing being unnecessary (and indeed I'd be happy to have something that did the versioning and caching without the aliasing). Adding the aliasing I think makes go run significantly more user-friendly (currently I either go install and then use the binary name, which leads to problems making sure the version matches between repos; or I write wrapper scripts to avoid having to type the full path; it'd be nice to just go run X instead).

ConradIrwin commented 1 year ago

@bcmills interesting thought. Currently go test builds the test binary with the current module's dependencies, but runs the tests in the directory containing that module's code. I think this means that it will "do the right thing" in most cases – a test that shells out to go run will pick up the run directives from the module being tested.

It does mean that if you have a different version of the tool required by the main module and by example.com/m then the test will be compiled with one version but go run will run with a different version. (This already is a problem today if you shell out to go run example.com/tool in the tests of example.com/m using the current tools.go hack).

I think it would be theoretically possible to fix the version mismatch in the specific case of go test, but I'm not sure that the cost would be worth it in practice. (We'd either need a new dependency resolution mode, or code to generate a new go.mod that merges two previous ones, and an environment variable to tell go run to change its behaviour).

I'm sure we shouldn't try to fix this in the case that you go build an arbitrary binary and then run it in a different directory – if it shells out to go run then go run would have no specific knowledge of the module used to build the binary; it would just use the working directory. For me that's a pretty strong argument that it should work the same way for tests too.

I don't think this is a problem for other go commands (go run doesn't change directory, go generate only works on the current module), but are there other places it's likely to show up (and cause actual problems)?

golang / go