cmd/go: get -u fails partially first time, succeeds second time when dependencies were refactored

dmitshur commented 9 years ago

Overview

In certain (reproducible and predictable) situations, running go get -u will partially fail the first time, but succeed the second time. During the first run, it will succeed to update the repository, but fail to build while printing an erroneous error message about a missing dependency. During the second run, everything will succeed.

This happens when Go code is refactored in a way so that one Go package that was previously imported ceases to exist, but it is also no longer imported hence the Go code is valid. For example, if a package that was previously imported is merged into another, or if a package is renamed and now has a new import path.

This is a problem because it hinders ones ability to refactor Go code. We now have ever-more-powerful refactoring tools like gorename that are able to help with refactoring. In its todo list is the ability to rename import paths. That ability is already present within another tool govers today. Being able to seamlessly improve and refactor Go code is important to allow it to get better. No one can come up with the perfect version the first time around, so it's possible they'll want to refactor after using the package for some time and having a better vision. An implementation detail limitation of go get -u should not hinder that.

What does `go version` print?

go version go1.4rc2 darwin/amd64

What steps reproduce the problem?

In order to prepare a (minimal) environment where the problem can be reproduced (i.e., it happens when go get -u runs and there are updates available), you need the following preparation steps first:

Run go get -d -u github.com/shurcooL/go-get-test-cmd. It has only one external dependency repo github.com/shurcooL/go-get-test-lib which will also be fetched.
Cd into $GOPATH/src/github.com/shurcooL/go-get-test-lib folder.
Being careful to do this in the right folder (!), run git reset --hard master^ to hard reset go-get-test-lib repo to previous commit. This will make it so that when we run go get -u there will be one update available.
- The go-get-test-lib repo has 2 commits. We need to pretend that at this time it only has 1 commit, and a new commit is made available later on.

That simulates a scenario where a user has an existing version of your Go package and its dependencies, and you decide to refactor one of the dependencies in this way:

https://github.com/shurcooL/go-get-test-lib/compare/master%5E...master

Now, you are ready to reproduce the problem, follow these steps:

Run go get -u github.com/shurcooL/go-get-test-cmd.
Run ls $GOPATH/bin/go-get-test-cmd and see that it failed to build.
Run go get -u github.com/shurcooL/go-get-test-cmd a second time.
Run ls $GOPATH/bin/go-get-test-cmd.
What happened?

$ go get -u github.com/shurcooL/go-get-test-cmd
package github.com/shurcooL/go-get-test-cmd
    imports github.com/shurcooL/go-get-test-lib/liba
    imports github.com/shurcooL/go-get-test-lib/libb
    imports github.com/shurcooL/go-get-test-lib/libb
    imports github.com/shurcooL/go-get-test-lib/libb: cannot find package "github.com/shurcooL/go-get-test-lib/libb" in any of:
    /usr/local/go/src/github.com/shurcooL/go-get-test-lib/libb (from $GOROOT)
    /Users/Dmitri/GoPath/src/github.com/shurcooL/go-get-test-lib/libb (from $GOPATH)
$ ls $GOPATH/bin/go-get-test-cmd
ls: /Users/Dmitri/GoPath/bin/go-get-test-cmd: No such file or directory
$ go get -u github.com/shurcooL/go-get-test-cmd
$ ls $GOPATH/bin/go-get-test-cmd
/Users/Dmitri/GoPath/bin/go-get-test-cmd
$

What should have happened instead?

Everything should've succeeded the on the first run of go get -u.

$ go get -u github.com/shurcooL/go-get-test-cmd
$ ls $GOPATH/bin/go-get-test-cmd
/Users/Dmitri/GoPath/bin/go-get-test-cmd
$ go get -u github.com/shurcooL/go-get-test-cmd
$ ls $GOPATH/bin/go-get-test-cmd
/Users/Dmitri/GoPath/bin/go-get-test-cmd
$

Please provide any additional information below.

This happens when Go code is refactored in a way so that one Go package that was previously imported ceases to exist, but it is also no longer imported hence the Go code is valid. However, when a user runs go get -u to update, it first parses the previous code to get a list of Go packages, then updates repos and does not re-parse the updated code (to find that a certain dependency is actually no longer imported) before returning an error that the previously imported package doesn't exist.

In the reproduce steps above, I used a refactor example where previously liba and libb were used, but libb is merged into liba. I would imagine the same issue would happen if libb were renamed to libc (and imported under new name) for example.

This likely affects /internal/ Go packages too. That means one can not use the argument that "you should not be removing old import path since someone might be importing it" for /internal/ Go packages as they're guaranteed not to be imported by anyone else, and you should be free to refactor them.

Workarounds:

Tell users to always run go get -u twice and ignore (possible) problems on first run.
Keep the old package around indefinitely.
Keep old package around for some amount of time. However, even if you keep it for a month, there's no guarantee some user will not run go get -u after 2 months and encounter this problem, it just reduces that chance.
- Alternatively, provide an empty package stub at the old location, for example see https://github.com/shurcooL/go/commit/c53996fe3f061319df88893596bfc50547f1768d. This allows the issue to be avoided.

dmitshur commented 9 years ago

This is an actual problem that has happened recently, see https://github.com/shurcooL/Go-Package-Store/issues/27 where two people reported running into it.

I've also run into this myself for other people's Go code occasionally (but ignored it at the time because running go get -u second time was okay).

gopherbot commented 9 years ago

CL https://golang.org/cl/12192 mentions this issue.

golang / go