golang / dep

Go dependency management tool experiment (deprecated)
https://golang.github.io/dep/
BSD 3-Clause "New" or "Revised" License
12.84k stars 1.05k forks source link

How do I build this repository? #786

Closed dzrw closed 7 years ago

dzrw commented 7 years ago

I'm having trouble building dep locally on my MacBook Pro. Below is a trace of what I've tried, some facts about my environment, and the errors raised. In particular, I cannot get past downloading dep's dependencies using dep. Hopefully, someone can point out what's wrong with my setup.

$ go version
go version go1.8 darwin/amd64
$ echo "$GOPATH,$GOROOT"
,
$ go get -u github.com/golang/dep/cmd/dep
$ which dep
/Users/politician/.go/workspace/bin/dep

I'm running Go 1.8. Neither GOPATH nor GOROOT are set. I fetch dep per the README. It lands in a go workspace that I mostly ignore and never work out of.

$ git clone git@github.com:golang/dep.git
... snip ...
$ cd dep
$ dep ensure
both /Users/politician/code/golang/dep and /Users/politician/Documents/Code/golang/dep are not within any known GOPATH

I clone the repo into my project area, and guess that I need to run dep ensure because I see that you have a Gopkg.toml file. This fails. I did not expect this error because elsewhere I've read that dep considers the Gopkg.toml to define the project root, so I expected it to be able to self-orient.

Note that ~/code is a symlink to ~/Documents/Code. This may or may not be relevant.

$ export GOPATH=$PWD
$ dep ensure
split absolute project root: /Users/politician/code/golang/dep not in any GOPATH

I set $GOPATH to the current directory and try again. No dice.

Discussion

dep fails in this exact same way for a couple other projects. I try to pull them down, and run dep ensure and it fails. I've tried several combinations of setting GOPATH to various different places and trying multi-entry GOPATHS. I've looked for, but haven't found any documentation on how to setup the environment to be compatible.

Now, I've been a heavy user of Dave Cheney's gb for several years. However, since dep seems like it's going to be the official way forward at some point, it seemed like a good idea to get some experience with it. Unfortunately, it's likely that I've forgotten something fundamental about how to use go build directly since gb takes care of most of the annoying bits.

Other folks seem to be able to use dep successfully, including some of my colleagues, but no one can explain why it doesn't work on my machine. I hope I've provided enough detail to figure out what's different about my environment, but let me know if there's anything I can provide to troubleshoot this.

Finally, since this repo includes a Gopkg.toml, it has a multi-step build process -- at least dep ensure and go build .... Would consider including a Makefile to make that build process explicit for newcomers?

ibrasho commented 7 years ago

@politician Welcome to dep! Thanks for the detailed issue.

I assume you are trying to compile dep? You don't need to get the dependencies of dep using dep ensure since our vendor directory is already committed. Just doing go get -u github.com/golang/dep/cmd/dep is enough to compile a binary in GOPATH/bin.

At the moment, you can't run dep in a project that doesn't exist within a GOPATH/src subdirectory. So assuming you want to try dep ensure in dep itself, you will need to ensure that you clone it in GOPATH/src/github.com/golang/dep (which go get most likely did).

Both /Users/politician/code/golang/dep and /Users/politician/Documents/Code/golang/dep are not within your GOPATH (which appears to be /Users/politician/.go/workspace. Setting the GOPATH to PWD also doesn't solve the issue since you code now is at the GOPATH root.

darkowlzz commented 7 years ago

@ibrasho hahaha, you commented again just while I was drafting mine πŸ˜„ Anyway, since I have drafted it, I'm posting it. Just my attempt to help.


Hi and welcome,

Thanks for trying dep. I'll share experiences from my workflow and try to help you out 😊

To get the code, I use go get -u and it clones the project to my default GOPATH workspace. Once I have the code, I use this same repo to develop dep. Since, it's already under a GOPATH and properly cloned, I don't have to worry about anything else.

Neither GOPATH nor GOROOT are set. I fetch dep per the README. It lands in a go workspace that I mostly ignore and never work out of.

I believe you have to work under a GOPATH, that's how all go projects are supposed to work at the moment. And you have to set a GOPATH. You can refer The GOPATH environment variable to set your GOPATH env var.

I see your go workspace is under /Users/politician/.go/, which isn't the default path if GOPATH is set, I guess. So ensure that your GOPATH is set to /Users/politician/.go/workspace.

dep ensure in dep itself works fine if the above setup is done right. But you don't have to run ensure because all the dep dependencies/vendors are already committed. Just fetching the code and running go install -v ./cmd/dep/ from the dep project root, should build and install dep from your copy of source.

And if you wanna contribute to dep with the same copy of repo, I would recommend reading GitHub and Go. It talks about setting up another git remote and all the things that you would need to contribute to any go project.

Finally, since this repo includes a Gopkg.toml, it has a multi-step build process -- at least dep ensure and go build .... Would consider including a Makefile to make that build process explicit for newcomers?

We don't run ensure anytime, until we decide to add another dependency. But since the dependencies are committed under vendor/, there's no need to ensure. And hence we don't need Makefile right now, but we might have one in future, if required.

Hope that was helpful.

dzrw commented 7 years ago

@ibrasho @darkowlzz Thanks for the quick response! I think I'm beginning to understand what's gone wrong.

@ibrasho said,

At the moment, you can't run dep in a project that doesn't exist within a GOPATH/src subdirectory. So assuming you want to try dep ensure in dep itself, you will need to ensure that you clone it in GOPATH/src/github.com/golang/dep (which go get most likely did).

Ok, I've found line 247 in context.go that makes the assumption that GOPATH is the default location referred to in the Go documentation. The way I use Go is different from the Go documentation because I work on many projects written in different languages on a daily basis.

Therefore my project area puts all of these projects side-by-side and mirrors the way Github, Bitbucket, and Gitlab structure their repository layout. For example,

Very different code bases, but the project paths all follow a similar path structure which makes it easy for me to navigate. Using gb allows me to maintain that same path structure for Go projects, but unfortunately, it looks like dep would require any dep-based project to be cloned into either:

clone-into set GOPATH to example path
~/.go/workspace/src ~/.go/workspace ~/.go/workspace/src/github.com/golang/dep
~/code/dep-projects/src ~/code/dep-projects ~/code/dep-projects/src/github.com/golang/dep
GOPATH/src (whatever the default Go install GOPATH is) $GOPATH/src/github.com/golang/dep

Takeaway 1: There is a strong assumption in dep that Go source code is located an folder structure that closely resembles the Go monorepo workspace concept. In particular, it is not possible to use a folder structure with a different top-level structure; dep does not orient itself around Gopkg.toml.

@darkowlzz said,

I believe you have to work under a GOPATH, that's how all go projects are supposed to work at the moment. And you have to set a GOPATH. You can refer The GOPATH environment variable to set your GOPATH env var.

Interesting, so it turns out that when using gb you don't have to set a GOPATH because the tool manages it for you. I believe that read something about Go getting rid of the GOPATH around the same time that dep started up, so since I haven't been using GOPATH explicitly for years it looks like what's happened is that I've forgotten what other folks deal with.

Now, I do need to set GOPATH to allow go get to know where to install binaries, so on the rare occasions that I go get something, I'll set the GOPATH to ~/.go/workspace.

From your description, it almost sounds like most folks are working out of a structure that also contains random 3rd party binaries and packages, and that they use another folder structure for handling projects written in other languages. Wild!

@darkowlzz said,

We don't run ensure anytime, until we decide to add another dependency. But since the dependencies are committed under vendor/, there's no need to ensure.

Ok, awesome. That makes sense now. I didn't realize that there was no need to fetch the vendored dependencies after cloning because I mistakenly thought that the presence of the Gopkg.toml and Gopkg.lock files implied a different workflow (i.e. the nonreproducible builds / npm / bundler workflows).

Takeaway 2: Don't automatically assume that you need to call dep ensure upon cloning a repo containing a Gopkg.toml file. Look for the vendor/ directory first, if it's there, then you're good to go.

dzrw commented 7 years ago

Would you guys be interested in entertaining a patch that allowed my use case to work?

I haven't thought deeply about it. Perhaps one could enable dep to discover Gopkg.toml and add it to the GOPATH in a subshell, or perhaps one could add a command-line flag that overrides the default behavior in some way.

ibrasho commented 7 years ago

dep is being developed with the goal of being absorbed into the go toolchain. So I'm not sure we want to support cases that don't comply with the toolchain assumptions. The final call of this is left to @sdboyer though. 😁

It seems gb is a totally different way for working with Go. I can propose the following workaround to keep you preferred structure and still be able to use dep.

You can either set GOPATH to ~/.go/workspace or move it to ~/go (the default GOPATH in Go 1.8). If you move it, everything in the go toolchain should know your GOPATH even if it is not set.

Use GOPATH/src to store the repositories and use symlinks to keep your current structure. Thus, /Users/politician/code/golang/dep (or /Users/politician/Documents/Code/golang/dep) should be a symlink to GOPATH/src/github.com/golang/dep.

This should work for dep, but I'm still not sure how it'll impact gb behavior.

dzrw commented 7 years ago

Thank you for that clarification and the workaround.

It seems gb is a totally different way for working with Go.

I wouldn't put this all on gb, for all I know my team and I are the only ones writing code in this way. πŸ˜„

sdboyer commented 7 years ago

hi @politician, thanks for such a patient and detailed writeup!

Being able to work from anywhere, not just within GOPATH, is certainly something we'd like to support. But it has deep implications and, as @ibrasho notes, takes us out of alignment with the go toolchain in a way that we can't currently afford to do. We do have some plans about how we might resolve it eventually, but unfortunately, there's little we can do safely right now. 😒