Godep doesn't work when symlinks are used to map projects into go workspace

[lox]

I keep my projects under ~/Projects and then sym-link it into my go workspace under ~/.go/src/github.com/xyz/xyz. This doesn't seem to work with godep, when I run it (with a cwd of ~/.go/src/github.com/xyz/xyz) I get the following error:

$ godep save
godep: directory "/Users/lachlan/Projects/xyz/xyz" is outside source root "."
godep: error loading packages

By comparison, go list ./... works fine.

If I copy the directory into place, rather than sym-link it, the problem goes away.

[kr]

Thanks for reporting this. We definitely need to work on handling symlinks better.

What does the go list ./... command you mentioned print?

[chinamerp]

go list ./... _/vagrant/src/qs-host

[chinamerp]

@lox you can try "godep save github.com/xyz/xyz". it works for me.

if the command "godep save" has no parameter it will set to "." the command "go list -e -json ." will get wrong message likes "_/vagrant/src/qs-host"

[pksunkara]

Having the same problem.

[autarch]

I'm doing some digging and I see that this comes from the vcs code - https://code.google.com/p/go/source/browse/go/vcs/vcs.go?repo=tools#339

I'm pretty sure the fix needs to happen there too.

[autarch]

I'd also note that the output of go list -json varies depending on whether it's run from a dir under the GOPATH or not. If it's run in a dir outside the GOPATH, it does not include a "Root" key. This means that that "Dir" key will never be under the "Root" key, which is ultimately what causes vcs.FromDir to error out.

[kr]

The originally reported error message says the working directory was /Users/lachlan/Projects/xyz/xyz, but the accompanying description says it was ~/.go/src/github.com/xyz/xyz. Which is it?

A working directory of /Users/lachlan/Projects/xyz/xyz is definitely not following the standard Go code layout described at http://golang.org/doc/code.html. This is a basic assumption of godep; it's not going to change.

[autarch]

@kr - it's both, that's the problem the original user reported! The $GOPATH/src/github.com/xyz/xyz dir is a symlink to /Users/lachlan/Projects/xyz/xyz (I assume).

FWIW, we decided to give up on godep and just manually copy external repos into our own internal git server for now.

I'm not even sure how one would fix this particular problem. At some point you'd have to detect that the value of cwd wasn't under $GOPATH and then try to figure out what the corresponding symlink is. AFAICT, there's no API for going from "dir" to "symlink that points to it".

All that said, AFAICT, the go command itself seems to have no problem with this setup.

[kr]

Yes, my apologies, I misremembered the direction of the symlink.

The problem here is that Getwd returns the actual directory on disk, rather than a path containing a symlink. In general, it is not possible for the Go process (or any other process) to determine which path was used to arrive at that directory.

$ mkdir a
$ ln -s a b
$ cd b
$ python -c 'import os; print os.getcwd()'
/Users/kr/tmp/a

[kr]

I'm not even sure how one would fix this particular problem.

Don't use symlinks.

[autarch]

@kr - there are reasons one might want to do this. Unfortunately, the "standard Go layout" doesn't work out so well when you're trying to, for example, include a repo of go code as a submodule in some other non-Go repo.

[kr]

All that said, AFAICT, the go command itself seems to have no problem with this setup.

If you run 'go list', it doesn't print github.com/xyz/xyz, even though that's the import path of the package. This is a problem.

[kr]

the "standard Go layout" doesn't work out so well when you're trying to, for example, include a repo of go code as a submodule in some other non-Go repo.

I'd like to hear more about this situation, and why it's necessary in the first place. What are you trying to do?

[autarch]

Well, I'm getting closer to giving up on this but ...

Most of our code at work is written in Perl, with some pieces where we want higher performance in Go. We already have an extensive test framework in Perl as well, including code for creating fixture databases on the fly. We'd like to reuse this code for writing tests in Go, rather than rewriting the entire fixture system (and maintaining two copies of it).

All of our main Perl code lives in a single repo, and our Go code is in another repo. I had hoped I could check out the Go code as a submodule in the main Perl repo and then add a symlink under the GOPATH to that checkout. However, this is not working very well.

The other alternative is to simply import all the Go code into the main Perl repo, then check that out under the GOPATH and adjust our Go code import paths to match this. It's a pain, but clearly fighting Go is a pain too.

[kr]

The other alternative is to simply import all the Go code into the main Perl repo, then check that out under the GOPATH and adjust our Go code import paths to match this. It's a pain, but clearly fighting Go is a pain too.

Yeah, that's what I'd recommend, if you want to keep your Go code in the same repo as the Perl. Move all the files and change the import paths. Sorry.

I'm still curious why you want it to be in a subdirectory in the first place (either via submodules or copying files).

[autarch]

I'm still curious why you want it to be in a subdirectory in the first place (either via submodules or copying files).

As opposed to what? Sticking it in the top-level dir on a repo that already has a lot of other subdirs? It's not like we have any Perl code at the top of the repo either.

[autarch]

@lox - btw, to fix your problem all you need to do is reverse the direction of the symlink. Make /Users/lachlan/Projects/xyz/xyz a symlink to the directory under $GOPATH.

[kr]

As opposed to what?

As opposed to keeping it in a separate repo.

You mentioned that you want to share a test harness, but I'm not quite making the connection. Even if the code for the test harness is in the Perl repo, that doesn't necessarily mean you can't use it when your working directory is in another repo.

[autarch]

The main reason to put everything in one repo is to faciliate merging and deployment. We do all our dev work in branches, and then submit a PR for someone else to review and merge. It's a lot simpler to submit a single PR against one repo that encompasses all the work that needs review.

This all works fine right now since we have a small team where everyone can (at least in theory) work on any part of the code base. Eventually, if we grow large enough, I suspect we'll have multiple separate teams. At that point, I imagine we'd have one repo per team, and possibly additional repos of shared code.

[kr]

Ok. Well at this point it seems like there's not really anything godep can do to help. You just have to keep your go code in a standard go workspace (a symlink mimicking a standard go workspace pointing to code that actually lives elsewhere isn't the same thing) if you want godep to work.

Note, there's a similar discussion happening in #43. It's almost always possible to use a standard go workspace, even in a repo that spans many projects over the whole organization, and uses multiple programming languages.

I'm going to close this issue, but feel free to keep the discussion going, and let me know if you have any other concrete ideas for how godep might help.

[kr]

(I'd use the "wontfix" label, but it's really more like "can't fix". :bow:)

[autarch]

Yes, I agree with closing it. We ended up going back to just doing things the Go way and using godep. It's mildly annoying to have to move my checkout around, but not a big deal. I just wrote a small script to help other developers move it.

Also, I'm working on some docs for internal use that I might publish. I think they might be useful for other people outside my company.

[kr]

Publishing your docs would be fantastic! Godep could use more guides written in a different voice and different communication styles.

[lox]

Thanks for the comments all, I've since given up fighting "the go way" and embraced GOPATH.

[intabulas]

I know this issue is closed, but I have also seen this exact error if there is a case sensative typo in GOPATH.. If for example in disk (OSX) is /Users/<username/Projects/gospace but GOPATH as /Users//projects/gospace (note lowercase p) then the outside of root error appears on a godep save

[kr]

@intabulas do you mind opening an issue for case sensitivity? I'm not sure what (if anything) godep can do to help there, but it would be a separate problem (despite resulting in the same error message).