glide support?

[nshalman]

I'm currently building the Cerana tools with what is probably a bit of a terrible hack: https://github.com/cerana/nixpkgs/blob/ceranaos/pkgs/os-specific/linux/cerana/default.nix#L21

We use glide to manage the dependencies, and I'm wondering if go2nix either already supports software doing vendoring with glide, if it's something that might come in the future, or if I should plan to stick with that hack for a while.

Thank you for all of your hard work making working with Go software in Nix as pleasant as possible!

[kamilchm]

I'll think about support for godep and glide in July. One thing that needs to be managed somehow are repo checksums for dependencies that other package managers don't provide :/

[rushmorem]

@nshalman That hack is not necessary at all. For example, you can do something like the following instead:-

export GOPATH=$(mktemp -d /tmp/cerana.XXXXXX)
mkdir -p $GOPATH/src/github.com/cerana
cd $GOPATH/src/github.com/cerana
git clone https://github.com/cerana/cerana.git
cd cerana
git checkout ea99f44a58faffcbfa9b68d284340c4187f046dd
glide install
go2nix save

At this point, all you need to do is copy the generated default.nix and deps.json (now deps.nix) to your cerena package in the nixpkgs tree and then edit default.nix to include your postBuild hook.

[sheenobu]

@rushmorem The deps.nix file becomes empty in your process. go2nix save seems to ignore the 'vendor' folder when determining dependencies.

[rushmorem]

@sheenobu This is a bug in the current version (v1.1.1). @kamilchm I meant to report this earlier but I never got around to doing it. For now I just use v1.1.0.

[kamilchm]

I don't think its a bug. Current Go vendoring scheme implies that every dependency found in vendor makes the same dependency in GOPATH obsolete and go2nix doesn't need to add it as an external dependency because it won't be used in the build process. But there are cases of managing vendor dir with tools like glide or govendor. The best solution I can think of is to add --manage-vendor flag and add vendor deps to nix when it's set.

[kamilchm]

I could also add glide support explicitly but it looks like it's better to align go2nix with https://github.com/sdboyer/gps in the long term.

[rushmorem]

Current Go vendoring scheme implies that every dependency found in vendor makes the same dependency in GOPATH obsolete and go2nix doesn't need to add it as an external dependency because it won't be used in the build process.

This assumes that the vendor directory is committed into the source repo which is not always the case. In my own repos, I always ignore the vendor dir so my deps.nix is always empty when using v1.1.1. I noticed this with other external repos as well. I think another option is to check if the vendor dir is being ignored by the scm's ignore file.

[domenkozar]

I'm trying to package https://github.com/DataDog/gohai with this process and I'm also getting an empty file. Is there a workaround atm?

[kamilchm]

Glide was changed a lot since this issue was created :/ I want to wait until https://github.com/golang/dep/issues/380 and implement new go2nix with dep support only. Quick & dirty workaround:

glide install
rm -rf vendor
cat glide.lock | grep -o "name:.*" | awk '{ pkg = $2; gsub(/\//, "-"); printf("mkdir -p $GOPATH/src/%s && rsync -aP ~/.glide/cache/src/https-%s/ $GOPATH/src/%s/\n", pkg, $2, pkg); }' | xargs -I {} sh -c '{}'
go2nix save

[rushmorem]

@kamilchm

cat glide.lock | grep -o "name:.*" | awk '{ pkg = $2; gsub(/\//, "-"); printf("mkdir -p $GOPATH/src/%s && rsync -aP ~/.glide/cache/src/https-%s/ $GOPATH/src/%s/\n", pkg, $2, pkg); }' | xargs -I {} sh -c '{}'

When I try that in my private repo I'm getting xargs: command too long.

[rushmorem]

It's an internal company project that's not available publicly. If I remove the last part to get the input being piped to xargs I get:-

mkdir -p $GOPATH/src/github.com/gin-gonic/gin && rsync -aP ~/.glide/cache/src/https-github.com-gin-gonic-gin/ $GOPATH/src/github.com/gin-gonic/gin/
mkdir -p $GOPATH/src/github.com/go-gremlin/gremlin && rsync -aP ~/.glide/cache/src/https-github.com-go-gremlin-gremlin/ $GOPATH/src/github.com/go-gremlin/gremlin/
mkdir -p $GOPATH/src/github.com/golang/protobuf && rsync -aP ~/.glide/cache/src/https-github.com-golang-protobuf/ $GOPATH/src/github.com/golang/protobuf/
mkdir -p $GOPATH/src/github.com/gorilla/websocket && rsync -aP ~/.glide/cache/src/https-github.com-gorilla-websocket/ $GOPATH/src/github.com/gorilla/websocket/
mkdir -p $GOPATH/src/github.com/kr/beanstalk && rsync -aP ~/.glide/cache/src/https-github.com-kr-beanstalk/ $GOPATH/src/github.com/kr/beanstalk/
mkdir -p $GOPATH/src/github.com/manucorporat/sse && rsync -aP ~/.glide/cache/src/https-github.com-manucorporat-sse/ $GOPATH/src/github.com/manucorporat/sse/
mkdir -p $GOPATH/src/github.com/miekg/dns && rsync -aP ~/.glide/cache/src/https-github.com-miekg-dns/ $GOPATH/src/github.com/miekg/dns/
mkdir -p $GOPATH/src/github.com/satori/go.uuid && rsync -aP ~/.glide/cache/src/https-github.com-satori-go.uuid/ $GOPATH/src/github.com/satori/go.uuid/
mkdir -p $GOPATH/src/golang.org/x/net && rsync -aP ~/.glide/cache/src/https-golang.org-x-net/ $GOPATH/src/golang.org/x/net/
mkdir -p $GOPATH/src/gopkg.in/go-playground/validator.v8 && rsync -aP ~/.glide/cache/src/https-gopkg.in-go-playground-validator.v8/ $GOPATH/src/gopkg.in/go-playground/validator.v8/
mkdir -p $GOPATH/src/gopkg.in/yaml.v2 && rsync -aP ~/.glide/cache/src/https-gopkg.in-yaml.v2/ $GOPATH/src/gopkg.in/yaml.v2/

Don't worry too much about it though. I just wanted to bring this to your attention so you know this doesn't always work out but I'm not really trying to make it work for this particular project. It's an old project I'm already rewriting in another language.

[kamilchm]

It must be something with -I {} in xargs. I follow dep and there's a good news https://github.com/golang/dep/pull/500 - one more step to unify go dependency management. I will probably drop all issues with current go2nix and start working on translating gopkg.lock into nix derivation.

[rushmorem]

I follow dep and there's a good news golang/dep#500 - one more step to unify go dependency management.

Awesome.

[copumpkin]

Another vote for this, since a package I use uses glide for dependency management and I'd like to put it in nixpkgs.

[CMCDragonkai]

What about other package managers like dep or gx? How does go2nix deal with all of these?

[kamilchm]

There's a new thing for dep -> https://github.com/nixcloud/dep2nix. You could try that. As for other package tools/managers, I think that we should stick with dep to generate nix derivations and use or write new importers for it https://github.com/golang/dep/tree/master/internal/importers

[CMCDragonkai]

There's also discussion about the new go package management on HN as well. What if that becomes official?