-skip-vendor option to skip creating vendor directory

[dmitris]

Suppose you have a project with lots of dependencies that are already all installed in your GOPATH, and you want to generate the glide.lock file without creating a vendor directory. One use case would be to create glide.yaml file locally on the developer's workstation and then check it in for the CI build system to be used for the builds. It would save a lot of time in cases where vendor directory grows to hundreds of megabytes. Maybe there could be different behavior with "glide update" as currently and "glide update --skip-vendor" - generate glide.lock file based on the $GOPATH, skip creating the vendor directory? In this case, I think it would be acceptable and desirable to abort the operation if glide finds out dependencies that are not installed in the $GOPATH and therefore the full analysis is not possible.

[mattfarina]

Can you help me understand your use case a little more.

How would you handle two projects with a shared dependency needing different versions of it? Where you switch between working on the two on a regular basis.

The first time you run glide update it will install the dependencies into the vendor/ folder. On large codebases the first pass can take a couple minutes. I'm already working to speed that operation up as most of the slow points have opportunities to be faster. This has been tested on Kubernetes which is a rather large codebase.

After the first time it will be faster.

What is the issue? Is there too much disk space being used? Duplication of code on your system? Avoiding bandwidth on download?

Duplication of code allows each project to have its own version of dependencies without another project changing the versions.

[dmitris]

On large codebases the first pass can take a couple minutes

I've seen a project (internal one) where "glide up" takes 30-40 minutes and the resulting vendor folder is a few hundreds of MBs! :smile:

But even for smaller projects I would prefer to live without the "vendor" folder with all the dependencies "globbed" into the local folder. I don't regularly have to switch between the projects with very disparate sets of dependencies - quite the opposite. My development workflow is: in the normal add-test-fix-test daily grind mode, I'm happy with what I have in my GOPATH and just using "go install" and "go test". When I submit a PR, we have a Travis-like CI system that checks out from a clean slate, builds, run tests etc - that's where I would like to run "glide update", generate the vendor folder and use it. For the cases where I want to run the build as the CI system would do it, we have a Makefile that imitates the same action (building from scratch in a "GOPATH sandboxed" place etc.) - in that case I don't mind and expect slower builds.

I dislike having the vendor source tree locally under each project because it requires extra keystrokes to filter-grep vendored files when running tests, you have to update it every time one of your own libraries change (in every project that depends on it), it litters search results in the editor with irrelevant hits etc. - of course, there is a work-around for all of those but it seems rather unnecessary pain (IMO of course).

Here what would be the functionality of my "pie-in-the-sky-dream" dependency tool:

• "init" - quickly run through the GOPATH and record all the dependencies including transitive ones - like "glide init" does now but record all the packages with vcs, repo, and version taken directly from the GOPATH repositories. (Maybe this could be an option - to turn off all recursive version calculation based on Godeps or glide.yaml files in the "upstream" dependencies - just use what is currently in the GOPATH. ) So if I have github.com/foo/bar checked out from the "git@opensource.git.company.com:foo/bar" internal mirror, that value would be placed in glide.yaml ("repo: git@opensource.git.company.com"), the same for vcs and version. Currently I have to go manually and add those fields. Also If possible, I would use an option to turn off all recursive version calculation based on Godeps or glide.yaml in the dependencies - just use what is currently in the GOPATH.
• "update" - sync the state of the dependencies in the GOPATH into the 'manifest' (either glide.lock or glide.yaml). Personally, I would love to "glide update" or some other command to do the following with glide.yaml - update all the versions for packages but only where they are recorded. Probably an explanation is in order: I tend to think about glide.yaml as a kind of template where some pieces are left as "variables" that are to be filled during the "runtime" (when you create glide.lock and install). So I would like to use glide.yaml to fully pin down the external (3rd party / open-source) dependencies (with repo/vcs/version) but leave those for the internal packages "free" (no version recorded) so that the CI system would fetch the latest versions and I would not need to update the sha1 sum in the version field every time one of them changes. I would check in glide.yaml (the "template") for the CI and maybe keep glide.lock (complete list) generated by "glide update" for documentation of what exactly was used for the build.
• "install" - create glide.lock based on glide.yaml, setup the vendor directory based on glide.lock and download all the dependencies - I think exactly as is now
• "diff" - show the version differences between the state of the GOPATH and what is recorded in the glide manifest (glide.yaml and/or glide.lock).

It would also be great to have a command that would be a reverse of init - to sync your GOPATH with the manifest and make sure it has the same versions as recorded in glide files (after you checked with diff that it is what you want). So that if a teammate sent a new version of the manifest that has a couple of packages updated, you can quickly upgrade your GOPATH set without going into every directory :smile:

I hope it does not sound too crazy :smile: - I'm still trying to figure out the best mode of using the tool and the possibilities. I feel glide comes very close to the ideal dependency tool and has a real chance to fill the hole left by the go tool ("go dep")! For example, it allows you go reconcile the requirement of using only internal sources / mirrors and desire to use the original pathnames for "gopkg.in"-hosted packages - since you can "pin" the actual source to vcs+repo+version - :100: Thanks for all your great work on this!

[technosophos]

I'm not sure syncing between GOPATH and vendor is something that I'd want to bite off as a supported feature. Maybe a plugin for that would be an option.

We've got some logic to grab things out of GOPATH into vendor (using --cache-gopath). We also just re-did the import scanning, so it should be much faster (and pull far fewer dependencies).

I don't think we should close this issue, since there are some interesting ideas here that we need to think harder about.

And as a total aside... for the grep thing, a friend just pointed me to ag (Silver Searcher), which is a really cool grep-like search tool that obeys .gitignore (and hence doesn't search vendor). I thought that was pretty cool.