Do not implicitly "Resolving imports" of the workspace

[mhoglan]

This on master glide version 0.9.0-rc1

This seems to occur on both glide get and glide update

This may be a misunderstanding or a better way to use the tool, feel free to educate me :)

Why when I perform a glide get <package> is there anything more being done than just retrieving the package and the package dependencies?

Why when I perform a glide update is there anything more being done than executing the retrieval of the dependencies in the glide.yaml and their transitive dependencies?

Do not inspect the current workspace *.go files and then try to also go get their imports and dependencies. If I want my workspace evaluated for imports and dependencies, I should have it be an explicit action. Or vice versa, let me turn off the implicit action that occurs with these commands.

Inspection of the current workspace is what should be done to create the glide.yaml. I know glide create is just guessing and is not a thorough analysis. Maybe we need this thorough part that is occurring be a separate command.

Reason why this is causing me such a headache, I am trying to take an existing project and apply glide to it. Either due to a problem with cycle dependencies / mismatches (highly likely) or a bug in glide (still digging... trying to isolate to a smaller use case), I cannot get my project buildable. Its buildable with Godep and while I could import that file over, I am wanting to cleanup the mess those files are by doing it from scratch. Because each of the commmands inspects the workspace, there is not a way to incrementally add / delete items from the glide.yaml or vendor directory to isolate down which dependency is causing the error and conflict.

I am looking through glide list and glide tree to see if they provide me help in it. Either way though, I do think the implicit action should be separate or controlled via a flag.

[mattfarina]

@mhoglan There are a few things going on here.

Are you familiar with package management in other languages? Such as Rust, Java, PHP, Python, JavaScript, Perl, or others. Glide is modeled after the best and successful practices from these so I'm curious of your foundation in them.

• glide create isn't just guessing. It's inspecting the imports in the local project. It also reads information from Godep, GB, and GPM to aide it.
• glide get retrieves a package and its dependencies. Transitive dependencies need to be resolved for their versions across everything else. They are resolved within the version constraints.
• glide update walks the dependencies and transitive dependencies to find the latest version that matches the version constraints. If there are no constraints it uses the tip of the default branch.

Can you share the project your having difficulty with. I can take a look at it and see what's going on.

[mhoglan]

I am pretty familiar with packaging systems across different languages and systems. That is where some of the friction with my usage was coming from. Lets use pip as an example. If I do a pip install <package> it installs that package and its dependencies into my site-packages, it does not go through my source code of my workspace and process its dependencies in conjunction.

Going to use the stringsvc1 example from go-kit as an example here

root@7982ce95e3e7:~# go get -d github.com/go-kit/kit/examples/stringsvc1
root@7982ce95e3e7:~# cd /go/src/github.com/go-kit/kit/examples/stringsvc1/
root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# glide create
[INFO] Generating a YAML configuration file and guessing the dependencies
[INFO] Attempting to import from other package managers (use --skip-import to skip)
[INFO] Found reference to github.com/go-kit/kit/endpoint
[INFO] Adding sub-package transport/http to github.com/go-kit/kit
[INFO] Found reference to golang.org/x/net/context
root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# cat glide.yaml
package: github.com/go-kit/kit/examples/stringsvc1
import:
- package: github.com/go-kit/kit
  subpackages:
  - endpoint
  - transport/http
- package: golang.org/x/net
  subpackages:
  - context

I started with a clean workspace. Created the glide.yaml file with glide create which determined the dependencies by looking at the source code. I assumed that it was a best effort since it said it was guessing.

I have not created my vendor directory yet. Doing this on purpose for this exercise As my problem I was having in my project was causing an error during the glide update in creating the vendor directory.

I would expect though to be able to add a new dependency via glide get and populate the vendor package with it.

Lets go get the goamz package

root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# glide get github.com/goamz/goamz
[INFO] Preparing to install 1 package.
[INFO] Importing github.com/goamz/goamz
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/go-kit/kit.
[INFO] Fetching updates for golang.org/x/net.
[INFO] Fetching updates for github.com/goamz/goamz.
[INFO] Resolving imports
[INFO] Fetching updates for golang.org/x/net.
[INFO] Fetching updates for github.com/go-kit/kit.
[INFO] Fetching gopkg.in/logfmt.v0 into /go/src/github.com/go-kit/kit/examples/stringsvc1/vendor
[INFO] Fetching gopkg.in/stack.v1 into /go/src/github.com/go-kit/kit/examples/stringsvc1/vendor
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/goamz/goamz.

Starts fine. I expect that it would have downloaded the dependencies for goamz. But then it goes and fetches all the dependencies that are in the glide.yaml. I feel it should have just updated the glide.yaml with a new dependency and ended there.

Suspected this is occurring because it wants to update the glide.lock file to have the new package and transitive dependencies pinned. Since I have not done a glide update to cause a glide.lockto be generated, maybe it was triggering a glide update or respective behavior.

Here though we can see that even with a populated vendor directory and a generated glide.lock file, it will still go through and fetch and analyze dependencies after getting the package.

root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# rm -rf glide.yaml glide.lock vendor/

root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# glide create
[INFO] Generating a YAML configuration file and guessing the dependencies
[INFO] Attempting to import from other package managers (use --skip-import to skip)
[INFO] Found reference to github.com/go-kit/kit/endpoint
[INFO] Adding sub-package transport/http to github.com/go-kit/kit
[INFO] Found reference to golang.org/x/net/context

root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# glide update
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/go-kit/kit.
[INFO] Fetching updates for golang.org/x/net.
[INFO] Resolving imports
[INFO] Fetching gopkg.in/logfmt.v0 into /go/src/github.com/go-kit/kit/examples/stringsvc1/vendor
[INFO] Fetching gopkg.in/stack.v1 into /go/src/github.com/go-kit/kit/examples/stringsvc1/vendor
[INFO] Downloading dependencies. Please wait...
[INFO] Setting references for remaining imports
[INFO] Project relies on 4 dependencies.

root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# glide get github.com/goamz/goamz
[INFO] Preparing to install 1 package.
[INFO] Importing github.com/goamz/goamz
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/go-kit/kit.
[INFO] Fetching updates for golang.org/x/net.
[INFO] Fetching updates for github.com/goamz/goamz.
[INFO] Resolving imports
[INFO] Fetching updates for golang.org/x/net.
[INFO] Fetching updates for github.com/go-kit/kit.
[INFO] Fetching updates for gopkg.in/logfmt.v0.
[INFO] Fetching updates for gopkg.in/stack.v1.
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/goamz/goamz.

It is this "Resolving Imports" stage that I am referring to should not be implicitly done.

Similar behavior with glide update

root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# rm -rf glide.yaml glide.lock vendor/
root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# glide create
[INFO] Generating a YAML configuration file and guessing the dependencies
[INFO] Attempting to import from other package managers (use --skip-import to skip)
[INFO] Found reference to github.com/go-kit/kit/endpoint
[INFO] Adding sub-package transport/http to github.com/go-kit/kit
[INFO] Found reference to golang.org/x/net/context
root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# cat glide.yaml
package: github.com/go-kit/kit/examples/stringsvc1
import:
- package: github.com/go-kit/kit
  subpackages:
  - endpoint
  - transport/http
- package: golang.org/x/net
  subpackages:
  - context

Lets remove the golang.org/x/net dependency

root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# vi glide.yaml
root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# cat glide.yaml
package: github.com/go-kit/kit/examples/stringsvc1
import:
- package: github.com/go-kit/kit
  subpackages:
  - endpoint
  - transport/http

When I do a glide update, it is still going to go get the glang.org/x/net dependency due to the "Resolving Imports" stage.

root@7982ce95e3e7:/go/src/github.com/go-kit/kit/examples/stringsvc1# glide update
[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/go-kit/kit.
[INFO] Resolving imports
[INFO] Fetching golang.org/x/net/context into /go/src/github.com/go-kit/kit/examples/stringsvc1/vendor
[INFO] Fetching gopkg.in/logfmt.v0 into /go/src/github.com/go-kit/kit/examples/stringsvc1/vendor
[INFO] Fetching gopkg.in/stack.v1 into /go/src/github.com/go-kit/kit/examples/stringsvc1/vendor
[INFO] Downloading dependencies. Please wait...
[INFO] Setting references for remaining imports
[INFO] Project relies on 4 dependencies.

glide update to me is analogous of process my requirements.txt and populate my site-packages if coming from the pip world of things.

Hope that clarifies some of what I am seeing and thinking. Not disagreeing with the ideas or practices that glide is based on, I agree with them and is one of the reasons I have come to use the tool from the others out there and I have liked what I have seen.

In regards to my problem in my project, it did turn out to be a transitive dependency mismatch issue, just was rearing its head in a very ugly way because the change between the two versions of the dependency was that the dependency vendored one of its dependencies into an "internal" folder. The latest version of that dependency was causing glide to look in the vendor folder for a directory that did not exist since in the vendor directory that dependency instance was an older version.

github.com/prometheus/common was the package that triggered all this.

I was trying to incrementally build my vendor directory and glide.yaml file so I can see where the trigger for this was occurring.

[technosophos]

First, this may help you understand one thing about Glide: Any glide operation from glide create onward should leave your environment in a buildable state, and should leave your glide.lock file accurate. To make it otherwise would be poor design. Users would quickly become frustrated.

The resolution logic for glide get updates other packages so that the version analysis and pinning can be done correctly. Before doing version analysis, we want to make sure that all of the repos are at their latest, otherwise we may end up with dependencies that are referencing repo tags that do not exist in the present tree. (Keep in mind that for any given package, N other packages may depend on it, and may stipulate particular version requirements.)

This differs from other package managers because we are always working on VCS repos, not packaged distributions. So scanning VCS repos is a common and necessary case when learning about the version of any given package.

Secondarily, we can (and must) updated the glide.lock for any glide.get operation. Otherwise the lock file is out of sync with your explicitly declared dependencies, which is precisely the situation we want to avoid. So more commands than glide up modify the lock file.

[mhoglan]

Probably is more of a philosophical or design decision discussion. I understand and follow the points you made, and the implementation choices based on those. There is a desire to keep everything in a consistent buildable state. I wouldn't call it a poor or good design choice, it is a goal, and one of convenience.

Some thoughts. Not advocating a certain viewpoint, but asking to raise thoughts, even if contrarian.

Glide is a dependency / vendor management tool. It is nice that the end result of running my dependency management tool is that my workspace is in a state that will build with go build.

However is this ensurance the job of a dependency management tool? The management of the glide.yaml, glide.lock and vendor artifacts and the ensurance that they are processed in a deterministic way and stay in a consistent state seems to be the primary purpose. If the end result of that execution is that the workspace is in a state that one can just execute go build without any externals, then that is a result of the input (glide.yaml and glide.lock) being defined in a way that created that state.

I would use this tool in a way to provide ensurance of a deterministic buildable state. Those ways on top of the tool are convenience and it is nice to have some of those conveniences come with the tool.

There can be both ways. Using git terminology, the plumbing and the porcelain.

That is ultimately what I am bringing up here. There should be some separation of the two. Whether that is additional commands or flags, either way, but the ability to do so.

• glide.yaml is a way to specify a list of dependencies for a project. Also provide capabilities to satisfy use cases that typically require workarounds or are tedious to perform, such as the ability to alias a dependency to a different origin (such as forks).
• glide.lock is a way to specify a terse definition of the dependencies specified in glide.yaml and their transitive dependencies. Satisfy the pinning use case and the ability to generate a vendor directory in a consistent fashion.
• vendor is the golang mechanism to provide a precedence over what is present in the GOPATH; essentially it is the projects own isolated version of GOPATH/src

I can see the need for those 3 items to stay consistent with each other, but why does the state of my workspace influence that? There is now the requirement that a workspace must be in a pristine state for the tool to work. The workspace is now influencing

If we take the case of the glide update I did in the previous post where I remove a dependency golang.org/x/net from the glide.yaml; I do not end up in a consistent state among the 3 items in that case. The tool went off and downloaded golang.org/x/net to my vendor directory and added it to the glide.lock file but that was not added in my glide.yaml; Not saying glide.yaml should be updated based off the analysis of transitive dependencies, but it is suppose to be representative of the external dependencies required of the workspace which I want to be eventually present in the vendor directory after using glide.

If I were to take the glide.yaml and analyze it by hand, tracing all the dependencies, I would never find golang.org/x/net in that graph. It is only because that glide.yaml existed in a source code workspace where a golang source file referenced it in an import did the tool go get it. If I executed the exact same same commands and glide.yaml in a different source workspace, I would get a different result, even though the input to the tool was the same.

Contrived use case, but what if I wanted a specific dependency to be pulled from the environment, from the GOPATH and everything else to come from the projects vendor directory. A site override like dependency that is populated based on environment. Cannot do that currently because the tool is requiring that the workspace be in a complete buildable state based on the glide.lock file.

Like I said earlier, there can be both worlds. Convenience ways that string together the usage of the tool. But the separation of the convenience and the plumbing.

Just tossing this out as an idea. I know there are existing commands and terminology etc... just showing this as example of the thoughts.

• glide analyze - Analyze the workspace and generate a list of external dependencies found in the workspace
  • maybe this is what glide list is doing, but I get a different list from that command than what glide create is currently doing
• glide create - Initialize a workspace for glide usage. Run glide analyze and use result to generate a glide.yaml
• glide update - Process the glide.yaml and generate a glide.lock file describing all the dependencies in a pinned fashion
• glide install - Process the glide.lock and populate the vendor directory

These commands can have flags (which could be defaulted to true) to ensure consistency / integrity amongst the files. Similar to what is specified in the notes on glide install and how it currently will error if the glide.lock file is out of sync with the glide.yaml;

There could be convenience commands for building the workspace and other wrappers around the go tools. Or the ability to string them together (like how glide novendor works).

---

Really can sum it up with this:

• I can be in an empty workspace, void of golang code, create a glide.yaml by hand, execute glide update and have a populated vendor directory and a glide.lock describing in a pinned fashion the dependencies present in the vendor directory.
  • This works today.
• I should get the exact same result (glide.lock and vendor directory) if I dropped a golang source file in the workspace and executed glide update (no changes were made to the glide.yaml);
  • This is not true today.

[adewes]

I'm also having issues with this approach: In my project, I import subpackages using a local naming scheme (e.g. my-org.com/my_project/my_package). These imports are local in the sense that they are satisfied by the repository itself. Now, when I run glide update it tries to resolve and download these dependencies, which obviously does not make any sense.

Coming from Python, the behavior is a bit surprising to me as I would simply expect that glide fetches the packages indicated in the glide.yaml file and installs them into the vendor directory, instead of automagically including imports from my code. Just my personal opinion though, as I'm not yet very experienced with the Go eco-system I might have a wrong understanding of this problem.

[sdboyer]

This is an interesting discussion, and one that i'd missed previously.

i'm fundamentally in agreement with @technosophos that the goal is to leave the repository in a buildable state, and that incorporating information from the local go source is a reasonable, often necessary part of doing that. But it does present, at minimum, possible issues when the local code is not itself in an analyzable state - what do we do then?

The biggest single issue with not performing analysis on the workspace is that that analysis gives us the starting points in a dependency analysis graph of packages, rather than projects. glide.yaml contains projects, but if we only look at projects, we often bring in more dependencies - many, many more, due to the Go community's fondness of public monorepos/"kit" repos - than we strictly need. Only by doing package-level analysis can we determine what's actually needed.

This is all problematic because, without those initial inputs to the package depgraph, we don't know which package import paths we actually need to follow.

• If our workspace lists project A as a dep in glide.yaml
• A contains A/foo and A/bar
• Our workspace imports ONLY A/foo
• A lists project B as a dep in glide.yaml
• ONLY A/bar imports a package from B

Without workspace analysis, we end up including B. With it, we know we don't need B. In part, this is a performance optimization, but it's costly enough that it was prohibitive of glide's use for people to not have it. Additionally, it opens up more possibility for conflicts - what does B depend on? Does that conflict with any of our other deps?

This whole area is an issue of significant disagreement within the community of people working on go package management. The expectation that we can "think" in packages, but "exchange" in repositories is at the heart of why analogies from other language package management contexts may not apply.

The basic issue with introducing more commands - @mhoglan's analogy to git's plumbing is apt - is that doing so inherently opens up new possible states that the program can exit in, which means new input states we have to be able to handle. That's more than just a drag - such input states can often be ambiguous with actual user input, which in turn means more command flags in order to get things to behave correctly...it becomes a UX-tanking spiral.

That said, towards the end of my writeup I described what I called a "sync-based PDM", in which we consider the relationship between each of the four on-disk states - workspace, manifest, lock, and dependencies - as discrete functions. I could see how a single flag that essentially says, "we don't care about workspace -> manifest sync for this op" could probably address the issues described here, without disrupting the integrity of the remainder of the system too much.

We'll have to see what such a thing looks like in the new engine (#384). Today, however, the --all-dependencies flag MIGHT cover some of your use case.

[shashankmehra]

I am a beginner at Go, and maybe my case is very specific, but I will leave it here. There are certain portions of my code which are generated, and the generator is also written in Go (go-swagger). That generator uses goimports to manage imports, so naturally it needs some go packages to be "discoverable". I thought of using glide alone to get those packages but when running glide install it throws a no such file or directory error when it tries to resolve imports. Obviously that file doesn't exist yet as it is yet to be generated using the generator for which I am installing dependencies, and I end up with a Catch-22 situation. I expected glide to work as pip would, as @mhoglan mentioned. I did not expect a dependency tool to check if my code is buildable. I had a separate dependency file for that.

Maybe keeping the generated code uncommitted in my repo is not a good idea. I am still experimenting with it. I am going to put go get commands in my build script for now just to initially bootstrap the generator. But I feel like being a developer tool it should not abstract out two tasks as one task. Maybe, let me run an install command with --skip-resolve flag and provide another command to make sure my project is ready to be built. That way I have full control of the tool.