markcampanelli
commented
6 years ago I hit same issue when trying to use dep to manage some non-Go build tools.
Closed davecheney closed 5 years ago
markcampanelli
commented
6 years ago I hit same issue when trying to use dep to manage some non-Go build tools.
markuswustenberg
commented
6 years ago This also happens when fetching e.g. github.com/docker/docker, which doesn't have Go code in the root (since v0.9.1 at least):
$ dep ensure -v -add github.com/docker/docker
Fetching sources...
(1/1) github.com/docker/docker
Root project is "code.uber.internal/infra/dep-no-go-code"
1 transitively valid internal packages
1 external packages imported from 1 projects
(0) ✓ select (root)
(1) ? attempt github.com/docker/docker with 1 pkgs; 206 versions to try
(1) try github.com/docker/docker@v1.13.1
(2) ✗ github.com/docker/docker at v1.13.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.13.0
(2) ✗ github.com/docker/docker at v1.13.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.12.6
(2) ✗ github.com/docker/docker at v1.12.6 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.12.5
(2) ✗ github.com/docker/docker at v1.12.5 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.12.4
(2) ✗ github.com/docker/docker at v1.12.4 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.12.3
(2) ✗ github.com/docker/docker at v1.12.3 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.12.2
(2) ✗ github.com/docker/docker at v1.12.2 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.12.1
(2) ✗ github.com/docker/docker at v1.12.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.12.0
(2) ✗ github.com/docker/docker at v1.12.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.11.2
(2) ✗ github.com/docker/docker at v1.11.2 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.11.1
(2) ✗ github.com/docker/docker at v1.11.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.11.0
(2) ✗ github.com/docker/docker at v1.11.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.10.3
(2) ✗ github.com/docker/docker at v1.10.3 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.10.2
(2) ✗ github.com/docker/docker at v1.10.2 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.10.1
(2) ✗ github.com/docker/docker at v1.10.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.10.0
(2) ✗ github.com/docker/docker at v1.10.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.9.1
(2) ✗ github.com/docker/docker at v1.9.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.9.0
(2) ✗ github.com/docker/docker at v1.9.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.8.3
(2) ✗ github.com/docker/docker at v1.8.3 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.8.2
(2) ✗ github.com/docker/docker at v1.8.2 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.8.1
(2) ✗ github.com/docker/docker at v1.8.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.8.0
(2) ✗ github.com/docker/docker at v1.8.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.7.1
(2) ✗ github.com/docker/docker at v1.7.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.7.0
(2) ✗ github.com/docker/docker at v1.7.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.6.2
(2) ✗ github.com/docker/docker at v1.6.2 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.6.1
(2) ✗ github.com/docker/docker at v1.6.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.6.0
(2) ✗ github.com/docker/docker at v1.6.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.5.0
(2) ✗ github.com/docker/docker at v1.5.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.4.1
(2) ✗ github.com/docker/docker at v1.4.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.4.0
(2) ✗ github.com/docker/docker at v1.4.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.3.3
(2) ✗ github.com/docker/docker at v1.3.3 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.3.2
(2) ✗ github.com/docker/docker at v1.3.2 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.3.1
(2) ✗ github.com/docker/docker at v1.3.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.3.0
(2) ✗ github.com/docker/docker at v1.3.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.2.0
(2) ✗ github.com/docker/docker at v1.2.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.1.2
(2) ✗ github.com/docker/docker at v1.1.2 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.1.1
(2) ✗ github.com/docker/docker at v1.1.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.1.0
(2) ✗ github.com/docker/docker at v1.1.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.0.1
(2) ✗ github.com/docker/docker at v1.0.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v1.0.0
(2) ✗ github.com/docker/docker at v1.0.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v0.12.0
(2) ✗ github.com/docker/docker at v0.12.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v0.11.1
(2) ✗ github.com/docker/docker at v0.11.1 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v0.11.0
(2) ✗ github.com/docker/docker at v0.11.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v0.10.0
(2) ✗ github.com/docker/docker at v0.10.0 has problem subpkg(s):
(2) github.com/docker/docker has err (*build.NoGoError); required by (root).
(1) try github.com/docker/docker@v0.9.1
(1) ✓ select github.com/docker/docker@v0.9.1 w/1 pkgsVersion:
$ dep version
dep:
version : devel
build date :
git hash :
go version : go1.9.1
go compiler : gc
platform : darwin/amd64
sdboyer
commented
6 years ago yeah...this is tricky. i'm considering the possibility that we change the rules on required to not complain if the named packages lack Go code. alternatively, we could relax the "no go code" rule in general and consider any files that may possibly be usable to Go good enough to make a package "valid" for import. that would open up the possibility of false positives with real imports...but that seems a lot less likely to be a real problem than the false negatives being described here.
the only other alternative i can think of right now is something like #269, which is a whole other can of worms.
@markuswustenberg
This also happens when fetching e.g. github.com/docker/docker, which doesn't have Go code in the root (since v0.9.1 at least):
yeah, the design here implicitly expects that you give -add a package - not necessarily a project root - that is actually importable. this is one of those confounded expectations arising from bumping up against the model, again - folks are thinking, "i want to grab this down and just have it there so i can reference stuff from it," but dep's model is "you don't actually use these, so we're gonna get rid of it."
this is gonna be a point of friction for a while, unfortunately. the easiest way to make e.g. your editor's autocomplete work is to just trick it by also having that project on GOPATH - and by the time that we get around to adding the new storage area, loosely outlined here, i think most of this pain should go away. but it's gonna be a while.
markuswustenberg
commented
6 years ago Thanks for your answer @sdboyer.
alternatively, we could relax the "no go code" rule in general and consider any files that may possibly be usable to Go good enough to make a package "valid" for import.
In my view, any arbitrary binary is potentially usable by Go code, so that's seems a bit of an odd restriction. Maybe there could just be an option to override this behaviour?
yeah, the design here implicitly expects that you give -add a package - not necessarily a project root - that is actually importable
In this case, I'm trying to grab the docker client which is under github.com/docker/docker/client, but dep won't let me import that directly. Is there any way to get this working with the current tools?
sdboyer
commented
6 years ago In my view, any arbitrary binary is potentially usable by Go code, so that's seems a bit of an odd restriction.
the key here is the definition of "usable" - dep is, for now, primarily concerned with what is necessary to compile Go code. not generate; not run. it's nice when we can also capture those things, but for now, they're not our primary objective. that puts arbitrary binaries out of scope, unless i've missed something quite big about extensible compiler capabilities. (entirely possible!)
In this case, I'm trying to grab the docker client which is under github.com/docker/docker/client, but dep won't let me import that directly. Is there any way to get this working with the current tools?
dep ensure -add github.com/docker/docker/client
discordianfish
commented
6 years ago I have a related issue. Not sure if I should open a new issue for that though:
I'm building a Dockerfile and I want to add Gopkg.* first, then run dep ensure before I add the code in a next step. That's a common pattern which allows Docker to cache the dependency-install step, so that I don't have to run this every time I change the code. Due to the 'no go code' rule this isn't possible when using dep.
sdboyer
commented
6 years ago @discordianfish this issue is specifically about the case where there's no Go code in a dependency, which is different from having no Go code in the current project. dep ensure -vendor-only should cover your case.
discordianfish
commented
6 years ago Oh perfect, thanks! Sorry for the noise.
davecheney
commented
6 years ago Have a look at github.com/heptio/contour. I’m using the pattern you describe and it is working well.
On 11 Nov 2017, at 01:40, discordianfish notifications@github.com wrote:
Oh perfect, thanks! Sorry for the noise.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
jhayotte
commented
6 years ago Hi @sdboyer, I tried out to migrate from Glide to Dep and dependencies including a list of protofile like https://github.com/googleapis/googleapis/tree/master/google/api cannot be vendored.
Would be nice to have a workaround in the meantime. Many people are trying to use dep and are blocked by this issue :/
sdboyer
commented
6 years ago yeah, i don't see any problems with expanding the definition of required to cover this case. so we have a path to a solution - just, need the elbow grease to get it done.
krisnova
commented
6 years ago Happy to lend some commits while I am enjoying my time of being unemployed if you want to assign this or any other issues to me
jhayotte
commented
6 years ago Hi @sdboyer,
I can contribute to this issue as well. We just need to clarify your expectations.
Do you wish that we remove the constraint looking for *.go files? (IMO that's what we should do) or do we handle it as a warning?
travisjeffery
commented
6 years ago Here's a PR for it https://github.com/golang/dep/pull/1398. Feedback welcome, thanks.
sdboyer
commented
6 years ago i'm delighted to see all the offers to help ❤️ ❤️ this is how these things get done!
but yeah, lemme clarify the requirements here.
we can't mess with ListPackages() in the way @jhayotte has outlined. that's fixing this problem by lying about the state of the world, and ListPackages() needs to not do that. we can't change what it reports in order to achieve a desired outcome in the solver; instead, we need to change the solver to deal with this more appropriately.
to the specifics. right now, the behavior of the solver is that:
required list from Gopkg.toml, or if it's in an import statement (we do not currently differentiate between these two cases)what we want to do here is tease apart actual import statements from Gopkg.toml's required. the behavior for imports should remain the same, but:
required list from Gopkg.tomlthis achieves the outcome i described in an earlier comment: https://github.com/golang/dep/issues/1306#issuecomment-339864942
there's going to be more work to do here. in addition to likely needing to update some of gps' tests, we'll want at least one new one to cover this particular path (and probably a few to cover combinations of conditions). crucially, we're also going to need to change how the hasherdoes its work, as we're changing the semantics of imports and required's so that they're no longer identical: we'll need to break them out into separate groups.
this change will also then entail a bump o the solver-version that you see in Gopkg.lock, which we'll have to do a little bit of footwork around, as i don't think we've accommodated that in dep yet.
johanbrandhorst
commented
6 years ago Lots of offers of work but I see no open PRs? Is anyone working on this at the moment? It's a blocker for our transition (and I unfortunately do not have spare cycles at this time 😞 ).
sdboyer
commented
6 years ago nope, nobody is on this at the moment, unfortunately. it involves solver changes, so it's a rather tricky task to take on.
carolynvs
commented
6 years ago I am working on this, there was a lot of overlap between the changes needed for this and the transitive constraint prototype so I had a head start. 😀
dt665m
commented
6 years ago Is there currently a work around for npulling non .go file dependencies? I have some cgo and some dynamic c++ libs that aren't getting pulled by dep. Is there any way to force dep to pull the whole repository? (in this case, github).
carolynvs
commented
6 years ago AFAIK, if the repo doesn't contain go code anywhere, there is no workaround yet. It is on our backlog to address. However if there is at least one directory with go code, then you can require that package from the repo, and then don't use pruning of non-go files or unused packages so that the entire repo is vendored.
zhexuany
commented
6 years ago @sdboyer I hit a confusing error.
In a project which depends on our one project tidb, I ran dep ensure and dep gave me the following error:
Solving failure: No versions of github.com/pingcap/tidb met constraints:
master: Could not introduce github.com/pingcap/tidb@master due to multiple problematic subpackages:
Subpackage github.com/pingcap/tidb/sessionctx does not contain usable Go code (*build.NoGoError).. (Package is required by (root).) Subpackage github.com/pingcap/tidb/store/mockstore is missing. (Package is required by (root).) Subpackage github.com/pingcap/tidb/session is missing. (Package is required by (root).)The thing is subpackage session and sessionctx contains useable go code. Could you please help m out?
session's link: https://github.com/pingcap/tidb/tree/master/session sessionctx's link: https://github.com/pingcap/tidb/tree/master/sessionctx
jackman0
commented
6 years ago FYI, this is also an issue with https://github.com/go-qml/qml. This package depends on C++ files located in a subdirectory.
hamid-elaosta
commented
6 years ago @carolynvs This is an interesting point, because what you suggest above does not appear to be the behaviour, perhaps I'm missing something.
I have a Go project A, which I'm trying to vendor into another Go project B. From A I am using only a Proto Buf .proto file, which resides in a subpackage of A. I am vendoring the top-level package, not the sub-package, and I have no pruning or other cleanup/removal configured in my toml, yet dep insists it cannot vendor A because its sub-package (the one I'm getting the proto file from) does not contain Go code.
Is there an override I can use to vendor the entirety of project A? I had assumed vendoring its top-level would vendor it entirely, regardless if I'm using only a file from a sub-package in my B project.
carolynvs
commented
6 years ago @hamid-elaosta Here is an example of how I vendored the k8's codegen repos:
hamid-elaosta
commented
6 years ago @carolynvs Thanks for the links. Unfortunately neither seems to work (and dep complains about redundant prune options) I'm not sure if it's some combination of my project requirements that is preventing it but the error message didn't change.
The project I'm attempting to vendor is on a specific branch (non-master), the constraint is overridden with an SSH path because it's a private (corporate) git server, and my local copy was not go-g[o]t but git cloned (because go-get won't check out over ssh from our git server).
EDIT: I should add, dep ensure worked when I had a go file in the sub-package that contains the proto, but it was removed in a recent update to that project (it's now generated after checkout).
tangblue
commented
6 years ago Hit the same issue when trying to use dep to get HTML static file from another project.
mysugar
commented
6 years ago I tried to use dep to get github.com/spinlock/jemalloc-go which is a c project.
Then i update the Gopkg.toml file , here is my file
# [prune]
# non-go = false
# go-tests = true
# unused-packages = true
required = ["github.com/spinlock/jemalloc-go"]
[[override]]
name = "github.com/spinlock/jemalloc-go"
revision = "26719b2ee6186ef794cd50b604f8d765d3be5a30"
[prune]Then i got all files in my vendor dir. So the config worked for me.Hope it can help others.
My dep version is 0.4.1.
ZhenhangTung
commented
6 years ago @zhexuany I think your problem is not related to this issue, and you could open a new issue or reach out to the community if this problem still bothers you.
The thing is subpackage session and sessionctx contains useable go code. Could you please help m out?
The context.go file in sessionctx package was created in this commit, which is on 22 Feb, so not sure if you was importing the version before this commit, since you didn't give us any conext or logs when dep was running.
From my understanding, useable go code doen't mean that you have some subpackages which contains go files, it means that there should be go files under this package.
Hope this could help you.
msolimans
commented
6 years ago tried to avoid pruning non-go files but did not work by just turning it off using non-go = false however removing root prune at all was good enough to do the magic for me and to include everything I want. didn't like to include unused-packages too but at least it is working for now
tico8
commented
5 years ago dependencies that do not have Go source code
My case was solved by following options. I wanted to get github.com/gogo/protobuf(do not have Go source) into "vender/".
[prune]
go-tests = true
unused-packages = true
non-go = true
[[prune.project]]
name = "github.com/gogo/protobuf"
non-go = false
unused-packages = false
truongnmt
commented
5 years ago Solution from @tico8 works! However I wonder if I dep ensure, that changes will be lost?
ruijundeng
commented
5 years ago @carolynvs I am following how you construct your Gopkg.toml file for k8s dependencies. However, I still encountered this problem while trying to get k8s.io/client-go, k8s.io/api, k8s.io/apimachinery.
My Dep Version:
dep: version : 0.5.0 build date : git hash : go version : go1.11.4 go compiler : gc platform : linux/amd64 features : ImportDuringSolve=false
My Gopkg.toml (I've exhausted every combination, with or without prune):
required = ["k8s.io/client-go"]
[[constraint]]
name = "k8s.io/client-go"
version = "kubernetes-1.11.1"
[prune]
go-tests = true
unused-packages = true
[[prune.project]]
name = "k8s.io/client-go"
non-go = false
unused-packages = false Run Dep ensure -v I got:
kubernetes-1.11.1: Could not introduce k8s.io/client-go@kubernetes-1.11.1, as its subpackage k8s.io/client-go does not contain usable Go code (*build.NoGoError).. (Package is required by (root).)
What version of
depare you using (dep version)?What
depcommand did you run?I am trying to convert a project that uses glide to vendor two repositories containing protocol buffer definitions. The abstract from
glide.yamlisThe two repositories in question are
github.com/envoyproxy/data-plane-apiandgithub.com/googleapis/googleapis. When I try to adddep ensure refuses to vendor the repositories
What did you expect to see?
those two repositories vendored into
vendor/What did you see instead?