Issue initing, maybe due to circular dependencies

[jefferai]

What version of dep are you using (dep version)?

v0.3.0-248-gd3e738d

What dep command did you run?

dep init -v (see output below)

What did you expect to see?

Dep builds Gopkg.* files.

What did you see instead?

Getting direct dependencies...
Checked 233 directories for packages.
Found 69 direct dependencies.
Root project is "github.com/hashicorp/vault"
 112 transitively valid internal packages
 102 external packages imported from 69 projects
(0)   ✓ select (root)
(1) ? attempt github.com/armon/go-metrics with 3 pkgs; 1 versions to try
(1)     try github.com/armon/go-metrics@master
(1) ✓ select github.com/armon/go-metrics@master w/3 pkgs
(2) ? attempt github.com/armon/go-proxyproto with 1 pkgs; 1 versions to try
(2)     try github.com/armon/go-proxyproto@master
(2) ✓ select github.com/armon/go-proxyproto@master w/1 pkgs
(3) ? attempt github.com/armon/go-radix with 1 pkgs; 1 versions to try
(3)     try github.com/armon/go-radix@master
(3) ✓ select github.com/armon/go-radix@master w/1 pkgs
(4) ? attempt github.com/fullsailor/pkcs7 with 1 pkgs; 1 versions to try
(4)     try github.com/fullsailor/pkcs7@master
(4) ✓ select github.com/fullsailor/pkcs7@master w/1 pkgs
(5) ? attempt github.com/hashicorp/errwrap with 1 pkgs; 1 versions to try
(5)     try github.com/hashicorp/errwrap@master
(5) ✓ select github.com/hashicorp/errwrap@master w/1 pkgs
(6) ? attempt github.com/duosecurity/duo_api_golang with 2 pkgs; 1 versions to try
(6)     try github.com/duosecurity/duo_api_golang@master
(6) ✓ select github.com/duosecurity/duo_api_golang@master w/2 pkgs
(7) ? attempt github.com/hashicorp/go-sockaddr with 1 pkgs; 1 versions to try
(7)     try github.com/hashicorp/go-sockaddr@master
(7) ✓ select github.com/hashicorp/go-sockaddr@master w/1 pkgs
(8) ? attempt github.com/go-errors/errors with 1 pkgs; 1 versions to try
(8)     try github.com/go-errors/errors@master
(8) ✓ select github.com/go-errors/errors@master w/1 pkgs
(9) ? attempt github.com/jefferai/jsonx with 1 pkgs; 1 versions to try
(9)     try github.com/jefferai/jsonx@master
(9) ✓ select github.com/jefferai/jsonx@master w/1 pkgs
(10)  ? attempt github.com/mitchellh/copystructure with 1 pkgs; 1 versions to try
(10)      try github.com/mitchellh/copystructure@master
(10)  ✓ select github.com/mitchellh/copystructure@master w/1 pkgs
(11)  ? attempt github.com/hashicorp/go-plugin with 1 pkgs; 1 versions to try
(11)      try github.com/hashicorp/go-plugin@master
(11)  ✓ select github.com/hashicorp/go-plugin@master w/1 pkgs
(12)  ? attempt github.com/hashicorp/go-rootcerts with 1 pkgs; 1 versions to try
(12)      try github.com/hashicorp/go-rootcerts@master
(12)  ✓ select github.com/hashicorp/go-rootcerts@master w/1 pkgs
(13)  ? attempt github.com/golang/protobuf with 1 pkgs; 1 versions to try
(13)      try github.com/golang/protobuf@master
(13)  ✓ select github.com/golang/protobuf@master w/1 pkgs
(14)  ? attempt github.com/mitchellh/go-testing-interface with 1 pkgs; 1 versions to try
(14)      try github.com/mitchellh/go-testing-interface@master
(14)  ✓ select github.com/mitchellh/go-testing-interface@master w/1 pkgs
(15)  ? attempt github.com/mitchellh/go-homedir with 1 pkgs; 1 versions to try
(15)      try github.com/mitchellh/go-homedir@master
(15)  ✓ select github.com/mitchellh/go-homedir@master w/1 pkgs
(16)  ? attempt github.com/hashicorp/go-uuid with 1 pkgs; 1 versions to try
(16)      try github.com/hashicorp/go-uuid@master
(16)  ✓ select github.com/hashicorp/go-uuid@master w/1 pkgs
(17)  ? attempt github.com/hashicorp/go-multierror with 1 pkgs; 1 versions to try
(17)      try github.com/hashicorp/go-multierror@master
(17)  ✓ select github.com/hashicorp/go-multierror@master w/1 pkgs
(18)  ? attempt gopkg.in/mgo.v2 with 1 pkgs; 1 versions to try
(18)      try gopkg.in/mgo.v2@v2
(18)  ✓ select gopkg.in/mgo.v2@v2 w/5 pkgs
(19)  ? attempt google.golang.org/api with 2 pkgs; 2 versions to try
(19)      try google.golang.org/api@master
(19)  ✓ select google.golang.org/api@master w/3 pkgs
(20)  ? attempt golang.org/x/sys with 1 pkgs; 1 versions to try
(20)      try golang.org/x/sys@master
(20)  ✓ select golang.org/x/sys@master w/1 pkgs
(21)  ? attempt github.com/mitchellh/reflectwalk with 1 pkgs; 1 versions to try
(21)      try github.com/mitchellh/reflectwalk@master
(21)  ✓ select github.com/mitchellh/reflectwalk@master w/1 pkgs
(22)  ? attempt github.com/ghodss/yaml with 1 pkgs; 2 versions to try
(22)      try github.com/ghodss/yaml@v1.0.0
(22)  ✓ select github.com/ghodss/yaml@v1.0.0 w/1 pkgs
(23)  ? attempt gopkg.in/yaml.v2 with 1 pkgs; 1 versions to try
(23)      try gopkg.in/yaml.v2@v2
(23)  ✓ select gopkg.in/yaml.v2@v2 w/1 pkgs
(24)  ? attempt github.com/hashicorp/go-syslog with 1 pkgs; 1 versions to try
(24)      try github.com/hashicorp/go-syslog@master
(24)  ✓ select github.com/hashicorp/go-syslog@master w/1 pkgs
(25)  ? attempt github.com/chrismalek/oktasdk-go with 1 pkgs; 2 versions to try
(25)      try github.com/chrismalek/oktasdk-go@master
(25)  ✓ select github.com/chrismalek/oktasdk-go@master w/1 pkgs
(26)  ? attempt github.com/google/go-querystring with 1 pkgs; 1 versions to try
(26)      try github.com/google/go-querystring@master
(26)  ✓ select github.com/google/go-querystring@master w/1 pkgs
(27)  ? attempt github.com/hashicorp/go-cleanhttp with 1 pkgs; 1 versions to try
(27)      try github.com/hashicorp/go-cleanhttp@master
(27)  ✓ select github.com/hashicorp/go-cleanhttp@master w/1 pkgs
(28)  ? attempt golang.org/x/oauth2 with 1 pkgs; 1 versions to try
(28)      try golang.org/x/oauth2@master
(28)  ✓ select golang.org/x/oauth2@master w/2 pkgs
(29)  ? attempt github.com/Jeffail/gabs with 1 pkgs; 3 versions to try
(29)      try github.com/Jeffail/gabs@1.0
(29)  ✓ select github.com/Jeffail/gabs@1.0 w/1 pkgs
(30)  ? attempt github.com/posener/complete with 1 pkgs; 1 versions to try
(30)      try github.com/posener/complete@master
(30)  ✓ select github.com/posener/complete@master w/4 pkgs
(31)  ? attempt github.com/michaelklishin/rabbit-hole with 1 pkgs; 7 versions to try
(31)      try github.com/michaelklishin/rabbit-hole@v1.3.0
(31)  ✓ select github.com/michaelklishin/rabbit-hole@v1.3.0 w/1 pkgs
(32)  ? attempt github.com/cockroachdb/cockroach-go with 1 pkgs; 3 versions to try
(32)      try github.com/cockroachdb/cockroach-go@master
(32)  ✓ select github.com/cockroachdb/cockroach-go@master w/1 pkgs
(33)  ? attempt github.com/mitchellh/mapstructure with 1 pkgs; 1 versions to try
(33)      try github.com/mitchellh/mapstructure@master
(33)  ✓ select github.com/mitchellh/mapstructure@master w/1 pkgs
(34)  ? attempt github.com/hashicorp/vault-plugin-auth-gcp with 1 pkgs; 1 versions to try
(34)      try github.com/hashicorp/vault-plugin-auth-gcp@master
(35)  ✗   github.com/hashicorp/vault-plugin-auth-gcp@master depends on github.com/hashicorp/vault with master, but that's already selected at 
(34)    ← no more versions of github.com/hashicorp/vault-plugin-auth-gcp to try; begin backtrack
(33)  ← backtrack: no more versions of github.com/mitchellh/mapstructure to try
(32)  ← backtrack: no more versions of github.com/cockroachdb/cockroach-go to try
(31)  ← backtrack: no more versions of github.com/michaelklishin/rabbit-hole to try
(30)  ← backtrack: no more versions of github.com/posener/complete to try
(29)  ← backtrack: no more versions of github.com/Jeffail/gabs to try
(28)  ← backtrack: no more versions of golang.org/x/oauth2 to try
(27)  ← backtrack: no more versions of github.com/hashicorp/go-cleanhttp to try
(26)  ← backtrack: no more versions of github.com/google/go-querystring to try
(25)  ← backtrack: no more versions of github.com/chrismalek/oktasdk-go to try
(24)  ← backtrack: no more versions of github.com/hashicorp/go-syslog to try
(23)  ← backtrack: no more versions of gopkg.in/yaml.v2 to try
(22)  ← backtrack: no more versions of github.com/ghodss/yaml to try
(21)  ← backtrack: no more versions of github.com/mitchellh/reflectwalk to try
(20)  ← backtrack: no more versions of golang.org/x/sys to try
(19)  ← backtrack: no more versions of google.golang.org/api to try
(18)  ← backtrack: no more versions of gopkg.in/mgo.v2 to try
(17)  ← backtrack: no more versions of github.com/hashicorp/go-multierror to try
(16)  ← backtrack: no more versions of github.com/hashicorp/go-uuid to try
(15)  ← backtrack: no more versions of github.com/mitchellh/go-homedir to try
(14)  ← backtrack: no more versions of github.com/mitchellh/go-testing-interface to try
(13)  ← backtrack: no more versions of github.com/golang/protobuf to try
(12)  ← backtrack: no more versions of github.com/hashicorp/go-rootcerts to try
(11)  ← backtrack: no more versions of github.com/hashicorp/go-plugin to try
(10)  ← backtrack: no more versions of github.com/mitchellh/copystructure to try
(9) ← backtrack: no more versions of github.com/jefferai/jsonx to try
(8) ← backtrack: no more versions of github.com/go-errors/errors to try
(7) ← backtrack: no more versions of github.com/hashicorp/go-sockaddr to try
(6) ← backtrack: no more versions of github.com/duosecurity/duo_api_golang to try
(5) ← backtrack: no more versions of github.com/hashicorp/errwrap to try
(4) ← backtrack: no more versions of github.com/fullsailor/pkcs7 to try
(3) ← backtrack: no more versions of github.com/armon/go-radix to try
(2) ← backtrack: no more versions of github.com/armon/go-proxyproto to try
(1) ← backtrack: no more versions of github.com/armon/go-metrics to try
  ✗ solving failed

Solver wall times by segment:
     b-list-versions: 15.993287545s
         b-list-pkgs:  1.449205562s
              b-gmal:  975.663417ms
  b-deduce-proj-root:    52.45496ms
         select-root:   14.219985ms
         select-atom:    4.123771ms
             satisfy:    3.600891ms
            unselect:     3.38189ms
            new-atom:     940.808µs
           backtrack:     500.397µs
     b-source-exists:      43.223µs
               other:      25.323µs
          b-pair-rev:      11.874µs
           b-matches:        5.97µs
      b-pair-version:       2.596µs

  TOTAL: 18.497468212s

No versions of github.com/hashicorp/vault-plugin-auth-gcp met constraints:
    master: Could not introduce github.com/hashicorp/vault-plugin-auth-gcp@master, as it has a dependency on github.com/hashicorp/vault with constraint master, which does not allow the currently selected version of

Note: I did not cut off the output -- it ends with selected version of <blank>.

This may be related to #1125.

Right now I have one package using dep and another using govendor. The project using dep is https://github.com/hashicorp/vault-plugin-auth-gcp which currently is depending on master of https://github.com/hashicorp/vault

(Side question: since its inception Vault has put a v in front of tags; does dep deal with stripping that off before semvering what's there?)

Right now Vault uses govendor and has these stanzas -- note how the packages are separate:

        {
            "checksumSHA1": "0+GdkCbe9GplobPLWizCNqhntbU=",
            "path": "github.com/hashicorp/vault-plugin-auth-gcp",
            "revision": "6d78c24a995a2df3188ca891db068fbecbc69262",
            "revisionTime": "2017-09-05T14:51:27Z"
        },
        {
            "checksumSHA1": "UsbYUscv6YGOBS7qNcCr3mVwuX8=",
            "path": "github.com/hashicorp/vault-plugin-auth-gcp/plugin",
            "revision": "6d78c24a995a2df3188ca891db068fbecbc69262",
            "revisionTime": "2017-09-05T14:51:27Z"
        },
        {
            "checksumSHA1": "2Fk5xIPOoKD4Of9cUIQPKXb2htg=",
            "path": "github.com/hashicorp/vault-plugin-auth-gcp/plugin/util",
            "revision": "6d78c24a995a2df3188ca891db068fbecbc69262",
            "revisionTime": "2017-09-05T14:51:27Z"
        },

I wanted to try switching Vault over to dep. But since the first package is a plugin of Vault, and since Vault builds the plugin internally into itself when possible, they both have a dependency on each other. In reality only certain packages in the Vault repo are needed, but splitting them out of the Vault repo isn't really feasible (again, the plugin is also built-in internally when possible).

No matter what revision/tag/whatever I try specifying in the plugin's Gopkg file, the result is the same other than in the output saying "...a dependency on github.com/hashicorp/vault with constraint ^0.8.2...". The last part of it "...currently selected version of" always ends blank.

Possibly this can be addressed using an override? But didn't quite see what I'd specify. Maybe I need to hand-create a Gopkg.toml? But if I do that, will future runs of dep ensure still have a problem, and/or will they add any direct dependencies I missed, and what should be inside that file?

[sdboyer]

hi, welcome! this is an interesting one 😄

easy thing first:

(Side question: since its inception Vault has put a v in front of tags; does dep deal with stripping that off before semvering what's there?)

yep. this actually the source of some consternation for me, because we consider tags with and without the leading v to be equivalent. i can't actually recall what happens when both exist, but i believe one clobbers the other. more tests & checks needed. anyway:

they both have a dependency on each other. In reality only certain packages in the Vault repo are needed,

we do support project-level cycles, at least in part. (technically we don't disallow package-level cycles, either, but only because it's yet to actually be a problem for anyone, and i've yet to find an efficient incremental graph cycle detection algorithm). we added support for it right around when we opened dep up - #116. but, back then, nobody was using dep, so there was no possibility of there being constraints declared back onto the root project. i didn't think about it in the initial implementation, so i just didn't cover that case - as evidenced by some of those failure messages simply stopping mid-sentence. that happens because we treat the root project specially in several ways within the solver, the relevant one here being that the root project is treated as versionless.

i don't think the mechanics of a fix here are going to be very hard, but it's also not immediately clear to me what we should do. the only solution that's readily apparent to me is that, when a project-level cycle passes back through the root project, we simply drop any explicit constraints there.

the only immediate drawback i can think of - and this is just OTTOMH, it might not be a thing - is that it might be easy to create situations that are unresolvable, or at least where certain versions become unreachable, for either vault or vault-plugin-auth-gcp's dependers because constraints declared in one or the other project are only satisfiable when the other project's constraints are knocked out by my proposed rule.

Possibly this can be addressed using an override? But didn't quite see what I'd specify.

maybe, although i think the only thing that really work in the override is an "any", which i believe we don't actually allow the user to explicitly specify.

[jefferai]

Thanks for the detailed write up!

The only thing I can think of offhand, and I realize this is likely yucky but I have 0.01% of your experience in this area, is to be able to declare an explicit binding between repos such that the constraint is that the revisions match. E.g. the branch must be master for both, or they must be using the same tag, or something. Then you can assume that they are in sync and add them as is rather than solving.

That probably doesn't make much sense, and it also doesn't solve the init problem, but thought I'd throw it out there.

[sdboyer]

an explicit binding between repos such that the constraint is that the revisions match. E.g. the branch must be master for both, or they must be using the same tag, or something. Then you can assume that they are in sync and add them as is rather than solving.

iiiiiinteresting!

it likely would be yucky, but i haven't seriously explored a line of thought like this before. it has a pretty clear antecedent in graph theory - vertex contraction - which at least means that i can consider its effects on the system in a more formal way. my gut says that we'd need to find more reasons for doing something like this than just this one, but because it rests on this mathematical foundation, we could likely find some. (though still, maybe too yucky)

doesn't solve the init problem

yeah, this is a trickier thing. we're having more and more init smoothness problems recently, but this is one that goes deep. i think we may not have much choice but to do the "ignore root constraints" thing i laid out in order to make init - and subsequent ensures - workable at all.

thought I'd throw it out there.

🎉 🎉 please, don't hesitate to do this!

[jefferai]

I believe other HashiCorp products might also have this issue. Terraform for instance split its providers out into separate packages to make it easier for contributors to work on just a single chunk of code, but IIRC it means they need to import helper libraries and structs/interfaces from the main repo and it has to import many of these providers to build into the binary.

I realize this is likely to only affect projects using a plugin based model. OTOH Go now added first party support for plugins which probably makes this scenario more likely going forward.

[sdboyer]

I believe other HashiCorp products might also have this issue

honestly, i'd like it to be more common for other orgs to at least feel comfortable with doing things like this. it's also something that might benefit k8s - we're in the midst of trying to figure out how to support their staging/ dir, which isn't a plugin-style use case, but is still an issue trying to have projects where the development is closely coupled with a given repository "move with" that repository, while still having a canonical identity external to it.

(in fact, /cc @sttts @kris-nova - this may end up relating to k8s things)

OTOH Go now added first party support for plugins which probably makes this scenario more likely going forward.

i opened #347 a while back, but we backburnered it b/c there are just so many problems with the language's plugin system that the whole area is rather anemic right now. i actually wonder if causality may end up flowing in the other direction - with improved package management, plugins become more feasible to actually use in production, which in turn structures and drives improvements in the language support for them. ¯\_(ツ)_/¯

[asticode]

Don't know where the issue is at, but a colleague of mine wrote a quick fix that I've added to a fork of mine, thought I'd throw it in there in case that may help finding a more durable solution:

https://github.com/asticode/dep/commit/942f050b345a2ae531651f42d28266a5889a5301

[sdboyer]

@asticode yeah, that's the naive fix (well done finding that deep in the solver, lol 😜). if we denote the root project with an (r), it solves this problem:

A(r) <-> B

The thing i'm trying to find time to grapple with in finding that more durable solution is what that means when A is published, and this situation then occurs:

C(r) -> A <-> B

i don't think there's really much to be done here, per se, but i want to get my head around the possible effects before introducing this kind of change into the ecosystem. this kind of variability, while convenient in the small, could potentially produce very confusing situations in the large.

[manuel-huez]

Jumping in, I've also been facing a similar issue with #1842. Perhaps a starting point would be to add an experimental flag when running dep to skip circular dependencies? It's currently breaking our CI as dep fails to resolve dependencies. The current dirty fix is to use -vendor-only when running dep ensure in CI and setting the versions manually, but it seems counter-intuitive.

[sdboyer]

@manuel-huez it'll likely require too much reworking of things to make this work cleanly in dep as-is, so i'm going to defer this to dep's successor tool. More info on that soon™ 😄

[kegsay]

I'd like to a see an experimental flag which implements https://github.com/asticode/dep/commit/942f050b345a2ae531651f42d28266a5889a5301 so at least dep can handle more cases than it can today, rather than waiting for months/years for another tool to do it. Gating it behind appropriately scary-sounding flag names seems reasonable.

I've recently hit this issue, and it's frustrating knowing that there is a workaround available.