"dep ensure -update foo/bar/baz && dep prune" takes ages to complete

dcelasun commented 7 years ago

Full description below.

What version of `dep` are you using (`dep version`)?

v0.3.0-142-g6ca8a48

What `dep` command did you run?

dep ensure -update foo/bar/baz && dep prune

What did you expect to see?

I expected a single isolated (i.e, not depending on or imported by anything else) dependency to be updated in a few seconds.

What did you see instead?

The whole thing took ~10 minutes to complete.

I wanted to update a single dependency with dep ensure -update. Since dep -update (unintuitively, imho) restores every single already-pruned stuff during -update I also had to run dep prune again.

Still, updating a single isolated dependency should not:

take more than a few seconds
affect all the other, completely unrelated dependencies
undo unrelated previously pruned packages

Not sure if it's helpful, but Gopkg.lock is below.

Is there anything I should be doing differently? Is it a bug or expected behaviour?

# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.

[[projects]]
  branch = "master"
  name = "private/stuff/p1"
  packages = ["."]
  revision = "e17bb5e1c62f888e3d69071ec17b5754dc2fa6d7"

[[projects]]
  branch = "master"
  name = "private/stuff/p2"
  packages = ["."]
  revision = "0a7924b7b3c7490bf15a94cb9ed644020fa0e0b1"

[[projects]]
  branch = "master"
  name = "private/stuff/p3"
  packages = ["."]
  revision = "26597a89f60c58361c003ad290243f1ca85f605c"

[[projects]]
  branch = "master"
  name = "private/stuff/p4"
  packages = ["."]
  revision = "62eeb0956c3dda89c32dbd54fc1bf5aa3cb08bba"

[[projects]]
  branch = "master"
  name = "private/stuff/p5"
  packages = ["."]
  revision = "a16255a8e6356ad3bb4bfe5816d8ec47239fca2d"

[[projects]]
  branch = "master"
  name = "private/stuff/p6"
  packages = ["service/hooks"]
  revision = "e8c5dd8615a69e2f32d4da3afa07588d73895287"

[[projects]]
  branch = "master"
  name = "github.com/Gurpartap/logrus-stack"
  packages = ["."]
  revision = "89c00d8a28f43c567d92eb81a2945301a6a9fbb9"

[[projects]]
  branch = "v1"
  name = "github.com/Masterminds/squirrel"
  packages = ["."]
  revision = "20f192218cf52a73397fa2df45bdda720f3e47c8"

[[projects]]
  branch = "master"
  name = "github.com/armon/consul-api"
  packages = ["."]
  revision = "dcfedd50ed5334f96adee43fc88518a4f095e15c"

[[projects]]
  branch = "master"
  name = "github.com/armon/go-proxyproto"
  packages = ["."]
  revision = "48572f11356f1843b694f21a290d4f1006bc5e47"

[[projects]]
  name = "github.com/blang/semver"
  packages = ["."]
  revision = "2ee87856327ba09384cabd113bc6b5d174e9ec0f"
  version = "v3.5.1"

[[projects]]
  name = "github.com/codegangsta/negroni"
  packages = ["."]
  revision = "fde5e16d32adc7ad637e9cd9ad21d4ebc6192535"
  version = "v0.2.0"

[[projects]]
  name = "github.com/coreos/etcd"
  packages = ["client","pkg/pathutil","pkg/srv","pkg/types","version"]
  revision = "c31bec0f29facff13f7c3e3d948e55dd6689ed42"
  version = "v3.2.4"

[[projects]]
  name = "github.com/coreos/go-semver"
  packages = ["semver"]
  revision = "8ab6407b697782a06568d4b7f1db25550ec2e4c6"
  version = "v0.2.0"

[[projects]]
  branch = "master"
  name = "github.com/facebookgo/stack"
  packages = ["."]
  revision = "751773369052141c013c6e827a71e8f35c07879c"

[[projects]]
  name = "github.com/fsnotify/fsnotify"
  packages = ["."]
  revision = "629574ca2a5df945712d3079857300b5e4da0236"
  version = "v1.4.2"

[[projects]]
  branch = "master"
  name = "github.com/getsocial-rnd/ip2location-go"
  packages = ["."]
  revision = "4df59af9693f152785acc5edb1ea257b156344b1"

[[projects]]
  name = "github.com/go-sql-driver/mysql"
  packages = ["."]
  revision = "a0583e0143b1624142adab07e0e97fe106d99561"
  version = "v1.3"

[[projects]]
  name = "github.com/gorilla/context"
  packages = ["."]
  revision = "1ea25387ff6f684839d82767c1733ff4d4d15d0a"
  version = "v1.1"

[[projects]]
  branch = "master"
  name = "github.com/gorilla/schema"
  packages = ["."]
  revision = "8b1100835db5bbdae88541510f70d114b91a7e4d"

[[projects]]
  branch = "master"
  name = "github.com/hashicorp/hcl"
  packages = [".","hcl/ast","hcl/parser","hcl/scanner","hcl/strconv","hcl/token","json/parser","json/scanner","json/token"]
  revision = "392dba7d905ed5d04a5794ba89f558b27e2ba1ca"

[[projects]]
  branch = "master"
  name = "github.com/jmoiron/sqlx"
  packages = [".","reflectx"]
  revision = "d9bd385d68c068f1fabb5057e3dedcbcbb039d0f"

[[projects]]
  branch = "master"
  name = "github.com/lann/builder"
  packages = ["."]
  revision = "f22ce00fd9394014049dad11c244859432bd6820"

[[projects]]
  branch = "master"
  name = "github.com/lann/ps"
  packages = ["."]
  revision = "62de8c46ede02a7675c4c79c84883eb164cb71e3"

[[projects]]
  name = "github.com/magiconair/properties"
  packages = ["."]
  revision = "be5ece7dd465ab0765a9682137865547526d1dfb"
  version = "v1.7.3"

[[projects]]
  name = "github.com/mattes/migrate"
  packages = [".","database","database/mysql","source","source/file"]
  revision = "035c07716cd373d88456ec4d701402df52584cb4"
  version = "v3.0.1"

[[projects]]
  branch = "master"
  name = "github.com/mediocregopher/radix.v2"
  packages = ["cluster","pool","pubsub","redis","util"]
  revision = "47da4d0e14739b95177d6e1e8521bb4f2ac84f59"

[[projects]]
  branch = "master"
  name = "github.com/mitchellh/mapstructure"
  packages = ["."]
  revision = "d0303fe809921458f417bcf828397a65db30a7e4"

[[projects]]
  name = "github.com/oschwald/maxminddb-golang"
  packages = ["."]
  revision = "d19f6d453e836d12ee8fe895d0494421e93ef8c1"
  version = "v1.2.0"

[[projects]]
  name = "github.com/pelletier/go-buffruneio"
  packages = ["."]
  revision = "c37440a7cf42ac63b919c752ca73a85067e05992"
  version = "v0.2.0"

[[projects]]
  name = "github.com/pelletier/go-toml"
  packages = ["."]
  revision = "5ccdfb18c776b740aecaf085c4d9a2779199c279"
  version = "v1.0.0"

[[projects]]
  name = "github.com/satori/go.uuid"
  packages = ["."]
  revision = "879c5887cd475cd7864858769793b2ceb0d44feb"
  version = "v1.1.0"

[[projects]]
  name = "github.com/sirupsen/logrus"
  packages = ["."]
  revision = "a3f95b5c423586578a4e099b11a46c2479628cac"
  version = "1.0.2"

[[projects]]
  branch = "master"
  name = "github.com/skip2/go-qrcode"
  packages = [".","bitset","reedsolomon"]
  revision = "0bb60b5a716144fa55da373bab578cca49a537d7"

[[projects]]
  branch = "master"
  name = "github.com/spf13/afero"
  packages = [".","mem"]
  revision = "9be650865eab0c12963d8753212f4f9c66cdcf12"

[[projects]]
  name = "github.com/spf13/cast"
  packages = ["."]
  revision = "acbeb36b902d72a7a4c18e8f3241075e7ab763e4"
  version = "v1.1.0"

[[projects]]
  branch = "master"
  name = "github.com/spf13/jwalterweatherman"
  packages = ["."]
  revision = "0efa5202c04663c757d84f90f5219c1250baf94f"

[[projects]]
  name = "github.com/spf13/pflag"
  packages = ["."]
  revision = "e57e3eeb33f795204c1ca35f56c44f83227c6e66"
  version = "v1.0.0"

[[projects]]
  branch = "master"
  name = "github.com/spf13/viper"
  packages = [".","remote"]
  revision = "25b30aa063fc18e48662b86996252eabdcf2f0c7"

[[projects]]
  branch = "master"
  name = "github.com/ua-parser/uap-go"
  packages = ["uaparser"]
  revision = "e0b1fe757c944e839263af90b3c74b76d816558e"

[[projects]]
  branch = "master"
  name = "github.com/ugorji/go"
  packages = ["codec"]
  revision = "5efa3251c7f7d05e5d9704a69a984ec9f1386a40"

[[projects]]
  branch = "master"
  name = "github.com/xordataexchange/crypt"
  packages = ["backend","backend/consul","backend/etcd","config","encoding/secconf"]
  revision = "b2862e3d0a775f18c7cfe02273500ae307b61218"

[[projects]]
  name = "goji.io"
  packages = [".","internal","pat","pattern"]
  revision = "0d89ff54b2c18c9c4ba530e32496aef902d3c6cd"
  version = "v2.0"

[[projects]]
  branch = "master"
  name = "golang.org/x/crypto"
  packages = ["cast5","openpgp","openpgp/armor","openpgp/elgamal","openpgp/errors","openpgp/packet","openpgp/s2k"]
  revision = "558b6879de74bc843225cde5686419267ff707ca"

[[projects]]
  branch = "master"
  name = "golang.org/x/net"
  packages = ["context","idna","lex/httplex"]
  revision = "f5079bd7f6f74e23c4d65efa0f4ce14cbd6a3c0f"

[[projects]]
  branch = "master"
  name = "golang.org/x/sys"
  packages = ["unix","windows"]
  revision = "396c9fc8fb0ce27fc7a1ff136dc5af797bba258a"

[[projects]]
  branch = "master"
  name = "golang.org/x/text"
  packages = ["internal/gen","internal/tag","internal/triegen","internal/ucd","language","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
  revision = "3bd178b88a8180be2df394a1fbb81313916f0e7b"

[[projects]]
  name = "gopkg.in/tylerb/graceful.v1"
  packages = ["."]
  revision = "4654dfbb6ad53cb5e27f37d99b02e16c1872fbbb"
  version = "v1.2.15"

[[projects]]
  branch = "v2"
  name = "gopkg.in/yaml.v2"
  packages = ["."]
  revision = "25c4ec802a7d637f88d584ab26798e94ad14c13b"

[solve-meta]
  analyzer-name = "dep"
  analyzer-version = 1
  inputs-digest = "2982228be97ef72b60341b541134eefdbcb57cb92f88dd8f359f5c21908c1376"
  solver-name = "gps-cdcl"
  solver-version = 1

darkowlzz commented 7 years ago

@dcelasun hi, thanks for creating an issue for this. Sadly, this is the expected behavior right now. We write the whole vendor every time, and that brings back the pruned files. There's work going on to perform prune automatically #1020 .

Due to vendor rewrite every time, dep needs all the dependencies in its cache. And if they don't already exist in cache, dep would clone all of them, and I think that's the reason for it taking so long to update in this case.

Vendor rewrite is done to ensure that vendor/ has all the dependencies in their expected state. There's work going on to implement vendor verification ( #121 ), which would eventually help us avoid full vendor rewrites.

Hope this helps.

dcelasun commented 7 years ago

@darkowlzz Thanks, I'll just have to wait for vendor verification and hope for the best then :)

sdboyer commented 7 years ago

good morning!

so, wrt slowness, we have an FAQ entry with plenty of specific information - and, fortunately, cases like these will eventually be able to be completed in a matter of seconds, if not <1s, because this is mostly struggling with the segments of dep that are extremely cacheable. but i'll explain things further in the context of this particular case, because i understand how this could seem like an easier problem than it is.

I expected a single isolated (i.e, not depending on or imported by anything else) dependency to be updated in a few seconds. Still, updating a single isolated dependency should not:

take more than a few seconds

affect all the other, completely unrelated dependencies

so, the problem here is these expectations :)

you might happen to know that these things are true beforehand for the particular dependency you're upgrading, but dep doesn't know it. dep has to prove it. isolation of any change within a depgraph can never be assumed safe\ a priori, because any update could add any new import, which could have its own new imports, which could end up overlapping with existing imports...

so, yes, we absolutely must recheck the entire graph in order to be safe - and dep's first goal is safety/reliability/predictability. this is why the command is dep ensure -update, and not dep update - because the former is a slightly stronger indicator that you're not "just updating one dependency." rather, dep is ensuring everything is still in sync, while also updating that particular dependency.

you'll note, however, in the dep ensure -v output, that if it turns out to be true that the dependency you're updating is isolated, that it takes only one attempt at every other dependency to recheck their validity in the new solution. algorithmically, that's the best case scenario. even then, though, that process can be slow, because we're still having to run git checkouts under the hood for each version of each dependency, then statically analyzing the code in those dependencies. however, we're working on a persistent caching layer for the analyzed results - #431 - that will obviate the need for all that analysis. work on that is well underway, though my ballpark is that it's still 3-4 months away, because...y'know, caching is hard, and it'd cause a lot of user pain to get that wrong.

undo unrelated previously pruned packages

so again, this is a mental model thing - in dep's world, vendor is dead code. and at the moment, it's basically bilious curds vomited out at the end of a previous run, because we can't yet verify it - #121 - though we're actually getting pretty close on that. a month, maybe two, i think.

because we can't yet verify anything in vendor, it means we have to regenerate vendor fully, on every single dep ensure run. this is one of the major reasons why dep is slow - that entire vendor dir is getting regenerated. this is also why prune is "undone" - because pruning, right now, is an afterthought of the model - again, vendor is dead code. however, we're working on absorbing prune back into ensure - #944 - which will make it a first-class part of the process (as well as providing more granular controls). that means pruning will never be "undone," as it'll just be the way that ensure works.

sdboyer commented 7 years ago

ah, i had this queued up and didn't see @darkowlzz' response :) oh well, double-covered!

golang / dep