Closed tve closed 8 years ago
I can add a little context and our direction which might help
glide up
now generates a glide.lock
file. If that file exists glide install
will install from there rather than detect dependencies. It downloads and sets the versions, when glide install
is run using concurrency so it's much faster. We recommend checking glide.lock
into your VCS alongside the glide.yaml
file.glide update
to use concurrency which will speed things up. Hope to have that in the next release.glide update
when some packages are vendored (e.g., docker)? Should the --update-vendored
flag be used? That will likely not speed things up though.Given this do you have any suggestions or thoughts? I do want to make update faster in the next release. We're hoping that most people can use install with a lockfile to get some good speed increases.
I'd be curious to see what the glide.lock
file looks like if it's generating the output above. Something doesn't look quite right. Working with Kubernetes, with over 200 dependencies, I still see glide install
times of between 1 and 2 minutes on Travis.
FWIW, some of the "Scanning..." lines that appear to be duplicates are actually not re-scanning. There's a cache check that occurs after that line, and if the scan is in the cache, it's skipped.
For comparison, this is what the output of glide install
ought to look like on a Travis-like system. This is for https://github.com/pipeviz/pipeviz:
⇒ time glide install
[INFO] Fetching updates for github.com/armon/consul-api.
[INFO] Fetching updates for github.com/codegangsta/inject.
[INFO] Fetching updates for github.com/bugsnag/panicwrap.
[INFO] Fetching updates for github.com/bradfitz/gomemcache.
[INFO] Fetching updates for github.com/coreos/go-etcd.
[INFO] Fetching updates for github.com/cpuguy83/go-md2man.
[INFO] Fetching updates for github.com/garyburd/redigo.
[INFO] Fetching updates for github.com/getsentry/raven-go.
[INFO] Fetching updates for github.com/go-martini/martini.
[INFO] Fetching updates for github.com/goji/param.
[INFO] Fetching updates for github.com/golang/protobuf.
[INFO] Fetching updates for github.com/gorilla/websocket.
[INFO] Fetching updates for github.com/inconshreveable/mousetrap.
[INFO] Fetching updates for github.com/jarcoal/httpmock.
[INFO] Fetching updates for github.com/kardianos/osext.
[INFO] Fetching updates for github.com/agtorre/gocolorize.
[INFO] Fetching updates for github.com/boltdb/bolt.
[INFO] Fetching updates for github.com/bugsnag/bugsnag-go.
[INFO] Fetching updates for github.com/bugsnag/osext.
[INFO] Fetching updates for github.com/BurntSushi/toml.
[INFO] Fetching updates for github.com/kr/pretty.
[INFO] Fetching updates for github.com/kr/pty.
[INFO] Fetching updates for github.com/kr/text.
[INFO] Fetching updates for github.com/magiconair/properties.
[INFO] Fetching updates for github.com/mitchellh/mapstructure.
[INFO] Fetching updates for github.com/mndrix/ps.
[INFO] Fetching updates for github.com/philhofer/fwd.
[INFO] Fetching updates for github.com/revel/revel.
[INFO] Fetching updates for github.com/robfig/config.
[INFO] Fetching updates for github.com/robfig/go-cache.
[INFO] Fetching updates for github.com/robfig/pathtree.
[INFO] Fetching updates for github.com/russross/blackfriday.
[INFO] Fetching updates for github.com/shurcooL/sanitized_anchor_name.
[INFO] Fetching updates for github.com/Sirupsen/logrus.
[INFO] Fetching updates for github.com/spf13/cast.
[INFO] Fetching updates for github.com/spf13/cobra.
[INFO] Fetching updates for github.com/spf13/jwalterweatherman.
[INFO] Fetching updates for github.com/spf13/pflag.
[INFO] Fetching updates for github.com/spf13/viper.
[INFO] Fetching updates for github.com/stretchr/objx.
[INFO] Fetching updates for github.com/stretchr/testify.
[INFO] Fetching updates for github.com/tinylib/msgp.
[INFO] Fetching updates for github.com/tobi/airbrake-go.
[INFO] Fetching updates for github.com/ttacon/chalk.
[INFO] Fetching updates for github.com/ugorji/go.
[INFO] Fetching updates for github.com/unrolled/secure.
[INFO] Fetching updates for github.com/xeipuuv/gojsonpointer.
[INFO] Fetching updates for github.com/xeipuuv/gojsonreference.
[INFO] Fetching updates for github.com/xeipuuv/gojsonschema.
[INFO] Fetching updates for github.com/xordataexchange/crypt.
[INFO] Fetching updates for github.com/zenazn/goji.
[INFO] Fetching updates for golang.org/x/crypto.
[INFO] Fetching updates for golang.org/x/net.
[INFO] Fetching updates for golang.org/x/text.
[INFO] Fetching updates for golang.org/x/tools.
[INFO] Fetching updates for google.golang.org/appengine.
[INFO] Fetching updates for gopkg.in/fsnotify.v1.
[INFO] Fetching updates for gopkg.in/yaml.v2.
[INFO] Setting version for github.com/agtorre/gocolorize to f42b554bf7f006936130c9bb4f971afd2d87f671.
[INFO] Setting version for github.com/coreos/go-etcd to 003851be7bb0694fe3cc457a49529a19388ee7cf.
[INFO] Setting version for github.com/gorilla/websocket to ecff5aabe41f13b4cdf897e3c0c9bbccbe552a29.
[INFO] Setting version for github.com/codegangsta/inject to 33e0aa1cb7c019ccc3fbe049a8262a6403d30504.
[INFO] Setting version for github.com/cpuguy83/go-md2man to 71acacd42f85e5e82f70a55327789582a5200a90.
[INFO] Setting version for github.com/goji/param to da86c81e3e3c23b1948bc7a003d381250a032aa7.
[INFO] Setting version for github.com/jarcoal/httpmock to 145b10d659265440f062c31ea15326166bae56ee.
[INFO] Setting version for github.com/golang/protobuf to 68415e7123da32b07eab49c96d2c4d6158360e9b.
[INFO] Setting version for github.com/garyburd/redigo to 6ece6e0a09f28cc399b21550cbf37ab39ba63cce.
[INFO] Setting version for github.com/inconshreveable/mousetrap to 76626ae9c91c4f2a10f34cad8ce83ea42c93bb75.
[INFO] Setting version for github.com/kardianos/osext to 8fef92e41e22a70e700a96b29f066cda30ea24ef.
[INFO] Setting version for github.com/kr/pretty to e6ac2fc51e89a3249e82157fa0bb7a18ef9dd5bb.
[INFO] Setting version for github.com/bradfitz/gomemcache to 72a68649ba712ee7c4b5b4a943a626bcd7d90eb8.
[INFO] Setting version for github.com/bugsnag/panicwrap to e5f9854865b9778a45169fc249e99e338d4d6f27.
[INFO] Setting version for github.com/getsentry/raven-go to 3966f3ab8333308d76b6cc83a29776a266bbdd92.
[INFO] Setting version for github.com/bugsnag/bugsnag-go to 02e952891c52fbcb15f113d90633897355783b6e.
[INFO] Setting version for github.com/BurntSushi/toml to 056c9bc7be7190eaa7715723883caffa5f8fa3e4.
[INFO] Setting version for github.com/kr/pty to f7ee69f31298ecbe5d2b349c711e2547a617d398.
[INFO] Setting version for github.com/philhofer/fwd to 4dbda5e2904d044d23cc4bef84e8360c458caf7f.
[INFO] Setting version for github.com/magiconair/properties to 23b847ebbbdaada7234bf4b3da1e50e601fbf63b.
[INFO] Setting version for github.com/armon/consul-api to dcfedd50ed5334f96adee43fc88518a4f095e15c.
[INFO] Setting version for github.com/mndrix/ps to 33ddf69629c1bcea76b04f4a4e9e00e4cce82eb6.
[INFO] Setting version for github.com/kr/text to bb797dc4fb8320488f47bf11de07a733d7233e1f.
[INFO] Setting version for github.com/boltdb/bolt to c2745b3c62985affcf08d0522135f4747e9b81f3.
[INFO] Setting version for github.com/revel/revel to a9a2ff45fae4330ef4116b257bcf9c82e53350c2.
[INFO] Setting version for github.com/robfig/go-cache to 9fc39e0dbf62c034ec4e45e6120fc69433a3ec51.
[INFO] Setting version for github.com/robfig/pathtree to 41257a1839e945fce74afd070e02bab2ea2c776a.
[INFO] Setting version for github.com/mitchellh/mapstructure to 281073eb9eb092240d33ef253c404f1cca550309.
[INFO] Setting version for github.com/bugsnag/osext to 0dd3f918b21bec95ace9dc86c7e70266cfc5c702.
[INFO] Setting version for github.com/robfig/config to 0f78529c8c7e3e9a25f15876532ecbc07c7d99e6.
[INFO] Setting version for github.com/shurcooL/sanitized_anchor_name to 10ef21a441db47d8b13ebcc5fd2310f636973c77.
[INFO] Setting version for github.com/spf13/cast to ee815aaf958c707ad07547cd62150d973710f747.
[INFO] Setting version for github.com/go-martini/martini to 15a47622d6a9b3e6a1eaca2681e4850f612471ea.
[INFO] Setting version for github.com/stretchr/objx to 1a9d0bb9f541897e62256577b352fdbc1fb4fd94.
[INFO] Setting version for github.com/russross/blackfriday to 510be64de022c25096c6f476b7b7c240a8eca1a5.
[INFO] Setting version for github.com/spf13/cobra to 2e6a42892123dda608922f8af8ce85c3bff19575.
[INFO] Setting version for github.com/Sirupsen/logrus to 9c060de643590dae45da9d7c26276463bfc46fa0.
[INFO] Setting version for github.com/spf13/jwalterweatherman to c2aa07df593850a04644d77bb757d002e517a296.
[INFO] Setting version for github.com/spf13/viper to e37b56e207dda4d79b9defe0548e960658ee8b6b.
[INFO] Setting version for github.com/spf13/pflag to 08b1a584251b5b62f458943640fc8ebd4d50aaa5.
[INFO] Setting version for github.com/stretchr/testify to e4ec8152c15fc46bd5056ce65997a07c7d415325.
[INFO] Setting version for github.com/xeipuuv/gojsonpointer to 636edb2500d21f2ed09ea96a00deb36bbd07cf70.
[INFO] Setting version for github.com/ttacon/chalk to e66ca4d608eb6dc36d012bceeb08ef9f379134c1.
[INFO] Setting version for github.com/unrolled/secure to 9b725bb5ee80564334176a116a99dd926cab63a1.
[INFO] Setting version for github.com/tobi/airbrake-go to a3cdd910a3ffef88a20fbecc10363a520ad61a0a.
[INFO] Setting version for github.com/xordataexchange/crypt to 749e360c8f236773f28fc6d3ddfce4a470795227.
[INFO] Setting version for github.com/ugorji/go to 357a44b2b13e2711a45e30016508134101477610.
[INFO] Setting version for github.com/xeipuuv/gojsonreference to 2df3c0c802434c5cb984dbc21425f5960bda4d16.
[INFO] Setting version for github.com/zenazn/goji to 845982030542a0fd9c8e8cbf76a84c7482cb3755.
[INFO] Setting version for github.com/xeipuuv/gojsonschema to 71b85f61a135e79143f3d3238d5175a4f29b6689.
[INFO] Setting version for golang.org/x/crypto to 7b85b097bf7527677d54d3220065e966a0e3b613.
[INFO] Setting version for golang.org/x/net to fa33dc7b893a6db61a44637ced689a85ee5208c4.
[INFO] Setting version for golang.org/x/text to dd164d33a23c6e075adc2b75ac1ae939b81324aa.
[INFO] Setting version for google.golang.org/appengine to 7f9f17f358c21d90f402980f83776898c53cb377.
[INFO] Setting version for github.com/tinylib/msgp to cd4fb1548c31d88af25205dc021be20935aec720.
[INFO] Setting version for gopkg.in/yaml.v2 to f7716cbe52baa25d2e9b0d0da546fcf909fc16b4.
[INFO] Setting version for gopkg.in/fsnotify.v1 to 2cdd39bd6129c6a49c74fb07fb9d77ba1271c572.
[INFO] Setting version for golang.org/x/tools to c0008c5889c0d5091cdfefd2bfb08bff96527879.
glide install 9.43s user 9.88s system 56% cpu 34.280 total
⇒ wc -l glide.lock
162 glide.lock
Has there been any update on this? I am getting a hard time updating. I only have 9 vendored in my glide.yaml file and it is taking too long when doing an glide update/install
I keep thinking that it may be stuck on "scanning"
@thinkong what version of Glide do you have? You can run glide --version
to get it. The point version matters which is why I ask. Also, what's your output when using the --debug
flag (for example glide --debug up
)?
In the next version of Glide, currently being worked on in branches, there will be some speed improvements.
@mattfarina
λ glide --version glide version dev
I am on windows atm
here is the debug stuff
λ glide --debug up
[INFO] Fetching updates for myproj.
myproj@192.168.0.10's password:
myproj@192.168.0.10's password:
[INFO] Fetching updates for github.com/tealeg/xlsx.
[INFO] Fetching updates for github.com/satori/go.uuid.
[INFO] Fetching updates for github.com/mattn/anko.
[INFO] Fetching updates for github.com/daviddengcn/go-colortext.
[INFO] Fetching updates for github.com/elgs/gostrgen.
[INFO] Fetching updates for github.com/mitchellh/mapstructure.
[INFO] Fetching updates for github.com/dancannon/gorethink.
[INFO] Fetching updates for github.com/gin-gonic/gin.
[INFO] Fetching updates for github.com/julienschmidt/httprouter.
[DEBUG] ---> Inspecting d:\myproj\src\main\vendor for changes (10 packages).
[DEBUG] ----> Scanning myproj
[DEBUG] Looking in d:\myproj\src\main\vendor/myproj/Godeps/ for a Godeps.json file.
[DEBUG] Looking in d:\myproj\src\main\vendor/myproj/vendor/ for a manifest file.
[INFO] Scanning myproj for dependencies.
[DEBUG] => Scanning encoding/json
@thinkong what commit is Glide on? There was a recent bug that in some situations could cause this that's already been fixed. Can you test the tip of master?
I just used go get github.com/Mastermind/glide
I also tested with the binaries given here but got the same result. Ill have to try this again after the holidays. Merry christmas
well sorry for the late reply.
I don't exactly get what you mean by "tip of master" but i just did a git pull
and then another go install github.com/Masterminds/glide
but same thing happens..
I use windows 7 64bit and go 1.5.1
I just created a seperate issue.. due to my issue being a windows only issue
I ran some numbers for comparison between master and 0.8.3
Project | 0.8.3 | master |
---|---|---|
go-boilerplate | 10.607s | 3.092s |
rsc | 24.780s | 7.025s |
Since rsc is managed by Godep I used glide init
to generate a glide.yaml
file.
We've sped up Glide a fair amount so I'm going to close this. There are a few other improvements we can still make but this is quite a speed up.
If there are still performance issues please let us know.
A recent update of glide has made glide install very slow. We use glide install in Travis CI and a project that used to take ~2 minutes to build and run tests now takes >3 minutes just for the glide install. Somehow it's scanning and rescanning dependencies. The worst piece is that we use github.com.samalba/dockerclient, which uses a couple of packages in github.com/docker/docker/pkg/* and that unleashes glide on the entire docker dependency tree. Thoughts?
Below is the glide install log: