gobuffalo / buffalo

Rapid Web Development w/ Go
http://gobuffalo.io
MIT License
8.07k stars 573 forks source link

Many duplicate module downloads #1743

Closed dankinder closed 5 years ago

dankinder commented 5 years ago

Description

For a new project with GO111MODULE=on, buffalo seems to be downloading a wildly varying number of versions of the exact same packages. I assume this has to do with how it install various packages/binaries at their latest versions rather than a specific version.

Ultimately this just makes the build slower, especially when building in a fresh environment, like a docker build. For one of our projects the buffalo build container hits about 3GB and 1000 package downloads.

Steps to Reproduce the Problem

$ buffalo new coke
$ cd coke
$ wc -l go.sum
     983 go.sum
$ cat go.sum | cut -d' ' -f1 | sort | uniq -c | sort -nr
  59 golang.org/x/tools
  42 github.com/gobuffalo/genny
  39 golang.org/x/sys
  28 golang.org/x/net
  28 golang.org/x/crypto
  25 github.com/gobuffalo/buffalo-plugins
  23 github.com/gobuffalo/packr/v2
  21 github.com/gobuffalo/packd
  20 github.com/gobuffalo/pop
  19 github.com/gobuffalo/flect
  18 github.com/gobuffalo/release
  17 github.com/gobuffalo/buffalo
  16 github.com/gobuffalo/meta
  16 github.com/gobuffalo/events
  15 github.com/gobuffalo/packr
  15 github.com/gobuffalo/buffalo-pop
  13 github.com/gobuffalo/plush
  13 github.com/gobuffalo/envy
  12 github.com/gobuffalo/licenser
  12 github.com/gobuffalo/fizz
  11 github.com/gobuffalo/logger
  10 github.com/gobuffalo/httptest
... and so on ...

Expected Behavior

One would not expect to download the same package so many times at different versions.

Actual Behavior

golang.org/x/tools, for example, would be downloaded only once.

Info

Please run buffalo info and paste the information below where it says "PASTE_HERE".

``` $ buffalo info -> Go: Checking installation ✓ The `go` executable was found on your system at: /usr/local/go/bin/go -> Go: Checking minimum version requirements ✓ Your version of Go, 1.12.4, meets the minimum requirements. -> Go: Checking GOPATH ✓ You are using Go Modules, so no need to worry about the GOPATH. -> Go: Checking Package Management ✓ You are using Go Modules (`go`) for package management. -> Go: Checking PATH ✓ Your PATH contains /Users/dkinder/go/bin. -> Node: Checking installation ✓ The `node` executable was found on your system at: /usr/local/bin/node -> Node: Checking minimum version requirements ✓ Your version of Node, v9.11.2, meets the minimum requirements. -> NPM: Checking installation ✓ The `npm` executable was found on your system at: /usr/local/bin/npm -> NPM: Checking minimum version requirements ✓ Your version of NPM, 6.9.0, meets the minimum requirements. -> Yarn: Checking installation ✓ The `yarnpkg` executable was found on your system at: /usr/local/bin/yarnpkg -> Yarn: Checking minimum version requirements ✘ Your version of Yarn, 0.27.5, does not meet the minimum requirements. Minimum versions of Yarn are: * >=1.12 For help setting up your Yarn environment please follow the instructions for you platform at: https://yarnpkg.com/en/docs/install -> PostgreSQL: Checking installation ✓ The `postgres` executable was found on your system at: /Applications/Postgres.app/Contents/Versions/latest/bin/postgres -> PostgreSQL: Checking minimum version requirements ✘ : : Invalid Semantic Version -> MySQL: Checking installation ✓ The `mysql` executable was found on your system at: /usr/local/bin/mysql -> MySQL: Checking minimum version requirements ✓ Your version of MySQL, 8.0.16, meets the minimum requirements. -> SQLite3: Checking installation ✓ The `sqlite3` executable was found on your system at: /usr/bin/sqlite3 -> SQLite3: Checking minimum version requirements ✓ Your version of SQLite3, 3.16.0, meets the minimum requirements. -> Cockroach: Checking installation ✘ The `cockroach` executable could not be found on your system. For help setting up your Cockroach environment please follow the instructions for you platform at: https://www.cockroachlabs.com/docs/stable/ -> Buffalo: Checking installation ✓ The `buffalo` executable was found on your system at: /Users/dkinder/go/bin/buffalo -> Buffalo: Checking minimum version requirements ✓ Your version of Buffalo, v0.14.6, meets the minimum requirements. -> Buffalo: Application Details Pwd /Users/dkinder/code/coke Root /Users/dkinder/code/coke GoPath /Users/dkinder/go PackagePkg coke ActionsPkg coke/actions ModelsPkg coke/models GriftsPkg coke/grifts WithModules true Name coke Bin bin/coke VCS git WithPop true WithSQLite false WithDep false WithWebpack true WithNodeJs true WithYarn true WithDocker true WithGrifts true AsWeb true AsAPI false InApp true PackageJSON {map[]} -> Buffalo: config/buffalo-app.toml name = "coke" bin = "bin/coke" vcs = "git" with_pop = true with_sqlite = false with_dep = false with_webpack = true with_nodejs = true with_yarn = true with_docker = true with_grifts = true as_web = true as_api = false -> Buffalo: config/buffalo-plugins.toml [[plugin]] binary = "buffalo-pop" go_get = "github.com/gobuffalo/buffalo-pop" -> Buffalo: go.mod module coke go 1.12 require ( github.com/codegangsta/negroni v1.0.0 // indirect github.com/gobuffalo/buffalo v0.14.6 github.com/gobuffalo/envy v1.7.0 github.com/gobuffalo/events v1.4.0 // indirect github.com/gobuffalo/genny v0.3.0 // indirect github.com/gobuffalo/httptest v1.4.0 // indirect github.com/gobuffalo/makr v1.2.0 // indirect github.com/gobuffalo/meta v0.1.0 // indirect github.com/gobuffalo/mw-csrf v0.0.0-20190129204204-25460a055517 // indirect github.com/gobuffalo/mw-forcessl v0.0.0-20190224202501-6d1ef7ffb276 github.com/gobuffalo/mw-paramlogger v0.0.0-20190224201358-0d45762ab655 github.com/gobuffalo/packr v1.25.0 github.com/gobuffalo/packr/v2 v2.5.2 github.com/gobuffalo/pop v4.11.2+incompatible github.com/gobuffalo/suite v2.8.1+incompatible github.com/jackc/pgx v3.3.0+incompatible // indirect github.com/markbates/grift v1.0.6 github.com/unrolled/secure v1.0.0 ) ```
markbates commented 5 years ago

This is how Go Modules work, unfortunately. This behavior is changing in Go 1.13, but it's completely out of Buffalo's control for the moment.