Hugo does not build with go install out of the box anymore

bep commented 5 years ago

I have been doing some work requiring me to clean my modcache several times, and without any GOPROXY set (which I assume is the default), Hugo does not build on my MacBook.

go clean -modcache && go install;

Looking at the log I'm still a bit puzzled by the high number of seemingly unrelated new deps coming from the Google Cloud add-ons. I assume these are indirect test dependencies, but I don't understand why I need to download them to my PC. I guess this ties into https://github.com/golang/go/issues/26913#issuecomment-411976222
With GOPROXY=https://proxy.golang.org it builds fine, which means that we as a minimum need to update mage script + documentation.
Consider if we can opt-in to this, somehow. I don't think making a build tag for this new feature will help out.

/cc @vangent

The runs below is master vs 0.55.5:

go: finding golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52
go: finding gonum.org/v1/gonum v0.0.0-20181121035319-3f7ecaa7e8ca
go: error loading module requirements
( go clean -modcache && go install; )   168.28s  user 102.04s system 85% cpu 5:16.02 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                198000 MB
page faults from disk:     25650
other page faults:         7069035

go: extracting golang.org/x/text v0.3.0
go: extracting github.com/jdkato/prose v1.1.0
( go clean -modcache && go install; )   70.57s  user 36.62s system 142% cpu 1:15.35 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                478540 MB
page faults from disk:     17659
other page faults:         2800501

With GOPROXY=https://proxy.golang.org:

go: extracting google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19
go: extracting github.com/jdkato/prose v1.1.0
( go clean -modcache && go install; )   32.90s  user 25.99s system 53% cpu 1:50.78 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                732620 MB
page faults from disk:     5352
other page faults:         640553

go: extracting github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
( go clean -modcache && go install; )   5.65s  user 7.11s system 46% cpu 27.353 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                111228 MB
page faults from disk:     3439
other page faults:         50679

bep commented 5 years ago

Now go get -u from fails for me from github.com/gohugoio/hugo even with a PROXY set :

▶ go get -u
go: finding github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc
go: finding github.com/shurcooL/issues v0.0.0-20190120000219-08d8dadf8acb
go: finding github.com/docker/distribution v2.7.1+incompatible
go get: upgrading github.com/ugorji/go/codec@v0.0.0-20181012064053-8333dd449516: unexpected status (https://proxy.golang.org/github.com/ugorji/go/codec/@v/list): 410 Gone
go: github.com/testcontainers/testcontainer-go@v0.0.4: parsing go.mod: unexpected module path "github.com/testcontainers/testcontainers-go"
go: finding github.com/Azure/azure-sdk-for-go v28.1.0+incompatible
go: finding golang.org/x/net v0.0.0-20190322120337-addf6b3196f6
go: sourcegraph.com/sourcegraph/go-diff@v0.5.1: parsing go.mod: unexpected module path "github.com/sourcegraph/go-diff"
go get: error loading module requirements

Curiously enough it fails with different errors when running with GOPROXY=direct go get -u

go: finding google.golang.org/api v0.0.0-20180829000535-087779f1d2c9
go: finding github.com/smartystreets/assertions v0.0.0-20180820201707-7c9eb446e3cf
go: finding golang.org/x/crypto v0.0.0-20190404164418-38d8ce5564a5
go: github.com/golang/lint@v0.0.0-20190409202823-959b441ac422: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding k8s.io/api v0.0.0-20180806132203-61b11ee65332
go: finding github.com/lyft/protoc-gen-validate v0.0.

I guess this somehow ties into https://github.com/golang/go/issues/26913#issuecomment-411976222

larzza commented 5 years ago

On master I'm doing this successfully (in a docker build with default GOPROXY-setting):

git clone --branch master --depth 1 https://github.com/gohugoio/hugo.git hugo
go clean -modcache && go install -ldflags 'all=-w -extldflags "-static"' -tags extended

The difference in size of pkg between master (1.9G) and v0.55.5 (260M) is really quite big. Hopefully that isn't a necessary requirement...!?

master /go/pkg# du -hs    
1.9G

v0.55.5:/go/pkg# du -hs
260M

bep commented 5 years ago

@larzza what is your default GOPROXY setting? I will have a more detailed look into this problem later, but I suspect most of this issue comes from https://github.com/golang/go/issues/26913#issuecomment-411976222 -- which goes away if you use a proxy server (then you only need to look at the go.mod and only need to download the actual code if you for some reason want to build the relevant package/test).

larzza commented 5 years ago

@bep Ok, I read the proxy-comment. Hopefully it will be better with using a proxy.

My default is GOPROXY=""

What is your? Have you tried the different ways of unsetting the GOPROXY?:

If GOPROXY is unset, is the empty string, or is the string "direct", downloads use the default direct connection to version control systems

From the go docker container:

$ docker run -it --rm  golang:latest bash

# go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build835505629=/tmp/go-build -gno-record-gcc-switches"

larzza commented 5 years ago

which goes away if you use a proxy server (then you only need to look at the go.mod and only need to download the actual code if you for some reason want to build the relevant package/test).

Ok, I see, this already works as expected! By using a locally installed goproxy (Athens) I've verified that the size of the downloaded modules is much smaller when building Hugo from master.

/go/pkg# du -hs
450M    .

hook-s3c commented 5 years ago

❯ go clean -modcache && go install
go: finding github.com/hashicorp/go-uuid v1.0.0
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/go-ini/ini v1.25.4
go: finding github.com/kisielk/errcheck v1.1.0
go: pack.ag/amqp@v0.8.0: unrecognized import path "pack.ag/amqp" (parse https://pack.ag/amqp?go-get=1: no go-import meta tags ())
go: finding github.com/gogo/protobuf v1.1.1
go: finding golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09
go: pack.ag/amqp@v0.11.0: unrecognized import path "pack.ag/amqp" (parse https://pack.ag/amqp?go-get=1: no go-import meta tags ())
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd
go: finding google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb
go: finding github.com/prometheus/client_golang v0.9.1
go: finding github.com/Azure/azure-pipeline-go v0.1.8
go: finding github.com/oklog/ulid v1.3.1
go: finding github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d
go: finding github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
go: finding gopkg.in/check.v1 v1.0.0-20161208181325-20d25e280405
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding github.com/go-logfmt/logfmt v0.3.0
go: finding github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce
go: finding github.com/go-kit/kit v0.8.0
go: finding github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8
go: finding github.com/cespare/xxhash v1.1.0
go: finding github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954
go: error loading module requirements

bep commented 5 years ago

@hook-s3c I just tried go clean -modcache && go install and it works fine for me now (go1.12.5 darwin/amd64)

bep commented 5 years ago

An added note, compared to 0.55.6:

~/go/pkg: 174M => 497M hugo binary: 25M => 39M

temoto commented 5 years ago

Hello. Sorry for maybe unpleasant question. Could you name a recommended last Hugo version before this amqp nonsense started creeping into static website generator? I could git bisect repo for less dependencies, but of course you would know better about bugfixes. Don't need any features above render template, include header/footer and os.Exec after success.

Thanks for work on Hugo.

bcmills commented 4 years ago

go clean -modcache && go install seems to be working for me with go1.14beta1, even when bypassing the proxy explicitly (using GOPROXY=direct). Is this still an issue?

bep commented 4 years ago

Is this still an issue?

No, it works for me, too.

github-actions[bot] commented 2 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

gohugoio / hugo

Hugo does not build with go install out of the box anymore #5954