x/pkgsite/cmd/pkgsite: local pkgsite with local GOROOT serves wrong contents for standard-library packages

bcmills commented 1 year ago

What version of Go are you using (`go version`)?

$ go version
go version devel go1.20-d03231d9ce Tue Jan 3 18:53:52 2023 +0000 linux/amd64

$ go version -m $(which pkgsite)
/usr/local/google/home/bcmills/bin/pkgsite: go1.20rc2
    path    golang.org/x/pkgsite/cmd/pkgsite
    mod golang.org/x/pkgsite    v0.0.0-20230109194820-035bfc02f3fa  h1:zwhsydzVshWFPyFE/ar8IspWLZgppufg2qBb7RTxnUU=
    dep cloud.google.com/go v0.99.0 h1:y/cM2iqGgGi5D5DQZl6D9STN/3dR/Vx5Mp8s752oJTY=
    dep cloud.google.com/go/cloudtasks  v1.0.0  h1:087lrtubHYhq4hdL8EN+xIdwprH9fx36XXCf6R/ZxLQ=
    dep cloud.google.com/go/container   v1.0.0  h1:k3IMcHEEJR4pQmj4/gNeUuA4azhW7i2RPX8xGhyneZw=
    dep cloud.google.com/go/errorreporting  v0.1.0  h1:z40EhrjRspplwbpO+9DSnC4kgDokBi94T/gYwtdKL5Q=
    dep cloud.google.com/go/logging v1.4.2  h1:Mu2Q75VBDQlW1HlBMjTX4X84UFR73G1TiLlRYc/b7tA=
    dep cloud.google.com/go/monitoring  v1.1.0  h1:ZnyNdf/XRcynMmKzRSNTOdOyYPs6G7do1l2D2hIvIKo=
    dep cloud.google.com/go/secretmanager   v1.0.0  h1:Wbw6lsRrpatsE8GVpuwYqImn+sY5DmRjaEImYPwcSMY=
    dep cloud.google.com/go/storage v1.18.2 h1:5NQw6tOn3eMm0oE8vTkfjau18kjL79FlMjy/CHTpmoY=
    dep cloud.google.com/go/trace   v1.0.0  h1:laKx2y7IWMjguCe5zZx6n7qLtREk4kyE69SXVC0VSN8=
    dep contrib.go.opencensus.io/exporter/prometheus    v0.1.0  h1:SByaIoWwNgMdPSgl5sMqM2KDE5H/ukPWBRo314xiDvg=
    dep contrib.go.opencensus.io/exporter/stackdriver   v0.13.4 h1:ksUxwH3OD5sxkjzEqGxNTl+Xjsmu3BnC/300MhSVTSc=
    dep contrib.go.opencensus.io/integrations/ocsql v0.1.4  h1:kfg5Yyy1nYUrqzyfW5XX+dzMASky8IJXhtHe0KTYNS4=
    dep github.com/Masterminds/squirrel v1.5.2  h1:UiOEi2ZX4RCSkpiNDQN5kro/XIBpSRk9iTqdIRPzUXE=
    dep github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7  h1:YoJbenK9C67SkzkDfmQuVln04ygHj3vjZfd9FL+GmQQ=
    dep github.com/aws/aws-sdk-go   v1.34.29    h1:4Yw8eC4nCXiIVmHJO5PD4oh0vI/df5o6cYTVzFV7vWA=
    dep github.com/aymerick/douceur v0.2.0  h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
    dep github.com/beorn7/perks v1.0.1  h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
    dep github.com/census-instrumentation/opencensus-proto  v0.3.0  h1:t/LhUZLVitR1Ow2YOnduCsavhwFUklBMoGVYUCqmCqk=
    dep github.com/cespare/xxhash/v2    v2.1.2  h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
    dep github.com/chris-ramon/douceur  v0.2.0  h1:IDMEdxlEUUBYBKE4z/mJnFyVXox+MjuEVDJNN27glkU=
    dep github.com/dgryski/go-rendezvous    v0.0.0-20200823014737-9f7001d12a5f  h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
    dep github.com/emirpasic/gods   v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg=
    dep github.com/evanw/esbuild    v0.14.10    h1:+7c1VNndl7uLLxVEeRH4rOUz0Y+nrSw8xfmE9rGtrtw=
    dep github.com/ghodss/yaml  v1.0.0  h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
    dep github.com/go-git/gcfg  v1.5.0  h1:Q5ViNfGF8zFgyJWPqYwA7qGFoMTEiBmdlkcfRmpIMa4=
    dep github.com/go-git/go-billy/v5   v5.3.1  h1:CPiOUAzKtMRvolEKw+bG1PLRpT7D3LIs3/3ey4Aiu34=
    dep github.com/go-git/go-git/v5 v5.4.2  h1:BXyZu9t0VkbiHtqrsvdq39UDhGJTl1h55VW6CSC4aY4=
    dep github.com/go-redis/redis/v8    v8.11.4 h1:kHoYkfZP6+pe04aFTnhDH6GDROa5yJdHJVNxV3F46Tg=
    dep github.com/go-redis/redis_rate/v9   v9.1.2  h1:H0l5VzoAtOE6ydd38j8MCq3ABlGLnvvbA1xDSVVCHgQ=
    dep github.com/golang-migrate/migrate/v4    v4.15.1 h1:Sakl3Nm6+wQKq0Q62tpFMi5a503bgGhceo2icrgQ9vM=
    dep github.com/golang/groupcache    v0.0.0-20200121045136-8c9f03a8e57e  h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
    dep github.com/golang/protobuf  v1.5.2  h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
    dep github.com/google/go-cmp    v0.5.8  h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
    dep github.com/google/licensecheck  v0.3.1  h1:QoxgoDkaeC4nFrtGN1jV7IPmDCHFNIVh54e5hSt6sPs=
    dep github.com/google/safehtml  v0.0.3-0.20211026203422-d6f0e11a5516    h1:pSEdbeokt55L2hwtWo6A2k7u5SG08rmw0LhWEyrdWgk=
    dep github.com/googleapis/gax-go/v2 v2.1.1  h1:dp3bWCh+PPO1zjRRiCSczJav13sBvG4UhNyVTa1KqdU=
    dep github.com/gorilla/css  v1.0.0  h1:BQqNyPTi50JCFMTw/b67hByjMVXZRwGha6wxVGkeihY=
    dep github.com/hashicorp/errwrap    v1.0.0  h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA=
    dep github.com/hashicorp/go-multierror  v1.1.0  h1:B9UzwGQJehnUY1yNrnwREHc3fGbC2xefo8g4TbElacI=
    dep github.com/hashicorp/golang-lru v0.5.1  h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU=
    dep github.com/imdario/mergo    v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU=
    dep github.com/jackc/chunkreader/v2 v2.0.1  h1:i+RDz65UE+mmpjTfyz0MoVTnzeYxroil2G82ki7MGG8=
    dep github.com/jackc/pgconn v1.10.1 h1:DzdIHIjG1AxGwoEEqS+mGsURyjt4enSmqzACXvVzOT8=
    dep github.com/jackc/pgio   v1.0.0  h1:g12B9UwVnzGhueNavwioyEEpAmqMe1E/BN9ES+8ovkE=
    dep github.com/jackc/pgpassfile v1.0.0  h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
    dep github.com/jackc/pgproto3/v2    v2.2.0  h1:r7JypeP2D3onoQTCxWdTpCtJ4D+qpKr0TxvoyMhZ5ns=
    dep github.com/jackc/pgservicefile  v0.0.0-20200714003250-2b9c44734f2b  h1:C8S2+VttkHFdOOCXJe+YGfa4vHYwlt4Zx+IVXQ97jYg=
    dep github.com/jackc/pgtype v1.9.1  h1:MJc2s0MFS8C3ok1wQTdQxWuXQcB6+HwAm5x1CzW7mf0=
    dep github.com/jackc/pgx/v4 v4.14.1 h1:71oo1KAGI6mXhLiTMn6iDFcp3e7+zon/capWjl2OEFU=
    dep github.com/jbenet/go-context    v0.0.0-20150711004518-d14ea06fba99  h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
    dep github.com/jmespath/go-jmespath v0.4.0  h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
    dep github.com/kevinburke/ssh_config    v0.0.0-20201106050909-4977a11b4351  h1:DowS9hvgyYSX4TO5NpyC606/Z4SxnNYbT+WX27or6Ck=
    dep github.com/lann/builder v0.0.0-20180802200727-47ae307949d0  h1:SOEGU9fKiNWd/HOJuq6+3iTQz8KNCLtVX6idSoTLdUw=
    dep github.com/lann/ps  v0.0.0-20150810152359-62de8c46ede0  h1:P6pPBnrTSX3DEVR4fDembhRWSsG5rVo6hYhAB/ADZrk=
    dep github.com/lib/pq   v1.10.2 h1:AqzbZs4ZoCBp+GtejcpCpcxM3zlSMx29dXbUSeVtJb8=
    dep github.com/matttproud/golang_protobuf_extensions    v1.0.2-0.20181231171920-c182affec369    h1:I0XW9+e1XWDxdcEniV4rQAIOPUGDq67JSCiRCgGCZLI=
    dep github.com/microcosm-cc/bluemonday  v1.0.5  h1:cF59UCKMmmUgqN1baLvqU/B1ZsMori+duLVTLpgiG3w=
    dep github.com/mitchellh/go-homedir v1.1.0  h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
    dep github.com/prometheus/client_golang v1.7.1  h1:NTGy1Ja9pByO+xAeH/qiWnLrKtr3hJPNjaVUwnjpdpA=
    dep github.com/prometheus/client_model  v0.2.0  h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M=
    dep github.com/prometheus/common    v0.10.0 h1:RyRA7RzGXQZiW+tGMr7sxa85G1z0yOpM1qq5c8lNawc=
    dep github.com/prometheus/procfs    v0.6.0  h1:mxy4L2jP6qMonqmq+aTtOx1ifVWUgG/TAmntgbh3xv4=
    dep github.com/russross/blackfriday/v2  v2.1.0  h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
    dep github.com/sergi/go-diff    v1.1.0  h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    dep github.com/xanzy/ssh-agent  v0.3.0  h1:wUMzuKtKilRgBAD1sUb8gOwwRr2FGoBVumcjoOACClI=
    dep github.com/yuin/goldmark    v1.4.13 h1:fVcFKWvrslecOb/tg+Cc05dkeYx540o0FuFt3nUVDoE=
    dep github.com/yuin/goldmark-emoji  v1.0.1  h1:ctuWEyzGBwiucEqxzwe0SOYDXPAucOrE9NQC18Wa1os=
    dep go.opencensus.io    v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M=
    dep go.uber.org/atomic  v1.6.0  h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk=
    dep golang.org/x/crypto v0.0.0-20210921155107-089bfa567519  h1:7I4JAnoQBe7ZtJcBaYHi5UtiO8tQHbUSXxL+pnGRANg=
    dep golang.org/x/mod    v0.6.0-dev.0.20220419223038-86c51ed26bb4    h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
    dep golang.org/x/net    v0.0.0-20220722155237-a158d28d115b  h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
    dep golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8  h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg=
    dep golang.org/x/sync   v0.0.0-20220722155255-886fb9371eb4  h1:uVc8UZUe6tr40fFVnUP5Oj+veunVezqYl9z7DYw9xzw=
    dep golang.org/x/sys    v0.0.0-20220722155257-8c9f86f7a55f  h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
    dep golang.org/x/text   v0.3.7  h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    dep golang.org/x/vuln   v0.0.0-20221019194534-41df7bda74f3  h1:UC9wxs/szrifgXyDocLmCPP+JF80lgIc5WshJqh08aA=
    dep golang.org/x/xerrors    v0.0.0-20200804184101-5ec99f83aff1  h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
    dep google.golang.org/api   v0.63.0 h1:n2bqqK895ygnBpdPDYetfy23K7fJ22wsrZKCyfuRkkA=
    dep google.golang.org/genproto  v0.0.0-20211208223120-3a66f561d7aa  h1:I0YcKz0I7OAhddo7ya8kMnvprhcWM045PmkBdMO9zN0=
    dep google.golang.org/grpc  v1.43.0 h1:Eeu7bZtDZ2DpRCsLhUlcrLnvYaMK1Gz86a+hMVvELmM=
    dep google.golang.org/protobuf  v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
    dep gopkg.in/warnings.v0    v0.1.2  h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
    dep gopkg.in/yaml.v2    v2.4.0  h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
    build   -buildmode=exe
    build   -compiler=gc
    build   CGO_ENABLED=1
    build   CGO_CFLAGS=
    build   CGO_CPPFLAGS=
    build   CGO_CXXFLAGS=
    build   CGO_LDFLAGS=
    build   GOARCH=amd64
    build   GOOS=linux
    build   GOAMD64=v1

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/bcmills/.cache/go-build"
GOENV="/usr/local/google/home/bcmills/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/usr/local/google/home/bcmills/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/bcmills"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/google/home/bcmills/go-review"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/google/home/bcmills/go-review/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel go1.20-d03231d9ce Tue Jan 3 18:53:52 2023 +0000"
GCCGO="/usr/local/google/home/bcmills/bin/gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1375491053=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Check out https://go.dev/cl/460999 in a local copy of the Go repo.

Using the go command in that GOROOT, run:

cd $(go env GOROOT)/src
pkgsite -http=:8080 -gorepo=$(go env GOROOT) -list=false

View http://localhost:8080/context.

What did you expect to see?

pkgsite documentation for the version of the context package in $GOROOT/src/context.

What did you see instead?

The local pkgsite displays documentation for go1.19, but claims that it comes from the repository at $(go env GOROOT).

jba commented 1 year ago

pkgsite is using the local repo you provide at the version it reports. Moreover, it is doing exactly what you asked. You asked for doc at the "latest" version (since you didn't specify a version), and its notion of latest is "greatest semver tag, ignoring pre-releases." That is indeed 1.19.4.

Using context@master should work. But it doesn't; I get a 500. Passing to @jamalc for further investigation.

seankhliao commented 1 year ago

Using latest for the standard library seems inconsistent with pkgsite's behaviour when passed a path for an outside module. There it will display files as is, disregarding any versioning information.

bcmills commented 1 year ago

Moreover, it is doing exactly what you asked. You asked for doc at the "latest" version (since you didn't specify a version), and its notion of latest is "greatest semver tag, ignoring pre-releases." That is indeed 1.19.4.

It didn't display docs for 1.19.4. It displayed docs for 1.19. (But maybe the tags in my copy of the repo are stale?)

But also: I didn't ask it for latest. I asked it for an unversioned path. I would expect that if I tell it a -gorepo explicitly it should treat the checked-out version as the default version to show. (If I pointed -gorepo at a bare repo with no checkout, then I could see defaulting to the latest release.)

Using context@master should work. But it doesn't; I get a 500. Passing to @jamalc for further investigation.

Yep. I did try context@master, and got a 500 page claiming that it was not supported.

jamalc commented 1 year ago

I believe this is working as intended. CL 347549 only mentions speeding up serving stdlib by avoiding the full zip download. This use case was never supported.

bcmills commented 1 year ago

So... do we have a recommended way to test pkgsite rendering for standard-library changes?

jamalc commented 1 year ago

We should add support. As a workaround for now can you take these steps:

Rename the module in /src from std to something like golang.org/x/toolchain.
Run pkgsite -list=false ./src from the repo root.
View docs at http://localhost:8080/golang.org/x/toolchain/context.

benhoyt commented 1 year ago

@jamalc I'm trying to submit some doc/example changes to the stdlib, and struggling with this issue as well. I've tried your workaround above of renaming the module, but I get an error HTTP 424 "This page is not supported by this datasource" when requesting http://localhost:8080/golang.org/x/toolchain/context (or other stdlib modules). Server logs:

$ pkgsite -list=false ./src
2023/06/22 21:33:42 Info: go/packages.Load(["{golang.org/x/toolchain /home/ben/h/go/src}/..."]) loaded 0 packages from ./src in 8.052336ms
2023/06/22 21:33:42 Info: Listening on addr http://localhost:8080
2023/06/22 21:33:44 Info: FetchDataSource: fetching golang.org/x/toolchain/context@latest
2023/06/22 21:33:44 Info: FetchDataSource: fetched golang.org/x/toolchain/context@latest using <nil> in 16.292µs with error golang.org/x/toolchain/context@latest: not found
2023/06/22 21:33:44 Info: FetchDataSource: fetching golang.org/x/toolchain@latest
2023/06/22 21:33:44 Info: FetchDataSource: fetched golang.org/x/toolchain@latest using <nil> in 1.534µs with error golang.org/x/toolchain@latest: not found
2023/06/22 21:33:44 Info: returning 424 (Failed Dependency) for error serveUnitPage(ctx, w, r, ds, &{golang.org/x/toolchain/context unknownModulePath latest}): servePathNotFoundPage(w, r, "golang.org/x/toolchain/context", "latest"): 424 (Failed Dependency): <nil> (epage=&{{   <nil> false false   false false    }  {<h3 class="Error-message">This page is not supported by this datasource.</h3>} <nil>})

jamalc commented 1 year ago

Interesting, its not loading any packages at startup. Does running without -list=false have the same result?

benhoyt commented 1 year ago

Hmm, no, running without -list=false actually works, thanks. The first request takes several seconds, but it seems to work, and it works after that. Here are the logs:

$ pkgsite ./src
2023/06/24 10:53:00 Info: go/packages.Load(["all"]) loaded 537 packages from ./src in 169.293176ms
2023/06/24 10:53:00 Info: Listening on addr http://localhost:8080
2023/06/24 10:53:06 Info: FetchDataSource: fetching golang.org/x/toolchain/slices@latest
2023/06/24 10:53:06 Info: FetchDataSource: fetched golang.org/x/toolchain/slices@latest using <nil> in 17.403µs with error golang.org/x/toolchain/slices@latest: not found
2023/06/24 10:53:06 Info: FetchDataSource: fetching golang.org/x/toolchain@latest
2023/06/24 10:53:11 Info: FetchDataSource: fetched golang.org/x/toolchain@latest using *fetch.goPackagesModuleGetter in 4.991958955s with error <nil>
2023/06/24 10:53:12 Warning: fetching url from deps.dev: Get "https://deps.dev/_/s/go/p/golang.org%2Fx%2Ftoolchain/v/v0.0.0/exists": context deadline exceeded

matloob commented 1 year ago

I think this should work now after https://golang.org/cl/517915. You need to be in the GOROOT/src directory for it to work. It still takes a while to do the initial load of a page in the standard library but loads faster after the first are faster.

golang / go