search

bazelbuild / bazel-gazelle

Gazelle is a Bazel build file generator for Bazel projects. It natively supports Go and protobuf, and it may be extended to support new languages and custom rule sets.
Apache License 2.0
1.18k stars 377 forks source link

Use rctx.download to fetch go modules #1936

Open sluongng opened 4 days ago

sluongng commented 4 days ago

Currently in go_repository, unless the url attribute is set specifically, a go module is fetched via calling cmd/fetch_repo

https://github.com/bazelbuild/bazel-gazelle/blob/d16fc424cfcfdbd8893a267e500ec5d04ec0bbf0/internal/go_repository.bzl#L147

https://github.com/bazelbuild/bazel-gazelle/blob/d16fc424cfcfdbd8893a267e500ec5d04ec0bbf0/cmd/fetch_repo/main.go#L55

fetch_repo will in turn call go mod download to (a) download, (b) extract and (c) verify the content using GOSUMDB.

Because we are not using Bazel's native downloader via rctx.download or rctx.download_and_extract, the downloaded go modules are not cached via Bazel's --repository_cache= or does it benefit from --experimental_remote_downloader=.

We should be able to replicate (a) and (b) of go mod download by reconstructing the download URLs in starlark and feed it to rctx.download_and_extract to expand the zip. The URLs could be a mix of proxy download and direct download to replicate GOPROXY config.

For (c), we can create a custom binary called cmd/verify_zip to validate the downloaded content against GOSUMDB and GONOSUMDB config.

fmeum commented 4 days ago

Just a note: Even if we implement this, we still wouldn't benefit from the repository cache in its current form as we can't provide a hash for the .zip file itself, only for its contents.

sluongng commented 4 days ago

Right. The current hash in go.sum is the hash of the extracted archive directory using https://pkg.go.dev/golang.org/x/mod/sumdb/dirhash#HashDir and not of the .zip archive itself.

I wonder if there is something we can do in Bazel to support rctx.download_directory that is mapped to Remote Asset API FetchDirectory rpc.