cmd/go: vgo and the OpenBSD packager's dilemma

[qbit]

OpenBSD's constraints

Packaging on OpenBSD follows a fairly strict workflow:

1) The build process has no network access. 2) Distribution files must be fetched using tooling in OpenBSD base. In this case using ftp(1) (a tool similar to wget), which has extra security measures like pledge(2) and is run as a user with reduced privileges. 3) Packaging of binaries is prohibited. This is largely because of how OpenBSD is developed; -current is where ports and the base system are worked on. At any point ABI breakage can occur, rendering binaries built pre-ABI-change potentially broken.

The current state

Currently packaging Go applications is a trivial task given any of the following (listed in order of ease from the packager's perspective)

• Upstreams include a populated vendor directory in their source tree.
• Upstreams provide source tarballs that contain a vendor directory even though they don't track it in their repository.
• Upstreams use tooling such as dep which allow packagers to re-roll distribution files that include the vendor directory.

The dilemma

Though it hasn't been explicitly stated, one can infer[1] that the vendor directory is on its way out. If this is the case, it will dramatically complicate the build process for packages distributed using frameworks like the OpenBSD ports tree (likely all BSDs and probably a few Linux distributions will be impacted as well).

At first glance, the Proxy Server mentioned in the module doc seems like it might be a decent workaround, however, sticking to the OpenBSD workflow would necessitate rebuilding large portions of the "fetch" mechanism (remember the requirement for fetching with base tooling). It would also require adding infrastructure (and administration overhead) to store the source files.

[1]: https://research.swtch.com/vgo-module - The End of Vendoring

The questions

• Will the concept of the vendor directory be going away entirely?
• Is there a plan for official, public facing "Proxy Servers" that could be used to grab versioned tarballs?
• If the above are false, is there some mechanism planned that will allow for the constraints listed above to be met?

[bradfitz]

Will the concept of the vendor directory be going away entirely?

I don't think vendor is going away anytime soon. Its usage might drop in popularity over time, though.

Is there a plan for official, public facing "Proxy Servers" that could be used to grab versioned tarballs?

There is the https://github.com/gomods/athens effort.

I will defer to @bcmills @rsc on actual answers to your questions, though.

[bcmills]

Is there a plan for official, public facing "Proxy Servers" that could be used to grab versioned tarballs?

You'll soon be able to fetch zip files of sources using the go tool itself: see #26610. And note that the GOPROXY protocol supports static file paths, including arbitrary filesystem directories, so once you've downloaded those files you can plop them down wherever you like and start building.

Are those features sufficient for your requirements?

[qbit]

You'll soon be able to fetch zip files of sources using the go tool itself

If the versioned zip's can be fetched outside of the go tool, most of our issues would be resolved. As I mentioned previously, fetching is to be done using in-base tools only, and there is no network access during the actual build phase.

Once we have the zips, we could then use the GOPROXY environment variable to do the actual build after the zip files have been extracted.

[bcmills]

Put the zips in a tarball on an FTP server?

(Specifically, run go clean -modcache; go mod download to prime the module cache, then tar that up and plop it on an FTP server somewhere.)

[qbit]

If the zips are just compressed versions of a given dependencies source tree, we would just need to extract them into the proper location in GOPROXY (correct?), no need to re-compress them with tar.

We do something similar for Rust applications that use Cargo. During the fetch phase dep source is downloaded from crates.io (using the ftp(1) command) and extracted into the ports build env. Then the offline build phase happens.

[qbit]

Also, just to clarify - ftp on OpenBSD does http requests. There is no requirement on ftp servers :)

[bcmills]

I don't think you even need to unzip them. Just put the raw zips (and metadata files) in the GOPROXY tree and I think you're good to go.

[rasky]

Why is "a tarball of binary zip files" mentioned as an alternative to vendoring, which already works well, has a very simple semantic, and it's easy to handle?

I'm a little worried about Go's team stance on vendoring. Vendoring is a very important tool for many workflows that might not exist within Google but are very common in the world outside. It's easy to explain, easy to use, easy to version. I'm not sure how side-loading a tarball of binary files into a different directory and then pointing a variable called GOPROXY that counterintuitively supports direct filesystem paths can ever be a good alternative to vendoring.

[qbit]

Why is "a tarball of binary zip files" mentioned as an alternative to vendoring, which already works well, has a very simple semantic, and it's easy to handle?

I think this was mentioned as a OpenBSD specific solution to the packaging problem.

I'm not sure how side-loading a tarball of binary files into a different directory and then pointing a variable called GOPROXY that counterintuitively supports direct filesystem paths can ever be a good alternative to vendoring.

Again, this is something that is specific to 3rd party packagers building packages for their respective systems (in this case, OpenBSD). Not a solution for general consumption - or a proposal for how vendoring should be.

@bcmills I am going to start playing around with athens to see if I can get a working build env inside the OpenBSD ports tree. Assuming that all works fine (it seems like it will based off the features you have mentioned), the only remaining question will be: will there be a publicly available "proxy instance" to pull down the zip files.

[rasky]

I think this was mentioned as a OpenBSD specific solution to the packaging problem.

OpenBSD problem can be solved using go mod vendor, which creates/updates the vendor directory, and thus makes sure that go build -mod=vendor works without ever touching the network.

If the vendor directory is not going away, this is by far the easiest solution: it doesn't require external scripts, directories, environment variables, binary files. The fact that @bcmills is suggesting a far more complicated solution that allows to avoid using vendoring is what worries me.

[qbit]

So from my point of view, having something that allows us to download distfiles (versioned sources packaged as a zip or tarball) for each component (dep) of a given package is ideal. Then there is no need to hand roll a new tarball that contains the vendor directory.

I agree go mod vendor would work, but then we are back to hand-rolling distfiles for every go application in the ports tree.

[bcmills]

If the vendor directory is not going away, this is by far the easiest solution:

The vendor directory is not going away in the foreseeable future.

Is there a plan for official, public facing "Proxy Servers" that could be used to grab versioned tarballs?

We are planning to have public module mirrors available circa Go 1.13 (see https://blog.golang.org/modules2019).

[thepudds]

@qbit I am curious what your latest thinking here is, and whether you are using or considering the go mod vendor approach as suggested by @rasky, or perhaps the Athens-based approach you mentioned, or perhaps something else? I am asking in part out of curiosity, but also some potentially related questions are being asked elsewhere.

[thepudds]

See outline of a solution for a related set of problems in https://github.com/golang/go/issues/29410#issuecomment-515203754