Expose go.sum for use by packaging systems.

[qbit]

Packaging frameworks like OpenBSD's follow a fairly strict workflow (FreeBSD and pkgsrc are have similar systems) :

• The build process has no network access.
• Distribution files must be fetched prior to building and by using tooling in OpenBSD base. In this case using ftp(1) (a tool similar to wget), which has extra security measures like pledge(2) and is run as a user with reduced privileges.
• Packaging of binaries is prohibited. This is largely because of how OpenBSD is developed; -current is where ports and the base system are worked on. At any point ABI breakage can occur, rendering binaries built pre-ABI-change potentially broken.

Currently with athens I am very close to being able to package Go applications under the above constraints. I am able to:

• Download versioned dependencies for a given module.
• Build a module by either setting GOPROXY=file:// or extracting the zip files into vendor.

The last remaining issue is getting a complete list of dependencies. If the go.sum file is exposed, I would be able to get a complete list of modules for a given application.

Alternatively, I could download the go.mod file and query/download the modules for each dependency, however, this is duplicating a lot of the work already gone by the Go tooling.

This was previously discussed here, which is how I found out about athens in the first place! :D

[Kunde21]

This is a little beyond the initial design of the proxy, but builder functionality that can "bundle" a dependency list would be especially helpful for these types of build systems. Basically, a 'GET' request endpoint for a specific module/version (w/ the hash for verification) and the server does a download of that module, validates it, unzips it, then runs a go mod download to fetch all the dependency modules. Bundle the /go/mod/pkg/cache/download (tar/zip/whatevs) and send that back with the module.

Build systems would then need to call that endpoint, unpack the bundle into a directory, set that directory as $GOPROXY, and build the binary. This would reduce the need to re-invent the dependency downloading for each build system. By pointing the go mod download call towards the local Athens proxy, all of the modules would be populated in storage and the build system would benefit (especially in low-bandwidth environments).

[qbit]

By pointing the go mod download call towards the local Athens proxy, all of the modules would be populated in storage and the build system would benefit (especially in low-bandwidth environments).

"local Athens proxy" here means the uncompressed files that $GOPROXY points to, correct? Would one still need to call go mod download, or is go build sufficient at that point?

[Kunde21]

I'm using "local" in reference to the go mod download command, not the user's machine. The endpoint that is collecting all the dependencies would set its $GOPROXY to the same Athens proxy server. There's no good reason to build on the server.

On the user's machine, $GOPROXY would point to the downloaded modules and the go build would run.

[qbit]

Just so I know we are on the same page, from my perspective:

• Users: will install signed packages distributed by OpenBSD that contain the built binaries.
• Build servers: will do fetching of all source files, then be disconnected from the network, at which point building of source can continue. Once everything is built, it is signed and distributed to mirrors for end users to install.

There's no good reason to build on the server.

I assume this means the server running Athens? If so, I agree - no need to do the build there, however, if you mean there is no good reason for us to build on our build servers, I can go into more detail on the numerous (quite good) reasons why it’s done this way.

[Kunde21]

Yes, I mean the Athens server wouldn't be building binaries since it doesn't know (or care) the target architecture.

Build servers/user install/etc would be able to fetch all the needed source code with a single GET call, since it cannot use the go tool during that phase to do package selection.