x/tools: write release tool to run your package's callers' tests

[bradfitz]

In https://github.com/golang/go/issues/24301#issuecomment-390788506 I proposed in a comment:

We've been discussing some sort of go release command that both makes releases/tagging easy, but also checks API compatibility (like the Go-internal go tool api checker I wrote for Go releases). It might also be able to query godoc.org and find callers of your package and run their tests against your new version too at pre-release time, before any tag is pushed. etc.

And later in that issue, I said:

With all the cloud providers starting to offer pay-by-the-second containers-as-a-service, I see no reason we couldn't provide this as an open source tool that anybody can run and pay the $0.57 or $1.34 they need to to run a bazillion tests over a bunch of hosts for a few minutes.

@SamWhited and @mattfarina had objections (https://github.com/golang/go/issues/24301#issuecomment-390792056, https://github.com/golang/go/issues/24301#issuecomment-390794717) about the inclusivity of such a tool, so I'm opening this bug so we don't cause too much noise in the other issue.

Such a tool would involve:

• querying a service (such as godoc.org or anything implementing the "Go workspace abstraction") to find your callers
• running the tests (or a fraction thereof) either
  • locally, possibly overnight
  • on a cloud provider of your choice (AWS Fargate, Azure Container Service, Digital Ocean, GCP, etc)

... and telling you if they pass before your change, but fail after your change, so you can release a new version of your package with confidence.

In local mode, it'd use your local GOOS/GOARCH. On cloud, Linux containers are cheapest, but it could also spin up Windows VMs like we do with Go. (Each Go commit gets a fresh new Windows VM that boots in under a minute and runs a few tests and then the VM is destroyed).

None of this costs much, and the assumption is that this would be used by people (optionally) who are willing to pay for the extra assurance, and/or those whose time needed to fix regressions later is worth more than the cloud costs.

And maybe some cloud CI/CD company(s) could sponsor such builders.

[mattfarina]

Here are a few problems to be solved/questions...

1. Can you share how my bold question here is solved by this?
2. How is this simpler or easier for people than something like dep that has configurable maximums on ranges?
3. What about all the people who don't use a release tool (because it's complicated and extra work to setup) and break compatibility?
4. How does costing work for this? If you're popular you pay more. Doesn't this enable those with a lot of money while being worse for those who don't?

[SamWhited]

you're taking issue over an optional feature of a hypothetical design

I had missed the "optional feature" part, it sounds to me when reading it that the point was that the tool spun up machines on some cloud service and used them to download lots of code and run tests. I'm not sure that it would be practical to run on a laptop if you have a popular project, but I'm less concerned and it seems worth exploring if that's the case.

[AlexRouSg]

In addition to pulling all sources and running in one VM, since the users of your package may have their own test environment (like cgo usage) and their own os/arch combo, also to allow closed source users of your package to have this benefit...

How about the tool somehow notifies users of your package (uses a list file for closed source users) to pull your changes and test on their builders.

Maybe even have a service/daemon tool to automate this somehow and report back to you through smtp/email.

[AlexRouSg]

btw would Go itself use the tool to check if some change to the compiler or std broke some popular package?

[randall77]

This sounds like a security minefield. A nefarious actor just has to import your project in order to get you to run their code for them. Any such tests would have to be very well sandboxed.

[AlexRouSg]

@randall77 i.e. bitcoin miners

One of the reasons I suggested they run the code.

[bradfitz]

@randall77 @AlexRouSg, all of this assumes sandboxes. I assume we'd limit CPU execution time & run things under gVisor where possible, and for Windows at least use firewalled VMs (limiting network in/out, so they can't e.g. send email). What I don't know how to do is nice sandboxing for GOOS=windows on GOHOSTOS=windows without new VMs. Maybe we assume Hyper-V or something.

[AlexRouSg]

@bradfitz do you think having a mode to inform/let users of your package to run the test on their builders is something to look at?

I feel like people with non trivial build processes or uses of C/C++ libs would be left out.

[dolmen]

Prior art in the Perl community: CPAN Testers. Community testing of Perl modules releases since 1998. Example report.

[mbyio]

This sounds like a great idea. It's like a generalization of what Rust does with https://github.com/rust-lang-nursery/crater, except instead of testing the compiler for breaking changes, it's testing libraries. Many many times I see people accidentally introduce breaking changes not because they are being irresponsible but because they didn't realize what the full scope of the change was. This tool could really help with that.

I do suspect that only the best supported libraries will end up using it, unless some company makes it easy to do test runs.

I guess the main downside that I see is some people might think it's okay to make a breaking change if they find that no tests failed. For some packages I bet there are fewer tests using them directly (eg. logging libraries). But I don't think that's really a reason to not make this tool.

[rsc]

Not entirely sure why this is in the proposal process. Is there a proposal for the Go team to evaluate?

[bradfitz]

@rsc, yeah, no need for the proposal process. Removed.

[bcmills]

See also #26420. As far as I can tell, these issues are basically dual: the tool proposed here is “test my dependencies with the current version of my module”, and #26420 is “test the current version of my module with the versions as seen by dependencies”.