search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.29k stars 628 forks source link

add pre-release integration test of remote execution #10801

Open tdyas opened 4 years ago

tdyas commented 4 years ago

Before making a release, the release process should run an integration test of the example-python repository against one or more of three major REAPI implementations: Buildbarn, Buildfarm, and BuildGrid. Configurations to set these up are in the remote-apis-testing framework.

The motivation for this request is issues like https://github.com/pantsbuild/pants/issues/10799 where Protobuf codegen failed in remote execution, but a subsequent change may have fixed it. It would be nice to catch these regressions earlier.

stuhood commented 3 years ago

@tdyas, @Eric-Arellano, @illicitonion : How would you feel about exposing our mock/stub servers from the Rust code to Python tests? Although spawning a real instance of a server would assist with testing successful cases, the inability to inject failures would make it hard to write other kinds of tests.

It's possible that I'm too optimistic about how "real" a test you can write against the mocks/stubs (TestServer requires mocking execution at the gRPC method level), but you could probably write simpler RuleRunner-level (not quite unit, but not quite integration) tests that ran only a handful of processes...? Exposing the TestServer to Python would probably have a fairly large API surface area too.

Having typed this out, maybe it is more of a stretch goal to allow for injecting failure here, and fetching and spawning a simple instance of one of the servers would be easier.

tdyas commented 3 years ago

I'll have to research a little of what features the mocks have.

As a related idea, in addition in this, what do you think of a pre-release run of remote-apis-testing to ensure no regressions against BuildBarn, Buildfarm, and Buildgrid? Much of the work to spin up each server and run Pants against them is already there.

stuhood commented 3 years ago

As a related idea, in addition in this, what do you think of a pre-release run of remote-apis-testing to ensure no regressions against BuildBarn, Buildfarm, and Buildgrid? Much of the work to spin up each server and run Pants against them is already there.

Probably worthwhile... possibly really challenging to debug if things were to break there in CI? But depending on how the "source copy" of Pants ended up used inside the docker "client" image (if you could easily edit the source and rerun), then could be fine.

illicitonion commented 3 years ago

@tdyas, @Eric-Arellano, @illicitonion : How would you feel about exposing our mock/stub servers from the Rust code to Python tests? Although spawning a real instance of a server would assist with testing successful cases, the inability to inject failures would make it hard to write other kinds of tests.

I'm not sure it's going to be worth the hassle... If we're going to go down that path, I'd probably suggest writing a simple instrument-able local execution service, rather than a stub one (which would be fairly simple to do, but at that point, maybe it's worth just spinning up one of the "real" ones). I suspect that for actual fault injection, you either want to be writing a rust unit test (so using TestServer as is from rust), or using a RuleRunner to stub out the whole process execution.

Exposing the TestServer to Python would probably have a fairly large API surface area too.

Yes, I suspect so :)