search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.24k stars 623 forks source link

Enable crossplatform speculation of requests and their inputs. #8353

Open hrfuller opened 4 years ago

hrfuller commented 4 years ago

As follow up work to #8090 we want to improve concurrency of multi-platform request speculation. Requests configured for different platforms may well have dependencies on different inputs, platform specific compiler binaries for instance. In situation like this the time needed to build the input binaries could vary greatly, say if one was cached while the other needed to be generated. In our current model of speculation all inputs to a rule would have to complete before speculation could begin. We would like to create a new syntax for declaring process requests and their input_files so that request speculation time can include the time needed to generate inputs. This probably relates to being able to push multiple params into the build graph with one Get (#7490)

stuhood commented 1 year ago

This was designed in https://docs.google.com/document/d/1hS0n6vp-hBxy4Xo-q9QA_elofzQtOubjfN4nSeIgp90/edit: #11148 covers cross-platform remote execution without speculation, and will be closed soon via the environments work from #7735 and #13682.

stuhood commented 1 year ago

Also relates to https://github.com/pantsbuild/pants/issues/16873, which is about platform-agnostic process outputs.