dascandy / evoke

Magic build tool
Apache License 2.0
166 stars 17 forks source link

Measure resource usage of processes to better limit invocation count #72

Closed dascandy closed 2 years ago

dascandy commented 5 years ago

Is your feature request related to a problem? Please describe. When evoke builds a large project it can at times build all long-building files last, or run 4 linkers at the same time causing them to exceed memory availability

Describe the solution you'd like Evoke should measure the maxrss, clock time taken, cpu time taken, disk IO requirements of a process as it exits (wait4 or getrusage) and use that in future rebuilds to schedule them more efficiently.

Describe alternatives you've considered Human specification could work, but is counter to Evoke's design goal. Any reactive solution is bound to be bad. One option that could work is to be able to poison targets - if it at any point failed / crashed, then mark it poisoned & re-run as singleton job. That would also "help" with fragile tests, although it would not necessarily understand why it fixes things, and it would also cause bad build performance because all often-failing things are then serialized.

Additional context N/A

dascandy commented 3 years ago

Memory use is taken into account so far. There is a project database that tracks past memory use for a given compile step and it tries to not go above the available memory during compilation.

Average CPU usage and IO load are not measured or taken into account. These are more flexible though; memory is a hard limit.

dascandy commented 2 years ago

Closing since fixed on branch.