search

taiki-e / cargo-hack

Cargo subcommand to provide various options useful for testing and continuous integration.
Apache License 2.0
628 stars 28 forks source link

Add --partition M/N to distribute builds #253

Closed ryoqun closed 2 months ago

ryoqun commented 3 months ago

if there are many workspace members, it takes very long to execute all runs. So, introduce --partition M/N to distribute runs across machines.

This pr is in draft still. If this addition of new flag sounds acceptable, I'll finish up it (maybe, writing tests and improve cli handling (edit: DONE), update docs?)

demo: https://buildkite.com/anza/agave/builds/7618#0190b45e-b9e3-4b64-a172-4d9fc6b52e0f

related: #180, #248 (... in the sense this is about accelerating the whole step of cargo hack in CIs)

the exact behavior is moderately mirrored from the counted partitioning of nextest: https://nexte.st/docs/ci-features/partitioning/#counted-partitioning

context: https://github.com/rust-lang/cargo/issues/10958#issuecomment-2214005359 and https://github.com/rust-lang/cargo/issues/8379#issuecomment-2201882061

ryoqun commented 3 months ago

@taiki-e hey, me again at different places. :) Could you review this in your free time?

ryoqun commented 3 months ago

Hey, sorry for delayed reply... I got some time to work on this.

Thanks for the PR! This seems to be an easy to use version of --print-command-list (#175) and would be a reasonable addition.

really thanks. :)

That said, it is usually always hard to review a PR that has no tests to indicate how it works...

sorry about that. I added tests.

ryoqun commented 3 months ago

@taiki-e hi, friendly ping. :)

ryoqun commented 2 months ago

@taiki-e i guess, maybe this pr can't be merged soon? this pr is blocking our ci fix: https://github.com/anza-xyz/agave/pull/2127, it seems that we have to install cargo-hack via our patched git repo...

ryoqun commented 2 months ago

@taiki-e just saw cargo-hack is released with this pr. now --partition is working at our ci like this: https://buildkite.com/anza/agave/builds/9044#01913215-4f43-4414-a2ae-5da9f173d9fa/170.

thank you very much!