search

LPGhatguy / aftman

Aftman, the prodigal sequel to Foreman
MIT License
157 stars 16 forks source link

Download & install already trusted tools in parallel #26

Open filiptibell opened 2 years ago

filiptibell commented 2 years ago

The majority of the time spent in some of our CI jobs (~5 seconds or sometimes more) is spent downloading and installing tools managed by aftman, and I suspect this will grow slightly over time.

It would be great if aftman could download & install tools in parallel similar to what cargo does for tools that are already trusted or when using --no-trust-check. Maybe install could even be split into two phases, where the first is accepting trust for any new tools and the second is the download & install process.

LPGhatguy commented 2 years ago

This is a great idea. I think that splitting the trust check out, and then doing all installation in parallel is a great idea.

We'll need to take some care to make sure that we don't race with our existing data structures. Using something like rayon or embracing async will probably help a lot.

filiptibell commented 2 years ago

Made a PR for splitting the trust check from installation in https://github.com/LPGhatguy/aftman/pull/30 and would like to try and implement this as well.

We'll need to take some care to make sure that we don't race with our existing data structures.

InstalledToolsCache and how it is being used right now seems like the main thing to look out for, is there some other area to also consider?

Using something like rayon or embracing async will probably help a lot.

Embracing async seems like a good idea. After looking around it definitely requires a bit of refactoring, but it makes installation in parallel way easier, something like try_join_all could work if install_exact becomes async. What do you think?