chshersh / tool-sync

🧰 Download pre-built binaries of all your favourite tools with a single command
https://crates.io/crates/tool-sync
Mozilla Public License 2.0
69 stars 16 forks source link

Adding async support for downloading assets #67

Open MitchellBerend opened 1 year ago

MitchellBerend commented 1 year ago

As it stands now the tool downloads all tools sequentially. This can actually take a very long time if there are a lot of tools defined in the config, or if there is some type of ansible/salt-stack deployment going on that also might run sequentially. I suppose these downloads can actually be done asynchronously. There are however a number of things to consider.

  1. This would make it harder for people that are not already familiar with the code base, to jump in and contribute.
  2. This would add a dependency.
  3. The ureq library does not support async capibilities so this would need to be replaced (with reqwuest for instance).
  4. There is a rate limit of 60 calls per hour on the github api ^1.
  5. We would have to restructure the way tags are collected before the tool downloads the actual assets.

This should be weighed against the potential speed increase that can come from asynchronous downloads.

Im not sure if im missing something here so I would appreciate feedback (from all sources) on this.

chshersh commented 1 year ago

I was thinking about this as well. Indeed, introduction of concurrency would make the code more complex and require to change a lot of things.

I've created the following to help a bit with the speed (although, it's not clear how to do it at the moment):

For now, it would be great to get more information about potential improvement. Say, write two shell scripts using curl to download assets sequentially and in parallel and compare the numbers. So we'll see whether it's actually worth it.