search

whyrusleeping / gx

A package management tool
MIT License
1.88k stars 111 forks source link

Use a single queue to install dependencies in `installDeps` #195

Closed schomatis closed 5 years ago

schomatis commented 5 years ago

Increasing ratelim (e.g., to 20 goroutines working simultaneously) seems to improve the download speed (considerably, but more tests are needed to state that with some certainty) up until the first ~100 packages (direct dependencies of go-ipfs) but then it slows down again to download speeds similar to the default ratelim of 2 scenario, when new levels of dependencies are processed in a different call of installDeps which normally has a small number of child dependencies and the parallel fetch is not leveraged.

schomatis commented 5 years ago

This will need to split installDeps into the fetch and the post-install hook process, as the current recursive implementation traversed the dependency DAG in DFS order to run the gx-go doRewrite function (through the hook) from bottom to top (because doRewrite needs the dependencies of a package already available). Since the network fetch seems like the most expensive operation (in contrast with the local rewrite/disk-install) delaying the second process to unify the first one wouldn't defeat the performance gain mentioned above.

schomatis commented 5 years ago

Also, this separation would allow to parallelize the packages rewrite process which is now happening in DFS sequence (although I need to check the doRewrite logic to see if that parallelism is supported).

Stebalien commented 5 years ago

we currently parallelize rewriting files within a package but yeah, rewriting multiple packages at the same time would be nice.

schomatis commented 5 years ago

Actually testing a recent version of gx-go with your commit improving the rewrite logic already makes a huge performance improvement, so more parallelization there won't help much, the fetching part seems like the only bottleneck now.