[Feature] improve scoop update performance

[ahrib]

Feature Request

Performing scoop update appears to perform a git clone with complete commit history. Dependant on network connection, time elapsed since last update, number of commits, and number of packages - this can sometimes take a long time to perform.

Additionally, as the repository commit history grows, and the number of packages grow, this process will take an increasingly longer amount of time to complete.

Suggestions:

• limit cloning to a single branch so that unnecessary branches are not obtained (--single-branch --branch=[branch])
• if the commit history is not required, utilise shallow cloning (--depth 1)
• if the commit history is required, omit blobs and download them as required using blobless clones (--filter=blob:none)

[rashil2000]

• scoop update does not perform git clone. It performs git pulls.
• A vast majority of buckets have a single branch, so the problem of pulling in multiple unnecessary branches is not prominent.
• Is there a way to speed up pulling/fetching?

[ahrib]

Thanks @rashil2000

There seems to be a mixture of clone, fetch and pull within the project. https://github.com/search?q=repo%3AScoopInstaller%2FScoop+git_cmd&type=code

Potential ways forward

• if the commit history is not required try using shallow clone/fetch (--depth 1)
• if applicable --jobs option that performs fetches from multiple remote concurrently https://github.com/gitextensions/gitextensions/issues/9937 e.g. --jobs=4
• If the repository contains large binary files, consider using Git LFS to manage them. This can significantly reduce the size of the repository and speed up clone and fetch operations.

[rashil2000]

• if the commit history is not required try using shallow clone/fetch (--depth 1)

Commit history is required because when you run scoop update, a list of commits is shown as the update log. However, there's a config option to turn the update logs off, in which case we can use a shallow fetch.

• if applicable --jobs option that performs fetches from multiple remote concurrently Use parallel fetch to improve performance when multiple remotes are used gitextensions/gitextensions#9937 e.g. --jobs=4

Again, there's only one remote here, i.e. origin which is here on GitHub.

• If the repository contains large binary files, consider using Git LFS to manage them. This can significantly reduce the size of the repository and speed up clone and fetch operations.

The repos do not contain binary files at all.

[r15ch13]

The update process has already been improved to use parallel jobs. But it's still only in the develop branch. https://github.com/ScoopInstaller/Scoop/pull/5122

You can try it by running:

scoop config scoop_branch develop
scoop update

to revert this change run

scoop config scoop_branch master
scoop update
scoop update

[slbb]

The update process has already been improved to use parallel jobs. But it's still only in the develop branch. #5122

You can try it by running:

scoop config scoop_branch develop
scoop update

to revert this change run

scoop config scoop_branch master
scoop update
scoop update

@r15ch13 I switched to the development brand for a while. It's a little bit faster, but still not completely parallel running. I notice that it does create multiple git.exe processes, but only one git-remote-https.exe is running at a time, which means scoop update still updates only one repo at a time. When I use Neovim package manager to update plugins, multiple git-remote-https.exe runs at the same time. I think this is what parallel updating is expected to look like. But still big thank you for all your efforts.

[r15ch13]

Are you using PowerShell 5 or 7? Running ForEach-Object -Parallel only works in PS7.

[slbb]

Are you using PowerShell 5 or 7? Running ForEach-Object -Parallel only works in PS7.

Thanks to the information. Mine is PS5. I'll try PS7.