bug: Lazy spawning "too many" git instances on windows

[miversen33]

Did you check docs and existing issues?

• [X] I have read all the lazy.nvim docs
• [X] I have searched the existing issues of lazy.nvim
• [X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)

NVIM v0.9.1

Operating system/version

Windows 10

Describe the bug

When setting up my dotfiles on a windows machine, I notice that Lazy fails to install pretty much any of the plugins I have specified for it. Additionally, my windows machine slows to an absolute crawl for about 5 minutes.

After digging into the task manager, I noticed that there are tons of Git for Windows and sh.exe processes that are being started and killed all the time. I have tested this both on my work laptop (which has an antivirus) and a personal Windows 10 vm (which only has Windows defender) and noticed the same result. A screenshot can be found below

Eventually Lazy will just fail with an error by each plugin stating that it was unable to download them

My plugin list isn't exactly "huge", its about 74 plugins. It can be found here though I imagine it will be an issue with pretty much any list of plugins as the issue happens during git clone and not setup.

If there is more info you need (or if there are things you need tested), please let me know!

Steps To Reproduce

1. Have a sizable list of plugins
2. Have a windows machine
3. Open neovim with your lazy configuration

Expected Behavior

I am going to guess that the issue is that Lazy is spawning a new git process for each plugin (maybe all at once, I am not completely sure and haven't really dug into the source of Lazy). Is there a way to configure a limit to the number of processes that are spawned and/or have Lazy use a process pool to reuse processes as opposed to creating new ones? It seems git on Windows is quite a bit slower than git on Linux, which is likely what is contributing to this issue.

It's worth noting that if you manually comment out blocks of plugins and let Lazy setup "less" plugins at once, it does kick through eventually. Basically manually performing the above process limiting.

Repro

-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
    vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
    vim.fn.system({ "git", "clone", "--filter=blob:none", "https://github.com/folke/lazy.nvim.git", lazypath, })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
    "folke/tokyonight.nvim",
    -- add any other plugins here
    -- A semi sizable list of plugins
    {
        url = "https://git.sr.ht/~whynothugo/lsp_lines.nvim",
    },
    "folke/trouble.nvim",
    "kevinhwang91/nvim-hlslens",
    "nvim-neorg/neorg",
    "nvim-lua/plenary.nvim",
    "nvim-treesitter/nvim-treesitter",
    "Mofiqul/vscode.nvim", -- Vscode type theme
    'nvim-zh/colorful-winsep.nvim',
    "nvim-lualine/lualine.nvim", -- Neovim status line
    "kyazdani42/nvim-web-devicons",
    "onsails/lspkind-nvim",
    "f-person/git-blame.nvim",
    "noib3/nvim-cokeline", -- Neovim Tab/Buffer Bar.
    'edluffy/specs.nvim',
    "RRethy/vim-illuminate",
    "phaazon/mind.nvim", -- Mind mapping/note taking
    "nvim-lua/plenary.nvim",
    'shellRaining/hlchunk.nvim',
    "kkharji/sqlite.lua", -- Neovim SQlite database
    "nvim-treesitter/nvim-treesitter", -- Neovim treesitter
    'nvim-treesitter/playground',
    "nvim-telescope/telescope.nvim", -- Fuzzy Finder
    "rcarriga/nvim-notify", -- Notify
    "ziontee113/icon-picker.nvim", -- Nerdfont picker
    "stevearc/dressing.nvim",
    "nvim-telescope/telescope.nvim",
    "RaafatTurki/hex.nvim", -- Enables hex editor for neovim
    "kevinhwang91/nvim-ufo", -- Better folding? Idk we will see
    "kevinhwang91/promise-async",
    "famiu/bufdelete.nvim", -- Better buffer deletion
    "anuvyklack/hydra.nvim", -- Keymaps
    "mrjones2014/smart-splits.nvim", -- Neovim better split handling?
    "stevearc/aerial.nvim", -- Better code outline??
    "akinsho/toggleterm.nvim", -- Neovim Floating Terminal Framework
    "m-demare/hlargs.nvim",
    "numToStr/Comment.nvim", -- Neovim Commenting
    "ojroques/nvim-osc52", -- Neovim clipboard integration
    "nvim-pack/nvim-spectre", -- Better search and replace?
    "williamboman/mason.nvim", -- Neovim Language Tools (LSP, Debugger, Formatter, Linter, etc)
    "tikhomirov/vim-glsl",
    "neovim/nvim-lspconfig", -- Neovim LSP Setup
    "williamboman/mason-lspconfig.nvim", -- Mason lsp config bindings
    "rcarriga/nvim-dap-ui", -- UI for Dap
    "mfussenegger/nvim-dap", -- Debugger, setup below
    "mfussenegger/nvim-lint", -- Neovim linter
    "mhartington/formatter.nvim", -- Neovim formatter
    "hrsh7th/cmp-nvim-lsp", -- Neovim LSP feeder for cmp
    "jbyuki/one-small-step-for-vimkind", -- Neovim Dap
    "simrat39/rust-tools.nvim", -- Neovim Rust Tools
    "mfussenegger/nvim-jdtls", -- Neovim java tools
    "hrsh7th/nvim-cmp", -- Neovim autocompletion
    "rcarriga/cmp-dap", -- Neovim autocomplete for dap
    "L3MON4D3/LuaSnip", -- Neovim Lua based snippet manager
    "saadparwaiz1/cmp_luasnip", -- Neovim LuaSnip autocompletion engine for nvim-cmp
    "hrsh7th/cmp-nvim-lsp", -- vim/neovim snippet stuffs
    "KadoBOT/cmp-plugins", -- Neovim plugin autocompletion
    "hrsh7th/cmp-buffer", -- vim/neovim snippet stuffs
    "hrsh7th/cmp-path", -- vim/neovim snippet stuffs
    "hrsh7th/cmp-cmdline", -- vim/neovim snippet stuffs
    "hrsh7th/cmp-nvim-lsp-signature-help",
    "windwp/nvim-autopairs", -- Auto pairs
    "theHamsta/nvim-dap-virtual-text", -- Neovim DAP Virutal Text lol what else do you think this is?
    "ray-x/cmp-treesitter", -- Neovim snippet for treesitter (Maybe replace the buffer completion?)
    "nvim-neo-tree/neo-tree.nvim", -- File Explorer
    "MunifTanjim/nui.nvim",
    "onsails/lspkind-nvim", -- Document Symbols
    "miversen33/netman.nvim", -- Remove Resource Browser
    "folke/lsp-colors.nvim", -- Neovim create missing lsp color highlight groups
    "haringsrob/nvim_context_vt",
    "uga-rosa/ccc.nvim",
    "kevinhwang91/nvim-bqf",
    "hkupty/iron.nvim",
    "Fildo7525/pretty_hover",
    "ellisonleao/glow.nvim",
    "toppair/peek.nvim",
    'nmac427/guess-indent.nvim',
    'fedepujol/move.nvim',
    "TimUntersberger/neogit",
    "Bekaboo/dropbar.nvim",
}
require("lazy").setup(plugins, {
    root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

[miversen33]

After a bit of thinking while submitting this, I wonder if this issue is actually present on linux machines as well, if they have a very slow network connection.

What appears to be happening is that Windows is actually killing the processes after a while, likely to due it believing they are "stale" or "bad". I imagine that the Linux Kernel will do the same thing with "stuck" linux processes if there are a ton of them and they all "appear" to be not moving. Just a theory, I am not completely sure here.

[folke]

It's not an issue on Linux and also not an issue on my windows machine.

You can configure the concurrency in lazy. See for more details in the readme.

[folke]

Fyi: synching takes less than 3seconds for me for over 100 plugins

[miversen33]

Is "Works on my machine" really a valid closure reason...?

Setting concurrency to "something lower" in lazy does not resolve the issue, I am still seeing tons of Git for Windows and sh.exe processes and I am seeing CPU usage of the powershell process that Neovim is running in max out.

I can confirm that setting the Concurrency value to something low (such as 15 in this case) does allow Lazy to successfully complete, it just takes forever on my Windows environment. Note, on the same machine in WSL it does not take nearly as long. Additionally running on linux machines in my network I see comparable times to what you see mentioned

More details I am able to find.

I can manually run git to pull any of the plugins listed and it runs as fast as you would expect a git clone to run, so I don't believe its actually an issue with Git (or the network connection on the windows machine).

I cannot run Profile without setting concurrency to something low as eventually Lazy fails due to Git processes being killed (I am unsure if they are being killed by Lazy or Windows, I did not initially notice that Lazy has a git timeout variable).

When I set concurrency to 15, it takes about 7 minutes to fully download all 75 plugins on windows using the above listed repro.lua.

On WSL on the same machine it takes less than 1 second for lazy to finish

And on a completely separate linux machine on the network

All these are ran on the same network over the same wifi network. What else can I provide to help here? Something is wrong and I don't really know what.

[folke]

I think this is just a Windows issue. Not sure if there's anything that can be done about that. Windows just doesn't seem to be able to cope well with a lot of processes.

Lazy just spawns the git processes.

On Windows for me it is also slower, but not that much compared to linux.

[folke]

You seem to be using https://gitforwindows.org/ This starts git in an emulated bash environment. Pretty sure that's the problem here. I use regular git on Windows.

[folke]

See also https://github.com/git-for-windows/git/issues/4459

[rimrul]

I use regular git on Windows.

Could you elaborate on that? What do you mean by "regular git on Windows"? Do you mean you download the git source code and build it yourself? Do you mean wsl git? Or do you mean the windows binaries available for download from git-scm.com (those are the same binaries as gitforwindows.org)?

This starts git in an emulated bash environment.

That shouldn't be the case, unless you're explicitly launching that bash or running a part of Git that is implemented as a shell script.

[folke]

winget install Git.Git. Might indeed be the same then. No idea. I don't regularly use Windows.

As you can see from OP's screenshot, there's a lot of sh.exe processes started as well, so I assume those are from the git exe.

This might also be useful: https://github.com/git-for-windows/git/wiki/Diagnosing-performance-issues

As I said before, all lazy does is use libuv to spawn a git process. I'm not doing anything special with shells or anything like that.

On my Windows 11 system, I don't have any problems. It's a bit slower than linux, but not a big difference.

I've also just pushed a change to configure concurrency on Windows by default. It seems that Windows simply can't cope with a lot of processes being spawned.

[rimrul]

I cannot run Profile without setting concurrency to something low as eventually Lazy fails due to Git processes being killed (I am unsure if they are being killed by Lazy or Windows, I did not initially notice that Lazy has a git timeout variable).

Does increasing that timeout also cause lazy to install all (or most) plugins?

I'm assuming the git clone processes take over 2 minutes each with high-ish concurrency for whatever reason and Lazy ends up killing them.

[folke]

You can change the timeout in the lazy config as well.

[rimrul]

winget install Git.Git. Might indeed be the same then. No idea. I don't regularly use Windows.

That is indeed also the same.

As you can see from OP's screenshot, there's a lot of sh.exe processes started as well, so I assume those are from the git exe.

Yes, though I assume those are spawned from within the initial git clone process to do some thing that git uses a shell for. Maybe --recurse-submodules.

On my Windows 11 system, I don't have any problems. It's a bit slower than linux, but not a big difference.

Yes, that could very well be  git-for-windows/git#4459, that issue seems to mainly affect domain joined machines.

I've also just pushed a change to configure concurrency on Windows by default. It seems that Windows simply can't cope with a lot of processes being spawned.

Seems like a good workaround.

[miversen33]

I'll give this update a try this morning.

I want to call out that my testing so far has been on both a domain joined machine using VPN (my work computer) and a non-domain joined, LAN connected virtual machine (my personal windows test vm), with the same results. So wildly different setups, and I don't believe the domain joined bit specifically is causing any issues here. I'll post an update later once I have had an opportunity to test this out

[miversen33]

@folke so good news and bad news. The good news, your update (the default change the concurrency) does appear to "work". Lazy is quite slow still, but that is due to there only being (from the looks of it) processors * 2 processes allowed to run at once. It does not cause timeouts, windows does not blow up, and it does finish all plugin downloads from the above Repro.

As we can see, still very slow but I agree that the lack of speed here is likely a Git for Windows thing.

The bad news, I did still see lots of processes spawn up, but not nearly the same as before, with my CPU usage for the parent process capping at 10-14% usage. So much better!

I appreciate you taking the time to investigate this more :)

[cyaconi]

I'm facing the same timeout problems in Sonoma, so I upgraded to 14.5, but it's the same. Sync (S) used to run very quickly, but since a couple of days I've been getting timeouts in some of the plugins on the list, usually 4-5 and not always the same. I did not change the concurrency configuration because it worked well as is, so I am not sure if this is a new issue on macOS, lazy, or Github.