Status buffer closes abruptly after refresh on giant repo

ioreshnikov commented 1 week ago

Description

Hey!

I am working on a giant monorepo and the status buffer is sadly not usable. So far I have noticed several issues:

When I open the status buffer in the new tab, the refresh process is triggered. On my repo it's a longer one, so a popup is shown. As soon as all the commands are run in the popup, it is closed automatically, but this also immediately closes the status buffer
Often this it shows an error message Invalid buffer id: X. Sometimes the error message is Buffer X does not exist
This never happens on a smaller projects

I will be happy to help you with figuring this out and can provide the logs if necessary

Thanks in advance!

Neovim version

NVIM v0.10.1 Build type: Release LuaJIT 2.1.1724512491

Operating system and version

macOS 14.2.1

Steps to reproduce

One obvious way is to clone a giant repo, for instance the Linux kernel, and try to open the status buffer in a new tab there.
I suspect there's a way to trigger the same behaviour in any repository by artificially inserting a timeout when running the git commands, but I am not familiar with neogit's source code enough to suggest the change

Expected behavior

Status buffer being slow, but usable

Actual behavior

Status buffer closing with an error message immediately after the refresh

Minimal config

-- NOTE: See the end of this file if you are reporting an issue, etc. Ignore all the "scary" functions up top, those are
-- used for setup and other operations.
local M = {}

local base_root_path = vim.fn.fnamemodify(debug.getinfo(1, "S").source:sub(2), ":p:h") .. "/.min"
function M.root(path)
  return base_root_path .. "/" .. (path or "")
end

function M.load_plugin(plugin_name, plugin_url)
  local package_root = M.root("plugins/")
  local install_destination = package_root .. plugin_name
  vim.opt.runtimepath:append(install_destination)

  if not vim.loop.fs_stat(package_root) then
    vim.fn.mkdir(package_root, "p")
  end

  if not vim.loop.fs_stat(install_destination) then
    print(string.format("> Downloading plugin '%s' to '%s'", plugin_name, install_destination))
    vim.fn.system({
      "git",
      "clone",
      "--depth=1",
      plugin_url,
      install_destination,
    })
    if vim.v.shell_error > 0 then
      error(string.format("> Failed to clone plugin: '%s' in '%s'!", plugin_name, install_destination),
        vim.log.levels.ERROR)
    end
  end
end

---@alias PluginName string The plugin name, will be used as part of the git clone destination
---@alias PluginUrl string The git url at which a plugin is located, can be a path. See https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols for details
---@alias MinPlugins table<PluginName, PluginUrl>

---Do the initial setup. Downloads plugins, ensures the minimal init does not pollute the filesystem by keeping
---everything self contained to the CWD of the minimal init file. Run prior to running tests, reproducing issues, etc.
---@param plugins? table<PluginName, PluginUrl>
function M.setup(plugins)
  vim.opt.packpath = {}                      -- Empty the package path so we use only the plugins specified
  vim.opt.runtimepath:append(M.root(".min")) -- Ensure the runtime detects the root min dir

  -- Install required plugins
  if plugins ~= nil then
    for plugin_name, plugin_url in pairs(plugins) do
      M.load_plugin(plugin_name, plugin_url)
    end
  end

  vim.env.XDG_CONFIG_HOME = M.root("xdg/config")
  vim.env.XDG_DATA_HOME = M.root("xdg/data")
  vim.env.XDG_STATE_HOME = M.root("xdg/state")
  vim.env.XDG_CACHE_HOME = M.root("xdg/cache")

  -- NOTE: Cleanup the xdg cache on exit so new runs of the minimal init doesn't share any previous state, e.g. shada
  vim.api.nvim_create_autocmd("VimLeave", {
    callback = function()
      vim.fn.system({
        "rm",
        "-r",
        "-f",
        M.root("xdg")
      })
    end
  })
end

-- NOTE: If you have additional plugins you need to install to reproduce your issue, include them in the plugins
-- table within the setup call below.
M.setup({
  plenary = "https://github.com/nvim-lua/plenary.nvim.git",
  telescope = "https://github.com/nvim-telescope/telescope.nvim",
  diffview = "https://github.com/sindrets/diffview.nvim",
  neogit = "https://github.com/NeogitOrg/neogit"
})
-- WARN: Do all plugin setup, test runs, reproductions, etc. AFTER calling setup with a list of plugins!
-- Basically, do all that stuff AFTER this line.
require("neogit").setup({}) -- For instance, setup Neogit

CKolkey commented 1 week ago

There's an "auto_show_console" config option - try setting it to false. I suspect whats going on is the popup is stealing focus, but the status buffer needs focus to refresh correctly.

ioreshnikov commented 1 week ago

I tried it and sadly it doesn't seem to help. The console is not being shown, but the behaviour is the same – once the refresh is over the status buffer is closed with an error.

After inspecting the logs I've noticed that the Registered repository for: is logged twice. Not sure if it's important. But in case it's relevant, a bit of extra info on the repository set up -- it's a monorepo with one common .git folder at the root of the project and multiple individual services. Each service is a separate typescript package, but there's no .git folder in any of them.

Also I am attaching the log file neogit.log

CKolkey commented 1 week ago

I checked out the linux kernel and got a good reproduction, so I'll use that as a performance benchmark and see what I can do :)

CKolkey commented 1 week ago

Figured out two things:

the "invalid buffer ID" issue is from the console popup. setting auto_show_console = false in neogit config fixes this. I'll do a proper fix for this too.
The slow loading speed is largely due to poor git log performance with --topo-order flag set, which is used behind the scenes. Try running git commit-graph write to pre-compute the commit graph, and check out the other steps here: https://github.com/jesseduffield/lazygit/issues/2875#issuecomment-1962138285

with those two things done, I can load the status buffer for the linux repo in about 2~3 seconds.

Also check out https://git-scm.com/docs/git-maintenance to automate this :)

NeogitOrg / neogit