MacVim hangs for minutes with hundreds of threads

[genivia-inc]

Describe the bug I'm a daily user of MacVim and my work depends on it. MacVim occasionally hangs for minutes with hundreds of threads (500 or more) when editing some specific large file(s). I had this happen several times, but only once a day at the most. It could be a syntax highlighting problem perhaps. It's difficult to reproduce exactly.

To Reproduce Detailed steps to reproduce the behavior:

1. Run open -a MacVim Forth500.s
2. Edit Forth500.s and perhaps some other files in other windows (but don't think that matters)
3. Change Syntax to Assembly/Turbo then navigate, search, type, etc.
4. Eventually the problem occurs, but I can't tell when and why. It may happen after a while, so it's not frequent, fortunately for me.

Expected behavior No hot laptop!

Screenshots N/A

Environment (please complete the following information):

• MacVim version: Custom Version 8.2.2681 (171)
• OS: MacOS 10.15.7
• Terminal: N/A

Additional context The Forth500.s file is here: https://github.com/Robert-van-Engelen/Forth500/blob/main/Forth500.s

[eirnym]

Hi, could you enlighten us with a bit more information about environment and test a few more things?

Please tell us:

• which MacVim version do you refer to?
• which macOS version do you use?
• how have you installed MacVim?
• have you overwrote vim supplied by MacVim by any other version of vim?

Please test your issue in terminal:

• using vim (in terminal, supplied by MacVim)
• using vim (in terminal, other version available)

Please test your issue with no plugins involved:

• using mvim --clean (no configurations or plugins involved)
• vim --clean (in terminal, supplied by MacVim)
• vim --clean (in terminal, other version available)

[genivia-inc]

See earlier info on version and OS. I run MacVim GUI. I don't use plugins (.vimrc attached as vimrc.txt), nor am I doing anything special other than just editing a few files with source code: C/C++, assembly (.s with Turbo syntax hl), Forth, markdown. The assembly file is large (see my previous comment). All other files are small/tiny. With this, MacVim generally uses 4.71GB of virtual memory and has 31,728 ports open when I use it for a few days.

I took two MacVim process samples (with MacOS Activity Monitor) of the MacVim process when it went crazy again, hanging for several minutes with hundreds of threads and all 8 cores running hot. The samples may offer some insight into the problem, because this intermittent problem is not easy to debug I'm sure. It happens about once a day when working with MacVim for hours.

vimrc.txt MacVimsamples.zip

I will also try and see if daily restarting MacVim and/or my system can avoid this problem, keeping an eye on the process stats.

[eirnym]

Thank you for sampling and your vimrc.

I have the same macOS version, I have x86_64 arch, I have the same version of MacVim installed from GitHub releases (no HomeBrew or MacPorts is involved)

In your vimrc I made GUI part be enabled all the time to have same configuration in GUI and terminal.

I opened your assembly file in using MacVim GUI, terminal vim from the same app bundle and /usr/bin/vim to make sure I covered all cases

I have no work within the file, so I scrolled a lot for few minutes in each editor and did some editing here and there.

I see no deviation in Application Monitor as you describe.

[eirnym]

I've also tried to install ugrep 3.3.7 with newest ctrlp.vim plugin (thanks!), but don't see any deviation at all.

[eirnym]

With all this, could you update your ugrep (which is not compatible with your configuration) and ctrlp.vim and try again?

I suppose, the old plugin version causes the problem. My vimrc you can find here (be careful with the dot in repository name)

[genivia-inc]

Thank you! I will follow your suggestions and will keep testing (I use MacVim a lot!). Like I said, it happens sometimes.

[genivia-inc]

This version of MacVim may have a memory leak that may explain the problem I'm seeing. I ran MacVim for three-four days without quitting. When I open the laptop daily and return to editing with MacVim, sometimes I see the spinning wheel. When that happens, the memory usage increases by 300MB or so. I started with less than 300MB and now at about 1GB reported by Activity Monitor on MacOS 10.15.7. I have just been editing the same three files (assembly and two MD files) in two windows. Eventually the spinning wheel issue gets progressively worse, hanging for minutes with several GB of memory use. That happens randomly, roughly once a day as reported earlier. The only remedy is to restart MacVim.

[eirnym]

could you try to edit this file in a terminal for a few days and see what happens, like /usr/bin/vim? the beginning of your configuration file could be as following to run same commands as GUI. To test with specific configuration file use /usr/bin/vim -u "/path/to/vimrc" -U "/path/to/vimrc". On my system with similar version /usr/bin/vim has version 8.1.2292 which is pretty close to MacVim.

"if has("gui_running")
  " Get the value of $PATH from a login shell.
  " If your shell is not on this list, it may be just because we have not
  " tested it.  Try adding it to the list and see if it works.  If so,
  " please post a note to the vim-mac list!
  if $SHELL =~ '/\(sh\|csh\|bash\|tcsh\|zsh\)$'
    let s:path = system("echo echo VIMPATH'${PATH}' | $SHELL -l")
    let $PATH = matchstr(s:path, 'VIMPATH\zs.\{-}\ze\n')
  endif
  " Change directory on startup.
  autocmd VimEnter * if getcwd()=="/" | if strlen(@%) | cd %:p:h | else | cd | endif | endif
  " If running in a Terminal window, set the terminal type to allow syntax highlighting.
  set guioptions-=T
  " set toolbar=text
  " set toolbariconsize=medium
  " set mousefocus
  set number
  set cursorline
"else
"  set term=builtin_ansi
"endif
" Grepping

[eirnym]

Maybe you have also an updated configuration to match newer versions of ugrep and ctrlp.vim?

[genivia-inc]

I've already made the ugrep and ctrlp.vim changes a few days ago and tested it out. I will try the GUI settings. It looks like you've commented out set term=builtin_ansi.

Besides increased memory, I also see a growing number of "Ports" in Activity Monitor: now 6273 open ports and 1GB memory. Restarting MacVim to edit the same files and the updated .vimrc shows 147MB with 276 ports. I'll run this for a few days and see what happens.

[eirnym]

Thank you, could you share your new vimrc for me to test it simultaneously?

[eirnym]

Could you share how did you installed MacVim?

[genivia-inc]

Vimrc: vimrc.txt

The problem is observed with the GUI, I have yet to test the terminal-based vim.

I installed MacVim a long time ago, pretty sure this was installed from a DMG file and later updated via the auto built-in updates (checking for updates reports fully up to date).

[eirnym]

Thank you for an updated version of vimrc file

I installed MacVim a long time ago, pretty sure this was installed from a DMG file

Thank you, it's very important that we use the same MacVim

[genivia-inc]

I still have this issue. I doubt it is the history mechanism that lets memory grow in big increments. I am not moving bug chunks of text around either and I've closed all windows except one (the assembly file). I am back to 2GB with 13,641 "ports". When the spinning disk appears, hundreds of threads will run for seconds to minutes. I noticed it happens when switching windows in MacVim and switching split screens with the cursor (with touchpad, I have not tried CTRL-W),

[eirnym]

I'm sorry to hear that. I'll profile MacVim. If I may to ask, could this file be separated into few ones. From my experience with Assembly language, I almost always separate code, use includes or link them together on the link stage.

[genivia-inc]

The original wasn't mine to begin with and I need/want to keep it this way for several reasons. I doubt it's the assembly file actually, since I witnessed the spinning wheel before when returning to work on a project with MacVim open, but didn't give it much attention. Restarting MacVim is something I usually do.

[eirnym]

I see, I'd split it anyway. It's way easier for me to find things in many files than in a single one.

I forgot to ask, which preferences (not vimrc) do you have? It's better to tell defaults than screenshots

[eirnym]

I need the output of the command:

$defaults read org.vim.MacVim

[genivia-inc]

Here you go. This intermittent problem is probably difficult to fix. I can live with it by restarting MacVim. But it should not have to be that way.

{
    MMAutosaveColumns = 152;
    MMAutosaveRows = 70;
    MMCurrentPreferencePane = General;
    MMLastWindowClosedBehavior = 2;
    MMTopLeftPoint = "{159, 1026}";
    NSFullScreenMenuItemEverywhere = 0;
    NSQuotedKeystrokeBinding = "";
    NSRepeatCountBinding = "";
    "NSWindow Frame FindAndReplace" = "113 351 490 169 0 0 1680 1028 ";
    "NSWindow Frame SUUpdateAlert" = "540 476 620 392 0 0 1680 1027 ";
    SUAutomaticallyUpdate = 0;
    SUCheckAtStartup = 0;
    SUEnableAutomaticChecks = 1;
    SUHasLaunchedBefore = 1;
    SULastCheckTime = "2021-09-14 20:58:47 +0000";
    SUSendProfileInfo = 0;
    SUUpdateGroupIdentifier = 3198150642;
    SUUpdateRelaunchingMarker = 0;
}

This is what I get when inspecting the MacVim process after it OK again (by comparison, when MacVim is unresponsive "spinning disk", hundreds of threads will run). I have very few other tasks running. MacVim consumes the more memory than all of them. The "real memory" size of MacVim is not bad, but Activity Monitor reports 2.18GB and this grows in big leaps suddenly (300MB or much more) when MacVim becomes unresponsive. Heap fragmentation may also be present, but that would not explain the unresponsiveness delay that gets worse over time when hundreds of threads are spawned.

[ychin]

Hi, I'm sorry for the late response. I'm taking a look, but since this seems to be a memory leak it's going to take a while to repro properly. If you see this again, can you do the following?

ps aux | grep -i vim

You should see something like this:

ychin            49390   0.0  0.0  4268424    720 s006  R+   12:16AM   0:00.00 grep --color=auto -i vim
ychin            49325   0.0  0.1  4467028  19704   ??  Ss   12:10AM   0:01.13 /Applications/MacVim.app/Contents/MacOS/Vim -g -f
ychin            49324   0.0  0.5  5200664  89640   ??  S    12:10AM   0:05.35 /Applications/MacVim.app/Contents/MacOS/MacVim

In MacVim, there is an overall macOS process called MacVim, and each window is a child process Vim (which is roughly the same process as terminal Vim). It would be interesting to see whether it's the parent process or the child process that's leaking.

[eirnym]

As per my tests, it seems to be parent process. Numbers obviously increasing, but I couldn't reproduce so huge growth as described, probably I do it wrong.

[ychin]

Huh ok that does make sense, given the thread count explosion. I couldn't really reproduce myself so far, but I did see thread count go up and down so that should give at least some lead. One thing that may be related is the CoreText stateful renderer but I'll need to take a look.

[ychin]

I'm still not super suer how to reproduce this. Is it basically opening said file, that do misc editing tasks? I have had it open for a while, and let the computer sleep/wake up, and do random searching for it, but it's staying in the 60's of MB of RAM usage for me…

[genivia-inc]

Still happening in the latest 8.2.3455 MacVim. Up to 2.4GB after a few days of editing some files. It's typically increasing memory when opening a new file from the terminal with open -a MacVim, which will open a new window in MacVim to edit the file.

The problem is not an issue the first day or so, but as time passes and more files are opened and closed (I keep only a handful open), this problem will always occur. It seems that MacVim tries to allocate memory while the process is swapped from physical to virtual memory. So when MacVim is not used for some time and other applications are active, swapping MacVim back to physical while opening a new file is probably the best symptom to describe this issue.