"too many open files" triggers crash

benvp commented 2 years ago

Sanity checks

[X] I have searched github.com/withfig/fig/issues and there are no duplicates of my issue
[X] I have run fig doctor in the affected terminal session
[ ] I have typed 'git ' and included a screenshot of the Debugger (click the Fig icon in the macOS menu bar)

Issue Details

After using Fig for a while it will eventually fully crash my MacBook. This happens quite often so that I have to fully stop using Fig. It's pretty hard to give some details but the behaviour is like:

Type a command.
Fig tries to do something, then CPU usage will somewhat ramp up.
MacBook becomes unresponsive.
MacBook eventually turns black and restarts.

Occasionally the terminal spits out something like "too many open files" and crashes afterwards. Maybe this helps.

Environment

# Fig Diagnostics
## Fig details:
  - Fig version: Version 1.0.56 (B421) [German]
  - Bundle path: /Applications/Fig.app
  - Autocomplete: true
  - Settings.json: true
  - Accessibility: true
  - Number of specs: 0
  - Symlinked dotfiles: false
  - Only insert on tab: false
  - Keybindings path: 
  - Installation Script: true
  - PseudoTerminal Path: /Users/benvp/.fly/bin:/usr/local/opt/openssl/bin:/Users/benvp/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/benvp/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin:/Applications/Little Snitch.app/Contents/Components:/Library/Apple/usr/bin:/Users/benvp/.local/bin:/Users/benvp/.fig/bin:/Users/benvp/.local/bin:/Users/benvp/.local/bin:/Users/benvp/Library/Android/sdk/emulator:/Users/benvp/Library/Android/sdk/tools:/Users/benvp/Library/Android/sdk/tools/bin:/Users/benvp/Library/Android/sdk/platform-tools:/opt/homebrew/opt/fzf/bin
  - SecureKeyboardInput: false
  - SecureKeyboardProcess: <none>
## Hardware Info:
  - Model Name: MacBook Air
  - Model Identifier: MacBookAir10,1
  - Chip: Apple M1
  - Cores: 8 (4 performance and 4 efficiency)
  - Memory: 16 GB
## OS Info:
  - macOS 12.3.1 (21E258)
## Environment:
  - User Shell: /bin/zsh
  - Current Directory: /Users/benvp/Development/energy-management-app
  - CLI Installed: true
  - Executable Location: /Users/benvp/.local/bin/fig
  - Current Window ID: 47/% (org.tabby)
  - Active Process: zsh (873) - /dev/ttys001
  - Environment Variables:
    - PATH=/Users/benvp/.fly/bin:/usr/local/opt/openssl/bin:/Users/benvp/.asdf/shims:/opt/homebrew/opt/asdf/libexec/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/benvp/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin:/Applications/Little Snitch.app/Contents/Components:/Library/Apple/usr/bin:/Users/benvp/.local/bin:/Users/benvp/.fig/bin:/Users/benvp/.local/bin:/Users/benvp/.local/bin:/Users/benvp/Library/Android/sdk/emulator:/Users/benvp/Library/Android/sdk/tools:/Users/benvp/Library/Android/sdk/tools/bin:/Users/benvp/Library/Android/sdk/platform-tools:/opt/homebrew/opt/fzf/bin
    - TERM=xterm-256color
    - TERM_SESSION_ID=55554146-6F86-4D2E-99BB-B525F935897D
    - FIG_INTEGRATION_VERSION=8
    - FIG_TERM=1
    - FIG_TERM_VERSION=4.3.0
## Integrations:
  - SSH: false
  - TMUX: false
  - iTerm: installed!
  - Hyper: application is not present.
  - Visual Studio Code: installed!
  - Docker: false

mschrage commented 2 years ago

Hey @benvp, are you free to jump on a call so we can figure out what's going on here? https://calendly.com/mschrage/30min

benvp commented 2 years ago

hey @mschrage. I think I now somewhat can reproduce the issue. It seems to be related with an alias I did set in my .zshrc file.

There was a line

alias devdir="~/Development

Most of the time when I opened up a shell and typed that alias, fig would break my system. I did change it today to

alias dev="(cd ~/Development)"

which seems to solve the issue. At least I didn't experience the issue for the last couple of hours. Does that help?

mschrage commented 2 years ago

Hey @benvp! Thank you for sharing this additional information. I'll add a similar alias to my zshrc and see if I can repro.

Might be an issue with the way we expand aliases and then look for associated completion specs.

Integralist commented 2 years ago

I got a new MacBook M1 Pro about two weeks ago and have had the following issues with the laptop since day 1 (I also installed fig immediately upon receiving the laptop)...

Reporting "too many open files" to the point that the laptop becomes unresponsive and I have to force shutdown.
The laptop suddenly crashes without warning or ramp up in CPU/MEM or fan etc.

These issues happened quite literally multiple times (at least 5 times) every single day since getting the laptop.

As you can imagine, quite infuriating.

A few days ago I decided to stop using fig as I had a suspicion it might be causing the issue (from a software perspective it hasn't been very stable, for me at least) and immediately the problems with my laptop stopped.

It's been three days without a single incident. So although this doesn't necessarily mean fig was at fault. The fact that turning off fig is the only thing I've done and found it to have had an immediate positive effect is something I can't ignore and felt I needed to raise.

Possibly related issues:

mschrage commented 2 years ago

@Integralist can you share your dotfiles? Perhaps we can triangulate the root cause of this issue based on configurations that both you and @benvp share.

I'm sorry that Fig hasn't been working for you. There is clearly some edge case that we don't fully understand right now. Would love to figure out what's going on.

Please grab some time on my calendar if either of you are free to debug. https://calendly.com/mschrage/30min

Integralist commented 2 years ago

Hi @mschrage

My dot files can be found here: https://github.com/integralist/dotfiles

~~One thing I've noticed is that my .bash_profile looks to have a possibly old version of the fig code:~~

~~https://github.com/Integralist/dotfiles/blob/main/.bash_profile#L16~~

~~Where as my .bashrc has both a 'start' and 'end' block...~~

~~https://github.com/Integralist/dotfiles/blob/main/.bashrc#L2~~ ~~https://github.com/Integralist/dotfiles/blob/main/.bashrc#L705~~

~~Not sure if the ~/.bash_profile is supposed to contain any fig code any more.~~

UPDATE: Ignore my comment about the .bash_profile. I hadn't pushed an update to that file in a while it seems. I've done that now.

Integralist commented 2 years ago

@mschrage

OK so I've done a few things with my dotfiles...

I've configured my terminal (Alacritty) to use an interactive shell rather than a login shell.

This means my .bash_profile is no longer loaded, only my .bashrc.

NOTE: I'm also loading a .localrc via a call within .bashrc.

This also means there is no bash_profile specific fig code being loaded, only bashrc fig code being loaded (e.g. "$HOME/.fig/shell/bashrc.pre.bash" and "$HOME/.fig/shell/bashrc.post.bash")

NOTE: I've moved these inside conditional checks for the file as I prefer checking for files before sourcing them (e.g. if test -f "$HOME/some/file"; then ... fi).

What I'm noticing is that the fig code that's sourced is causing my .bashrc shell to be loaded twice...

Integralist commented 2 years ago

@mschrage last night I switched from Bash to Zsh to be sure there wasn't anything odd about Bash and Fig (as I had noticed issues with, just as an example, using Starship and Bash which disappeared when switching to Zsh), but since switching to Zsh I still see that the Fig zshrc.pre.zsh script is causing Zsh to be loaded twice (now that may or may not be an issue).

That said, now I'm using Zsh I'm going to turn Fig back on and see what happens (e.g. does the issues I've been having magically disappear).

Integralist commented 2 years ago

@mschrage OK. I've turned off fig again as after a few days of no laptop issues. Turning on fig has caused my machine to suddenly start becoming unresponsive and ultimately needing to be restarted (which then requires a force shutdown).

mschrage commented 2 years ago

Is it possible for you to reproduce this reliably @Integralist ?

The multiple loads of your rc file is expected and shouldn't cause issues - we fork into a different on the first execution.

Integralist commented 2 years ago

The multiple loads of your rc file is expected and shouldn't cause issues

👌🏻

Is it possible for you to reproduce this reliably?

Unfortunately not.

The closet I could say to consistency would be that I'm almost guaranteed at least one laptop shutdown a day. 😞

Previously when I was consciously monitoring the situation I was keeping track of lsof because I was seeing lots of "too many files open", but typically I would be too focused on work and forget about it until all of a sudden things would become unresponsive (and by that point it was too late for me to do anything), or I would start to see the "too many files" errors and then the unresponsiveness would quickly follow and nothing in the lsof output really indicated which process was having opening the majority of files (so maybe the "too many files open" error is a red herring).

Unless there are better suggestions I could try and use something like:

#!/usr/bin/env zsh

threshold=6000

while true; do
  local files=$(lsof | wc -l | xargs) # xargs for trimming white space
  local replace="\033[0K\r"
  local threshold=$threshold
  echo -ne "lsof:$files (threshold: $threshold)$replace"
  if [ $files -ge $threshold ]; then
    say "threshold $threshold exceeded"
    threshold=$( expr $threshold + 1000 )
  fi
  sleep 1
done

chaynabors commented 2 years ago

@mschrage

withfig / fig