search

lexical-lsp / lexical

Lexical is a next-generation elixir language server
874 stars 80 forks source link

Fix crashes on OTP 27 by disabling ETS table compression #796

Closed zachallaun closed 2 months ago

zachallaun commented 2 months ago

Fixes #788.

We've had mysterious OTP 27 crashes for quite some time now, but I was finally able to track it down to what I believe is a bug in :ets. When the :compressed table option is used, the system either segfaults or crashes with a memory-related error on any call that traverses the entire table. In our case, it was a call to :ets.foldl/3, but other full traversals like :ets.tab2list/1 or :ets.tab2file/2 also crash.

The temporary fix to get things working is to just not use :compressed on OTP 27. Unfortunately, this has a very sizeable impact on memory:

# memory usage after indexing Lexical itself
# using :erlang.memory()

# OTP 26
[
  total: 200081864,
  processes: 97217168,
  processes_used: 97187736,
  system: 102864696,
  atom: 1466705,
  atom_used: 1452046,
  binary: 2958496,
  code: 35956838,
  ets: 14065992 # 14MB
]

# OTP 27
[
  total: 481878632,
  processes: 79210832,
  processes_used: 79184048,
  system: 402667800,
  atom: 1360193,
  atom_used: 1358458,
  binary: 50157968,
  code: 26346438,
  ets: 279298656 # 279MB
]

We may want to add user-level configuration to disable indexing for those who have to run OTP 27 on very large projects, but I think that should go in another PR.

One final note: I haven't reported this upstream yet because I was trying to create a more minimal reproduction, but I've unfortunately not been able to. I'm going to spend a bit more time this morning trying to reproduce, but I'll likely submit a bug report later regardless; perhaps someone more familiar with ETS will have suggestions for how to minimally recreate the environment that's causing the crash.