Huge memory usage (10GB+) during Phoenix debugging on a new project

fnicastri commented 1 year ago

Current behavior

During a debug session thru ElixirLS Debugger there is a huge spike in memory usage during Ecto's query. Loading the index page (no queries) the memory spike is under a gigabyte.

I've noticed this behavior previously in the last few months.

Steps to Reproduce

make a new phoenix project (standard Postgres), add a new model and corresponding views with:
mix phx.gen.html
start the debugger and add a new record
reload the model index page
in activity monitor watch one of the two beam.smp processes consume GBs of memory (10+)

Expected behavior

Not consuming over 10 GB of memory ;)

Environment

Elixir & Erlang versions (elixir --version):


Erlang/OTP 25 [erts-13.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Elixir 1.14.4 (compiled with Erlang/OTP 25)

* Elixir Language Server version:

Started ElixirLS Debugger v0.17.5 ElixirLS Debugger built with elixir "1.14.4" on OTP "25" Running on elixir "1.14.4 (compiled with Erlang/OTP 25)" on OTP "25" Protocols are not consolidated Starting debugger in directory: /Users/frank/projects/hello Running with MIX_ENV: dev MIX_TARGET: host


* Operating system: `macOS 13.6.1 (22G313)`
* Editor or IDE name (e.g. Emacs/VSCode): `VSCode`
* Editor Plugin/LSP Client name and version: `ElixirLS: Elixir support and debugger v0.17.5`
* Phoenix: `v1.7.9 `

lukaszsamson commented 1 year ago

Did you try attaching observer to the node to see what is actually taking up the memory? You can set sname/name via ELS_ELIXIR_OPTS https://github.com/elixir-lsp/elixir-ls#environment-variables in env launch config option https://github.com/elixir-lsp/elixir-ls#debugger-configuration-options

fnicastri commented 1 year ago

@lukaszsamson I can see eheap_alloc going crazy in Memory allocator tab and cowboy_stream_h:request_process/3 in Processes

lukaszsamson commented 1 year ago

It's hard to tell what is the cause. It may be a drawback of OTP debugger. Excluding some modules from auto interpreting may help here.

fnicastri commented 1 year ago

@lukaszsamson

It's hard to tell what is the cause. It may be a drawback of OTP debugger. Excluding some modules from auto interpreting may help here.

TBH I don't know this side of Erlang/Elixir, so it is no easy for me to move around in this. If you can provide me a bit of guidance I could be able to explore more the issue and produce more info.

Anyhow here you can find the repo where I reproduced the issue.

fnicastri commented 1 year ago

@lukaszsamson

I've excluded almost every module (in a real Phoenix project) except from the code from the actual application and the issue persists.

Tried whit OTP26 as well, nothing changes.

In the demo project the situation is better but the queries are small (just a couple of records from a single table)

I think this is very impacting and make the debugger unusable, sadly I can't try on another platform, do you managed to reproduce the behavior?

Any thoughts?

fnicastri commented 1 year ago

@lukaszsamson can I at least know if there is no interest in this? I can understand if this is the case, but would be nice to know...

lukaszsamson commented 1 year ago

@fnicastri There is interest. I'm not closing the issue but currently I'm involved in language server stability fixes

fnicastri commented 1 year ago

@lukaszsamson thanks for the clarification

lukaszsamson commented 1 year ago

@fnicastri I reproduced easily the problem with your sample app. The memory goes even up to 20GB after a few breakpoints. I'm not sure what is the cause. As you mentioned one of the processes that starts taking up insane amounts of memory is cowboy_stream_h but also ElixirLS.Debugger.Server. It seems the state is not the problem here as it holds only a few kB there. I suspect binaries - if I force GC it goes back to manageable 4GB. I will experiment with Jason settings (the only place I know when one can control how the binaries are handled).

I've excluded almost every module (in a real Phoenix project) except from the code from the actual application and the issue persists.

In the example you did not exclude anything. That will not work as this option needs full module names

"excludeModules": [
                "ecto",
                "cowboy"
]

Instead try this:

"debugAutoInterpretAllModules": false,
"debugInterpretModulesPatterns": ["Hello*"],

This will disable interpreting of everything but modules matching the glob. In case of the example app this completely resolves the issue and debugger stays at ~90MB.

lukaszsamson commented 1 year ago

It's not binaries but heap process heap as you wrote. when running with "ELS_ERL_OPTS": "+hmax 101483800 +hmaxk false"

[error]      Process:            #PID<0.1679.0> on node :"mynode@MBP"
     Context:            maximum heap size reached
     Max Heap Size:      101483800
     Total Heap Size:    1115915790
     Kill:               false
     Error Logger:       true
     Message Queue Len:  0
     GC Info:            [
  old_heap_block_size: 850246556,
  heap_block_size: 395480352,
  mbuf_size: 0,
  recent_size: 65433676,
  stack_size: 423,
  old_heap_size: 406186569,
  heap_size: 67929058,
  bin_vheap_size: 474246280,
  bin_vheap_block_size: 33514735,
  bin_old_vheap_size: 77068941,
  bin_old_vheap_block_size: 165709447
]

I tried setting ERL_FULLSWEEP_AFTER to some low value but this does not make any difference. Maybe some allocator tuning would help https://www.erlang.org/doc/man/erts_alloc.html Anyway, I'm afraid it's some strange interaction between cowboy and OTP debugger that we can't do anything about besides disabling interpreting. Please let me know if debugAutoInterpretAllModules and debugInterpretModulesPatterns resolve the issue in your project.

fnicastri commented 1 year ago

@lukaszsamson

In the example you did not exclude anything. That will not work as this option needs full module names
"excludeModules": [
                "ecto",
                "cowboy"
]

Sorry, I never pushed that but I excluded them

fnicastri commented 1 year ago

@lukaszsamson

Anyway, I'm afraid it's some strange interaction between cowboy and OTP debugger that we > can't do anything about besides disabling interpreting. Please let me know if > debugAutoInterpretAllModules and debugInterpretModulesPatterns resolve the issue in your > project.

Tried on a real project, no changes... :(

lukaszsamson commented 1 year ago

Sorry, I never pushed that but I excluded them

Just to be clear in Debugger dumps to console all interpreted modules

Interpreting module Hello.Controller.Some

I wonder which process is taking up memory in your project when you disable interpretting cowboy.

Some other ideas to try:

do you use phoenix reloader? Debugger is known to be incompatible with it
try to interpret only the single module you need to debug
disable interpreting altogether and put dbg in your code when you would want a breakpoint - the debugger will stop the process using elixir Kernel.dbg backend. Note that dbg breakpoints are pretty limited in features - you can examine variables, eval and step through pipes. Step in/out, function breakpoints, logpoints, hit conditions, breakpoint setting and unsetting from UI, breaking in erlang code is not supported

lukaszsamson commented 8 months ago

Closing as this is not addressable in this project. Please reopen an issue to either cowboy or OTP

lukaszsamson commented 6 months ago

@fnicastri can you share your launch.json that I could use with an empty phoenix app? Last time I tried the workarounds from https://github.com/elixir-lsp/elixir-ls/issues/1017#issuecomment-1806933298 fixed the issue. Other people also reported that it works in their case

fnicastri commented 6 months ago

Doing some tests with current releases...

fnicastri commented 6 months ago

Things are better now but still, if you add some db calls or try to debug the Web app, memory goes up.

I've tried many scenarios, some are okayish, in some others it's just a huge spike (3GB+ or much more) but then memory goes down to around a GB (not always), others are totally unusable (10GB+ or till saturation).

repo main is clean with_data has just a model and a view

LiveView is much worse than a plain view

launch.js

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "mix_task",
            "name": "mix (Default task)",
            "request": "launch",
            "task": "phx.server",
            "projectDir": "${workspaceRoot}",
            "env": {},
            "exitAfterTaskReturns": false,
            "debugAutoInterpretAllModules": false,
            "debugInterpretModulesPatterns": [
                "DebuggerTestWeb.*"
            ],
        }
    ]
}

fnicastri commented 6 months ago

With the Erlang debugger the memory always remains around 100 MB with the same project

jsermeno commented 3 months ago

Some related discussion occurred in this issue as well: https://github.com/elixir-lsp/elixir-ls/issues/1101. Copying relevant message here:

Referring to that other issue though, I don't think it is Cowboy specific. I've created a minimal reproduction in this repo: https://github.com/jsermeno/elixir_ls_debug_failure. The only dependency I need to recreate this issue in a brand new project is {:stream_data, "~> 1.0", only: [:dev, :test]} and the following code:

  def gen_data() do
    StreamData.list_of(StreamData.fixed_map(%{
      "e" => StreamData.string(:utf8)
    }), min_length: 1, max_length: 1)
  end

  def gen_data_all() do
    gen all fixed_map <- gen_data() do
      fixed_map
    end
  end

  test "greets the world" do
    data = Enum.at(gen_data_all(), 0)
    assert ElixirLsFailure.hello() == :world
  end

I'll open an issue with StreamData as well. However, it seems it's likely not something specific this library since I don't believe Phoenix relies on this library.

jsermeno commented 3 months ago

Minimal reproduction : https://github.com/jsermeno/elixir_ls_debug_failure.

elixir-lsp / elixir-ls