mix deps.get consume 32GB of RAM when having livebook as dependency

ardhitama commented 2 years ago

Precheck

For proposing a new feature, please start a discussion on the Elixir Core mailing list: https://groups.google.com/group/elixir-lang-core
For bugs, do a quick search and make sure the bug has not yet been reported
Please disclose security vulnerabilities privately at elixir-security@googlegroups.com
Do not use the issues tracker for guidance, questions or support (try Elixir Forum, Stack Overflow, Slack, etc. instead)
Finally, be nice and have fun!

Environment

Elixir & Erlang/OTP versions (elixir --version):


Erlang/OTP 24 [erts-12.1.5] [source] [64-bit] [smp:24:24] [ds:24:24:10] [async-threads:1] [jit]

Elixir 1.13.0 (compiled with Erlang/OTP 24)

* Operating system:

WSL2

Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal


### Current behavior

Include code samples, errors and stacktraces if appropriate.
If reporting a bug, please include the reproducing steps.

`mix deps.get` for the following deps will cause beam.smp to consume 32GB of RAM and eventually will fail.

defp deps do [ {:phoenix, "~> 1.6"}, {:phoenix_html, "~> 3.0"}, {:phoenix_live_reload, "~> 1.2", only: :dev}, {:phoenix_live_view, "~> 0.17"}, {:floki, ">= 0.30.0", only: :test}, {:phoenix_live_dashboard, "~> 0.5"}, {:esbuild, ">= 0.0.0", runtime: Mix.env() == :dev}, {:swoosh, "~> 1.3"}, {:telemetry_metrics, "~> 0.6"}, {:telemetry_poller, "~> 1.0"}, {:gettext, "~> 0.18"}, {:jason, "~> 1.2"}, {:plug_cowboy, "~> 2.5"}, {:nimble_publisher, "~> 0.1"}, {:makeup_elixir, ">= 0.0.0"}, {:makeup_erlang, ">= 0.0.0"}, {:livebook, ">= 0.0.0"} ] end



![image](https://user-images.githubusercontent.com/966731/147340346-7404a7dc-56b6-42e8-a02e-19f3c8542c55.png)

### Expected behavior

A short description on how you expect the code to behave.

Should download the dependencies or give fail reason for an invalid confguration

josevalim commented 2 years ago

Run mix local.hex and please try again?

ardhitama commented 2 years ago

mix local.hex
Found existing entry: /home/ardhi/.mix/archives/hex-0.21.3
Are you sure you want to replace it with "https://repo.hex.pm/installs/1.13.0/hex-1.0.1.ez"? [Yn]
* creating /home/ardhi/.mix/archives/hex-1.0.1

still happening, one of the core goes to 100% and the RAM usage keep on increasing

josevalim commented 2 years ago

Can you please try the new resolver then? Still beta...

mix archive.install github hexpm/hex branch emj/new-solver

To revert back to stable run:

mix local.hex --force

Both the old and new solver will run by default and their results will be compared to check for discrepancies. Then you do:

HEX_OLD_SOLVER=0 mix deps.get

ardhitama commented 2 years ago

both solver:

mix deps.get                                               
Resolving Hex dependencies...
Old solver completed resolution in 0.093s
New solver completed resolution in 0.108s
Both old and new solver failed to resolve
New error message:

Because livebook >= 0.3.0 depends on telemetry_poller 0.5.1 and livebook < 0.3.0 depends on telemetry_poller ~> 0.4, livebook requires telemetry_poller ~> 0.4.
And because your app depends on livebook >= 0.0.0, telemetry_poller ~> 0.4 is required.
So, because your app depends on telemetry_poller ~> 1.0, version solving failed.

Old error message:

Failed to use "castore" (version 0.1.14) because
  esbuild (version 0.4.0) requires >= 0.0.0
  livebook (versions 0.3.0 to 0.4.1) requires 0.1.12
  mix.lock specifies 0.1.14

Failed to use "phoenix" (version 1.6.5) because
  livebook (versions 0.1.0 to 0.2.2) requires ~> 1.5.7
  phoenix_live_reload (version 1.3.3) requires ~> 1.4
  phoenix_live_view (version 0.17.5) requires ~> 1.5.9 or ~> 1.6.0
  mix.lock specifies 1.6.5

Failed to use "phoenix" (version 1.6.5) because
  livebook (version 0.2.3) requires 1.5.10
  phoenix_live_reload (version 1.3.3) requires ~> 1.4
  phoenix_live_view (version 0.17.5) requires ~> 1.5.9 or ~> 1.6.0
  mix.lock specifies 1.6.5

** (Mix) Hex dependency resolution failed, change the version requirements of your dependencies or unlock them (by using mix deps.update or mix deps.unlock). If you are unable to resolve the conflicts you can try overriding with {:dependency, "~> 1.0", override: true}

the beta solver:

 HEX_OLD_SOLVER=0 mix deps.get
Resolving Hex dependencies...
New solver completed resolution in 0.088s

Because livebook >= 0.3.0 depends on telemetry_poller 0.5.1 and livebook < 0.3.0 depends on telemetry_poller ~> 0.4, livebook requires telemetry_poller ~> 0.4.
And because your app depends on livebook >= 0.0.0, telemetry_poller ~> 0.4 is required.
So, because your app depends on telemetry_poller ~> 1.0, version solving failed.
** (FunctionClauseError) no function clause matching in Access.get/3    

    The following arguments were given to Access.get/3:

        # 1
        :ok

        # 2
        :phoenix

        # 3
        nil

    Attempted function clauses (showing 5 out of 5):

        def get(%module{} = container, key, default)
        def get(map, key, default) when is_map(map)
        def get(list, key, default) when is_list(list) and is_atom(key)
        def get(list, key, _default) when is_list(list)
        def get(nil, _key, default)

    (elixir 1.13.0) lib/access.ex:283: Access.get/3
    (hex 0.21.3) lib/hex/remote_converger.ex:408: Hex.RemoteConverger.deps/2
    (mix 1.13.0) lib/mix/dep/converger.ex:112: anonymous fn/4 in Mix.Dep.Converger.all/4
    (mix 1.13.0) lib/mix/dep/converger.ex:185: Mix.Dep.Converger.all/9
    (mix 1.13.0) lib/mix/dep/converger.ex:123: Mix.Dep.Converger.all/7
    (mix 1.13.0) lib/mix/dep/converger.ex:108: Mix.Dep.Converger.all/4
    (mix 1.13.0) lib/mix/dep/converger.ex:51: Mix.Dep.Converger.converge/4
    (mix 1.13.0) lib/mix/dep/fetcher.ex:16: Mix.Dep.Fetcher.all/3

josevalim commented 2 years ago

So there you go! You have incompatible versions in your mix.exs:

Because livebook >= 0.3.0 depends on telemetry_poller 0.5.1 and livebook < 0.3.0 depends on telemetry_poller ~> 0.4, livebook requires telemetry_poller ~> 0.4.
And because your app depends on livebook >= 0.0.0, telemetry_poller ~> 0.4 is required.
So, because your app depends on telemetry_poller ~> 1.0, version solving failed.

Solve this and you should be able to move forward. Keep in mind livebook is not really meant to be used as dependency (we offer no support for such). Have fun!

ardhitama commented 2 years ago

looking forward for the new solver then :)

elixir-lang / elixir

mix deps.get consume 32GB of RAM when having livebook as dependency #11525

Precheck

Environment