bitwalker / distillery

Simplify deployments in Elixir with OTP releases!
MIT License
2.96k stars 397 forks source link

Can't run two remote tasks at the same time against a release node #623

Closed elbrujohalcon closed 5 years ago

elbrujohalcon commented 5 years ago

Steps to reproduce

The steps to reproduce this bug are:

  1. Build a release with distillery (let's say we built one for my_app).
  2. Start the release (e.g. _build/dev/rel/my_app/bin/my_app console_clean).
  3. On another terminal open an rpc that takes a while (e.g. build/my_app/bin/my_app rpc "Process.sleep(10000)).
  4. Try to do something else like that from a different terminal (e.g. build/my_app/bin/my_app ping).

At this point, you should see your ping crashing with a message like …

▸  Could not start distribution: {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [[:"my_app_maint_@127.0.0.1", :longnames], false]}, :permanent, 1000, :supervisor, [:erl_distribution]}}

Description of issue

Associated code

I was able to track down the issue to Mix.Releases.Runtime.Control.suffix_name/1,2, where a hardcoded name is assigned to maintenance nodes.

bitwalker commented 5 years ago

I've added support for passing a flag --id=<string with valid node name chars> to all of the release commands which take --name and --cookie; you can use this to uniquely identify tasks for multiple concurrent connections.

The reason I didn't autogenerate IDs is because node names are atoms, and when connecting to a node those atoms end up in both atom tables, after enough time of your tasks running, a node could blow the atom table limit, so it is better to decide on names per-task, so that those names are re-used each time that task runs.

elbrujohalcon commented 5 years ago

Cool. Thanks :)