Remote console can't connect when using -sname

andreausu commented 6 years ago

Hello, we've run into an issue after upgrading distillery from 1.5.2 to 2.0.9: remote_console is failing to connect to our application whereas before the upgrade it worked (no config settings were changed).

Steps to reproduce

Given the following vm.args:

## Name of the node
-sname crash@host-machine

## Cookie for distributed erlang
-setcookie ${erlang_cookie}

-kernel inet_dist_listen_min 30038 inet_dist_listen_max 30038

# Enable SMP automatically based on availability
-smp auto

First, I start the app, as you can see REPLACE_OS_VARS and erlang_cookie env variables are properly set:

app@0a9505fc8025:/app$ echo $REPLACE_OS_VARS
true
app@0a9505fc8025:/app$ echo $erlang_cookie
ewua7iequ1eadiudaecomeiThoht8turaefeedee2zoo5feeleZiem8sahj0Iech7Chu5gaeR5akohpha2aiheezeebuech4reiVoubeepoY5biecugei1so7yeiLu0Tai2geexahlaiphoh3Pos7r

app@0a9505fc8025:/app$ /app/bin/crash foreground

From docker exec you can see that the application was started with the proper cookie, however remote_console is failing to connect.

ps auxwww | grep crash
app     17840  0.5  2.6 3778228 207136 ?      Ssl  Aug27  16:35 /app/releases/0.0.0-dev/crash.sh -Bd -- -root /usr/local/lib/erlang -progname app/releases/0.0.0-dev/crash.sh -- -home /root -- -boot_var ERTS_LIB_DIR /usr/local/lib/erlang/lib -config /app/var/sys.config -pa /app/lib/absinthe-1.4.12/ebin /app/lib/absinthe_ecto-0.1.3/ebin /app/lib/absinthe_plug-1.4.4/ebin /app/lib/absinthe_relay-1.4.3/ebin /app/lib/amqp-1.0.3/ebin /app/lib/amqp_client-3.7.5/ebin /app/lib/artificery-0.2.6/ebin /app/lib/bamboo-1.0.0-rc.3/ebin /app/lib/bamboo_smtp-1.5.0-rc.4/ebin /app/lib/base64url-0.0.1/ebin /app/lib/basic_auth-2.2.2/ebin /app/lib/briefly-0.3.0/ebin /app/lib/cachex-3.0.2/ebin /app/lib/certifi-2.3.1/ebin /app/lib/combine-0.10.0/ebin /app/lib/connection-1.0.4/ebin /app/lib/cors_plug-1.5.2/ebin /app/lib/cowboy-1.0.4/ebin /app/lib/cowlib-1.0.2/ebin /app/lib/crash-0.0.0-dev/ebin /app/lib/crontab-1.1.3/ebin /app/lib/csv-2.1.1/ebin /app/lib/db_connection-1.1.3/ebin /app/lib/decimal-1.5.0/ebin /app/lib/distillery-2.0.8/ebin /app/lib/ecto-2.2.10/ebin /app/lib/ecto_enum-1.1.0/ebin /app/lib/eex-1.6.6/ebin /app/lib/elixir-1.6.6/ebin /app/lib/erlsom-1.2.1/ebin /app/lib/eternal-1.2.0/ebin /app/lib/ex_aws-2.0.2/ebin /app/lib/ex_aws_s3-2.0.0/ebin /app/lib/exjsx-3.2.1/ebin /app/lib/exq-0.11.0/ebin /app/lib/exq_ui-0.9.0/ebin /app/lib/fetcher-0.3.0/ebin /app/lib/fun_with_flags-1.0.0/ebin /app/lib/fun_with_flags_ui-0.4.1/ebin /app/lib/gen_smtp-0.12.0/ebin /app/lib/gen_stage-0.14.0/ebin /app/lib/gettext-0.15.0/ebin /app/lib/goldrush-0.1.9/ebin /app/lib/guardian-1.1.0/ebin /app/lib/hackney-1.12.1/ebin /app/lib/http_mock_pal-0.1.0/ebin /app/lib/httpoison-1.1.1/ebin /app/lib/idna-5.1.1/ebin /app/lib/iex-1.6.6/ebin /app/lib/jose-1.8.4/ebin /app/lib/jsx-2.8.2/ebin /app/lib/lager-3.5.1/ebin /app/lib/logger-1.6.6/ebin /app/lib/logger_logstash_backend-5.0.0/ebin /app/lib/mariaex-0.8.4/ebin /app/lib/metrics-1.0.1/ebin /app/lib/mime-1.3.0/ebin /app/lib/mimerl-1.0.2/ebin /app/lib/mix-1.6.6/ebin /app/lib/ok-1.11.0/ebin /app/lib/parallel_stream-1.0.6/ebin /app/lib/phoenix-1.3.3/ebin /app/lib/phoenix_ecto-3.3.0/ebin /app/lib/phoenix_html-2.11.2/ebin /app/lib/phoenix_pubsub-1.0.2/ebin /app/lib/phoenix_pubsub_redis-2.1.4/ebin /app/lib/plug-1.6.2/ebin /app/lib/poison-3.1.0/ebin /app/lib/poolboy-1.5.1/ebin /app/lib/posexional-0.4.2/ebin /app/lib/postgrex-0.13.5/ebin /app/lib/quantum-2.2.7/ebin /app/lib/rabbit_common-3.7.5/ebin /app/lib/ranch-1.5.0/ebin /app/lib/ranch_proxy_protocol-1.5.0/ebin /app/lib/recon-2.3.2/ebin /app/lib/redix-0.6.1/ebin /app/lib/redix_pubsub-0.4.2/ebin /app/lib/soap-1.0/ebin /app/lib/ssl_verify_fun-1.1.1/ebin /app/lib/sweet_xml-0.6.5/ebin /app/lib/timex-3.3.0/ebin /app/lib/tzdata-0.5.16/ebin /app/lib/unicode_util_compat-0.3.1/ebin /app/lib/uuid-1.1.8/ebin -pa /app/lib/crash-0.0.0-dev/consolidated -noshell -noshell -noinput -boot /app/releases/0.0.0-dev/crash -sname crash@host-machine -setcookie ewua7iequ1eadiudaecomeiThoht8turaefeedee2zoo5feeleZiem8sahj0Iech7Chu5gaeR5akohpha2aiheezeebuech4reiVoubeepoY5biecugei1so7yeiLu0Tai2geexahlaiphoh3Pos7r -kernel inet_dist_listen_min 30038 inet_dist_listen_max 30038 -smp auto -mode embedded -extra --

app@0a9505fc8025:/app$ echo $REPLACE_OS_VARS
true
app@0a9505fc8025:/app$ echo $erlang_cookie
ewua7iequ1eadiudaecomeiThoht8turaefeedee2zoo5feeleZiem8sahj0Iech7Chu5gaeR5akohpha2aiheezeebuech4reiVoubeepoY5biecugei1so7yeiLu0Tai2geexahlaiphoh3Pos7r

app@0a9505fc8025:/app$ ERL_EPMD_PORT=20038 /app/bin/crash remote_console
▸  Received 'pang' from crash@host-machine!
▸  Possible reasons for this include:
▸    - The cookie is mismatched between us and the target node
▸    - We cannot establish a remote connection to the node
Node crash@host-machine is not running!

EPMD is indeed reachable:

app@0a9505fc8025:/app$ telnet host-machine 20038
Trying 10.43.55.71...
Connected to host-machine.
Escape character is '^]'.

Verbose Logs

Paste the output of the release command you ran with the --verbose flag below in the summary tags (this helps keep the issue easy to navigate):

Description of issue

What are the expected results? remote_console should work as before the upgrade
What version of Distillery? The problem started right after upgrading distillery to 2.0.9 from 1.5.2.
What OS, Erlang/Elixir versions are you seeing this issue on? The docker image elixir:1.6.6 (same as the last working distillery version)
If possible, also provide your rel/config.exs, as it is often my first troubleshooting question, and you'll save us both time :)

use Mix.Releases.Config,
  # This sets the default release built by `mix release`
  default_release: :default,
  # This sets the default environment used by `mix release`
  default_environment: :dev

# For a full list of config options for both releases
# and environments, visit https://hexdocs.pm/distillery/configuration.html

# You may define one or more environments in this file,
# an environment's settings will override those of a release
# when building in that environment, this combination of release
# and environment configuration is called a profile

environment :dev do
  set(dev_mode: true)
  set(include_erts: false)

  set(
    commands: [
      migrate: "rel/commands/migrate.sh",
      rebuild: "rel/commands/rebuild_read_model.sh"
    ]
  )
end

environment :qa do
  set(include_erts: false)
  set(include_src: false)

  set(
    commands: [
      migrate: "rel/commands/migrate.sh",
      rebuild: "rel/commands/rebuild_read_model.sh"
    ]
  )
end

environment :staging do
  set(include_erts: false)
  set(include_src: false)

  set(
    commands: [
      migrate: "rel/commands/migrate.sh",
      rebuild: "rel/commands/rebuild_read_model.sh"
    ]
  )
end

environment :production do
  set(include_erts: false)
  set(include_src: false)

  set(
    commands: [
      migrate: "rel/commands/migrate.sh",
      rebuild: "rel/commands/rebuild_read_model.sh"
    ]
  )
end

# You may define one or more releases in this file.
# If you have not set a default release, or selected one
# when running `mix release`, the first release in the file
# will be used by default

release :crash do
  set(version: current_version(:crash))
  set(vm_args: "rel/vm.args")

  set(
    overlays: [
      {:mkdir, "config"},
      {:copy, "config/secrets.yml", "config/secrets.yml"}
    ]
  )
end

Is there documentation that says one thing, but Distillery does another? If so, please link the doc here so it can be updated if it's a documentation issue, or so that the fix can be based around what's documented.

We're using the method described here to set the cookie from an env variable: https://hexdocs.pm/distillery/config/vm.args.html and it was working before upgrading to 2.0.9.

Please let me know if you need anything else to debug this.

Thank you!

bitwalker commented 6 years ago

Looking into this - I haven't had trouble with this yet, but I wonder if it has anything to do with the interaction with ERL_EPMD_PORT

bitwalker commented 6 years ago

Looks like the problem is setting -sname using the format name@host - the release_ctl script sees the @ and assumes it is a fully-qualified name. This is technically a bug, since name@shorthost is a valid form for -sname, but you can workaround it by using just -sname name instead. Erlang will infer the short hostname in that case.

andreausu commented 6 years ago

@bitwalker thank you for the extremely quick turnaround on this! I can confirm that https://github.com/bitwalker/distillery/commit/c83332804bce88089f627d4753bfcfb2c31f2504 fixes the issue.

bitwalker / distillery