Two releases on same host are conflicting

faithandbrave commented 7 years ago

Steps to reproduce

There are 2 servers:

application server
admin site server (for management application server)

Both servers uses Distillery.

When release application server by admin site server (execute deploy shell script), application server's sys_config and vm_args path be admin site path.

$ app_server/_build/prod/rel/app_server/bin/app_server describe
erts:        8.1
path:        /app_server/_build/prod/rel/app_server/releases/0.0.1-8820a10
sys.config:  /admin_site/_build/dev/rel/admin_site/var/sys.config
vm.args:     /admin_site/_build/dev/rel/admin_site/var/vm.args
name:        admin_site@127.0.0.1
cookie:      pvp{@]Nu4w;:p$@JkYr<r%XyjY~9?Bb9=q0t|y%+EDi0X!=B(U.D<uQ!V}D_3UkX
erl_opts:    none provided
run_erl_env: none provided

When start the application server, shutdown admin site.

Verbose Logs

[36m==> Loading configuration..[0m
[1m[36m==> Assembling release..[0m
[1m[36m==> Building release app_server:0.0.1-531aeeb using environment prod[0m
[36m==> Discovered applications:[0m
[36m  [0mlogger_file_backend-0.0.9[36m
    from: _build/prod/lib/logger_file_backend[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0muuid-1.1.5[36m
    from: _build/prod/lib/uuid[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0meredis-1.0.8[36m
    from: _build/prod/lib/eredis[0m
[36m    applications:
      :kernel
      :stdlib[0m
[36m    includes: none
[0m
[36m  [0mexredis-0.2.5[36m
    from: _build/prod/lib/exredis[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :eredis[0m
[36m    includes: none
[0m
[36m  [0mredis_poolex-0.0.6[36m
    from: _build/prod/lib/redis_poolex[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :logger[0m
[36m    includes: none
[0m
[36m  [0mex_json_schema-0.2.2[36m
    from: _build/prod/lib/ex_json_schema[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0mconnection-1.0.4[36m
    from: _build/prod/lib/connection[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0mdb_connection-1.0.0-rc.5[36m
    from: _build/prod/lib/db_connection[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :connection[0m
[36m    includes: none
[0m
[36m  [0mmariaex-0.7.8[36m
    from: _build/prod/lib/mariaex[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :crypto
      :decimal
      :db_connection[0m
[36m    includes: none
[0m
[36m  [0mpoolboy-1.5.1[36m
    from: _build/prod/lib/poolboy[0m
[36m    applications:
      :kernel
      :stdlib[0m
[36m    includes: none
[0m
[36m  [0mdecimal-1.2.0[36m
    from: _build/prod/lib/decimal[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0mecto-2.0.5[36m
    from: _build/prod/lib/ecto[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :decimal
      :poolboy[0m
[36m    includes: none
[0m
[36m  [0mphoenix_ecto-3.0.1[36m
    from: _build/prod/lib/phoenix_ecto[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :ecto
      :plug[0m
[36m    includes: none
[0m
[36m  [0mgettext-0.11.0[36m
    from: _build/prod/lib/gettext[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :logger[0m
[36m    includes: none
[0m
[36m  [0mcowlib-1.0.2[36m
    from: _build/prod/lib/cowlib[0m
[36m    applications:
      :kernel
      :stdlib
      :crypto[0m
[36m    includes: none
[0m
[36m  [0mranch-1.2.1[36m
    from: _build/prod/lib/ranch[0m
[36m    applications:
      :kernel
      :stdlib[0m
[36m    includes: none
[0m
[36m  [0mcowboy-1.0.4[36m
    from: _build/prod/lib/cowboy[0m
[36m    applications:
      :kernel
      :stdlib
      :ranch
      :cowlib
      :crypto[0m
[36m    includes: none
[0m
[36m  [0mphoenix_html-2.7.0[36m
    from: _build/prod/lib/phoenix_html[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :plug[0m
[36m    includes: none
[0m
[36m  [0mphoenix_pubsub-1.0.1[36m
    from: _build/prod/lib/phoenix_pubsub[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :crypto[0m
[36m    includes: none
[0m
[36m  [0meex-1.3.3[36m
    from: /usr/local/Cellar/elixir/1.3.3/bin/../lib/eex[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0mpoison-2.2.0[36m
    from: _build/prod/lib/poison[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0mmime-1.0.1[36m
    from: _build/prod/lib/mime[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0mlogger-1.3.3[36m
    from: /usr/local/Cellar/elixir/1.3.3/bin/../lib/logger[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0mcrypto-3.7.1[36m
    from: /usr/local/Cellar/erlang/19.1/lib/erlang/lib/crypto-3.7.1[0m
[36m    applications:
      :kernel
      :stdlib[0m
[36m    includes: none
[0m
[36m  [0mplug-1.2.1[36m
    from: _build/prod/lib/plug[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :crypto
      :logger
      :mime[0m
[36m    includes: none
[0m
[36m  [0mphoenix-1.2.1[36m
    from: _build/prod/lib/phoenix[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :plug
      :poison
      :logger
      :eex[0m
[36m    includes: none
[0m
[36m  [0mcompiler-7.0.2[36m
    from: /usr/local/Cellar/erlang/19.1/lib/erlang/lib/compiler-7.0.2[0m
[36m    applications:
      :kernel
      :stdlib[0m
[36m    includes: none
[0m
[36m  [0melixir-1.3.3[36m
    from: /usr/local/Cellar/elixir/1.3.3/bin/../lib/elixir[0m
[36m    applications:
      :kernel
      :stdlib
      :compiler[0m
[36m    includes: none
[0m
[36m  [0mstdlib-3.1[36m
    from: /usr/local/Cellar/erlang/19.1/lib/erlang/lib/stdlib-3.1[0m
[36m    applications:
      :kernel[0m
[36m    includes: none
[0m
[36m  [0mkernel-5.1[36m
    from: /usr/local/Cellar/erlang/19.1/lib/erlang/lib/kernel-5.1[0m
[36m    applications: none[0m
[36m    includes: none
[0m
[36m  [0mapp_server-0.0.1-531aeeb[36m
    from: _build/prod/lib/app_server[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir
      :phoenix
      :phoenix_pubsub
      :phoenix_html
      :cowboy
      :logger
      :gettext
      :phoenix_ecto
      :mariaex
      :ex_json_schema
      :redis_poolex
      :exredis
      :uuid
      :logger_file_backend[0m
[36m    includes: none
[0m
[36m  [0miex-1.3.3[36m
    from: /usr/local/Cellar/elixir/1.3.3/bin/../lib/iex[0m
[36m    applications:
      :kernel
      :stdlib
      :elixir[0m
[36m    includes: none
[0m
[36m  [0msasl-3.0.1[36m
    from: /usr/local/Cellar/erlang/19.1/lib/erlang/lib/sasl-3.0.1[0m
[36m    applications:
      :kernel
      :stdlib[0m
[36m    includes: none
[0m
[36m==> Generated overlay vars:[0m
[36m    release_name=:app_server
    release_version="0.0.1-531aeeb"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=false
    include_erts=true
    include_src=false
    include_system_libs=true
    erl_opts=""
    run_erl_env=""
    erts_vsn="8.1"
    output_dir="_build/prod/rel/app_server"[0m
[36m==> Copying applications to _build/prod/rel/app_server[0m
[36m==> Generating nodetool[0m
[36m==> Generating start_erl.data[0m
[36m==> Generating vm.args[0m
[36m==> Generating sys.config from config/config.exs[0m
[1m[36m==> Including ERTS 8.1 from /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1[0m
[36m==> Generating boot script[0m
[36m==> Generating RELEASES[0m
[36m==> Generating start_clean.boot[0m
[36m==> Applying overlays[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/hooks[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/hooks/pre_configure.d[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/hooks/pre_start.d[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/hooks/post_start.d[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/hooks/pre_stop.d[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/hooks/post_stop.d[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/hooks/pre_upgrade.d[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/hooks/post_upgrade.d[0m
[36m==> Applying [0mmkdir[36m overlay
    dst: releases/0.0.1-531aeeb/commands[0m
[36m==> Applying [0mcopy[36m overlay
    src: priv
    dst: priv[0m
[1m[36m==> Packaging release..[0m
[36m==> Archiving app_server-0.0.1-531aeeb[0m
[36m==> Writing tarball to _build/prod/rel/app_server/releases/0.0.1-531aeeb/app_server.tar.gz[0m
[36m==> Updating tarball[0m
[36m==> Tarball updated![0m
[1m[32m==> Release successfully built!

Description of issue

What are the expected results?
- A. admin site be alive.
What OS, Erlang/Elixir versions are you seeing this issue on?
- Erlang/OTP 19 [erts-8.1]
- Elixir 1.3.3
- Distillery 1.3.4
If possible, also provide your rel/config.exs, as it is often my first troubleshooting question, and you'll save us both time :)

admin site rel/config.exs:

use Mix.Releases.Config,
    # This sets the default release built by `mix release`
    default_release: :default,
    # This sets the default environment used by `mix release`
    default_environment: :dev

# For a full list of config options for both releases
# and environments, visit https://hexdocs.pm/distillery/configuration.html

# You may define one or more environments in this file,
# an environment's settings will override those of a release
# when building in that environment, this combination of release
# and environment configuration is called a profile

environment :dev do
  set dev_mode: true
  set include_erts: false
  set cookie: :"pvp{@]Nu4w;:p$@JkYr<r%XyjY~9?Bb9=q0t|y%+EDi0X!=B(U.D<uQ!V}D_3UkX"
end

environment :prod do
  set include_erts: true
  set include_src: false
  set cookie: :"pvp{@]Nu4w;:p$@JkYr<r%XyjY~9?Bb9=q0t|y%+EDi0X!=B(U.D<uQ!V}D_3UkX"
end

# You may define one or more releases in this file.
# If you have not set a default release, or selected one
# when running `mix release`, the first release in the file
# will be used by default

release :admin_site do
  set version: current_version(:admin_site)
end

app server rel/config.exs:

use Mix.Releases.Config,
    # This sets the default release built by `mix release`
    default_release: :default,
    # This sets the default environment used by `mix release`
    default_environment: :dev

# For a full list of config options for both releases
# and environments, visit https://hexdocs.pm/distillery/configuration.html

# You may define one or more environments in this file,
# an environment's settings will override those of a release
# when building in that environment, this combination of release
# and environment configuration is called a profile

environment :dev do
  set dev_mode: true
  set include_erts: false
  set cookie: :"fF7YQ;ZC2RO)4YVc5W7r5N$ve5CJZH$_:nq.^Mr}xm$0`,WDMgY22QPGOBAHd4(M"
end

environment :prod do
  set include_erts: true
  set include_src: false
  set cookie: :"fF7YQ;ZC2RO)4YVc5W7r5N$ve5CJZH$_:nq.^Mr}xm$0`,WDMgY22QPGOBAHd4(M"
end

# You may define one or more releases in this file.
# If you have not set a default release, or selected one
# when running `mix release`, the first release in the file
# will be used by default

release :app_server do
  set version: current_version(:app_server)
  set overlays: [
    {:copy, "priv", "priv"}
  ]
end

Is there documentation that says one thing, but Distillery does another? If so, please link the doc here so it can be updated if it's a documentation issue, or so that the fix can be based around what's documented.
- A. No. I tried specify sys_config and vm_args paths to app_server/rel/config.exs. However, no effect.
- https://hexdocs.pm/distillery/configuration.html#environment-release-settings

bitwalker commented 7 years ago

Have you set RELEASE_CONFIG_DIR, RELEASE_MUTABLE_DIR, VM_ARGS_PATH or SYS_CONFIG_PATH in the environment of the host you are deploying to? If these are set globally, it will not be possible to run two releases on the same host, those env vars must be set in the context of the release itself only.

faithandbrave commented 7 years ago

No changed.

bitwalker commented 7 years ago

@faithandbrave I don't understand, are you saying you unset these globally and there was no change to the issue; or are you saying you didn't change them in the first place? Can you describe in more detail how you are deploying and running the releases?

faithandbrave commented 7 years ago

I don't set environment variables. I'll provide sample project.

faithandbrave commented 7 years ago

I created sample project. Please see:

https://github.com/faithandbrave/distillery_issue254_sample

The project is faithandbrave/elixir-phoenix-chunked-response-example based.

bitwalker commented 7 years ago

@faithandbrave It looks like your first problem is that you are running the app from the same directory the release is built in - you should not be doing that, you need to deploy the tarball (releases/<version>/<app>.tar.gz) to some directory, and run your app from there. Distillery can't guarantee things will work as expected when running from the release build directory because things can change underneath the app. It is actually explicitly called out in the documentation that you cannot perform hot upgrades in that directory.

Secondly, you aren't checking the exit code of the stop command you are issuing. If something is failing there, then the subsequent start command will naturally conflict with the release which is already running.

Lastly, stopping the daemon may not immediately be noticed by epmd, and thus when the next version is started the name may still appear to be registered. One approach would be to run epmd -kill && epmd -daemon after stop to ensure that it's flushed, but I would recommend instead sleeping for a second (e.g. sleep 1) before issuing the start command to avoid the race here.

If you resolve all of the above and you are still able to reproduce this problem, feel free to reopen this issue, and I'll look further, but for now there doesn't appear to be an actionable issue with Distillery here yet.

faithandbrave commented 7 years ago

@bitwalker Thanks a lot! I'll redesign my application.

bitwalker / distillery