search

bitwalker / distillery

Simplify deployments in Elixir with OTP releases!
MIT License
2.96k stars 396 forks source link

:mix and :distillery are included in release applications #678

Closed ymtszw closed 5 years ago

ymtszw commented 5 years ago

Steps to reproduce

  1. (from bash) mix phx.new disttest
  2. cd disttest
  3. Add {:distillery, "~> 2.0", runtime: false} in mix.exs and deps.get
  4. Immediately mix release.init
  5. MIX_ENV=prod DATABASE_URL=foobar SECRET_KEY_BASE=foobar mix release
  6. cat _build/prod/rel/disttest/releases/0.1.0/disttest.rel
  7. You can see that mix, distillery, artificery included
  8. Whereas MIX_ENV=prod DATABASE_URL=foobar SECRET_KEY_BASE=foobar mix app.tree does not reveal any apps depending on mix at runtime

Verbose Logs

MIX_ENV=prod DATABASE_URL=hogehoge SECRET_KEY_BASE=foobar mix release --verbose 
==> Loading configuration..
==> Assembling release..
==> Building release disttest:0.1.0 using environment prod
==> Discovered applications:
  > phoenix_ecto-4.0.0
  |
  |  from: _build/prod/lib/phoenix_ecto
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :ecto
  |      :plug
  |  includes: none
  |_____

  > plug_crypto-1.0.0
  |
  |  from: _build/prod/lib/plug_crypto
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :crypto
  |  includes: none
  |_____

  > kernel-6.3.1
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/kernel-6.3.1
  |  applications: none
  |  includes: none
  |_____

  > asn1-5.0.8
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/asn1-5.0.8
  |  applications:
  |      :kernel
  |      :stdlib
  |  includes: none
  |_____

  > public_key-1.6.5
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/public_key-1.6.5
  |  applications:
  |      :asn1
  |      :crypto
  |      :kernel
  |      :stdlib
  |  includes: none
  |_____

  > ssl-9.2.2
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/ssl-9.2.2
  |  applications:
  |      :crypto
  |      :public_key
  |      :kernel
  |      :stdlib
  |  includes: none
  |_____

  > ranch-1.7.1
  |
  |  from: _build/prod/lib/ranch
  |  applications:
  |      :kernel
  |      :stdlib
  |      :ssl
  |  includes: none
  |_____

  > cowlib-2.7.3
  |
  |  from: _build/prod/lib/cowlib
  |  applications:
  |      :kernel
  |      :stdlib
  |      :crypto
  |  includes: none
  |_____

  > cowboy-2.6.3
  |
  |  from: _build/prod/lib/cowboy
  |  applications:
  |      :kernel
  |      :stdlib
  |      :crypto
  |      :cowlib
  |      :ranch
  |  includes: none
  |_____

  > plug_cowboy-2.0.2
  |
  |  from: _build/prod/lib/plug_cowboy
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :cowboy
  |      :plug
  |  includes: none
  |_____

  > artificery-0.4.2
  |
  |  from: _build/prod/lib/artificery
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |  includes: none
  |_____

  > distillery-2.0.14
  |
  |  from: _build/prod/lib/distillery
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :runtime_tools
  |      :artificery
  |  includes: none
  |_____

  > db_connection-2.1.0
  |
  |  from: _build/prod/lib/db_connection
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :connection
  |  includes: none
  |_____

  > decimal-1.7.0
  |
  |  from: _build/prod/lib/decimal
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |  includes: none
  |_____

  > connection-1.0.4
  |
  |  from: _build/prod/lib/connection
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |  includes: none
  |_____

  > postgrex-0.14.3
  |
  |  from: _build/prod/lib/postgrex
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :crypto
  |      :connection
  |      :decimal
  |      :db_connection
  |  includes: none
  |_____

  > phoenix-1.4.6
  |
  |  from: _build/prod/lib/phoenix
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :eex
  |      :crypto
  |      :phoenix_pubsub
  |      :plug
  |  includes: none
  |_____

  > mime-1.3.1
  |
  |  from: _build/prod/lib/mime
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |  includes: none
  |_____

  > plug-1.8.2
  |
  |  from: _build/prod/lib/plug
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :mime
  |      :plug_crypto
  |  includes: none
  |_____

  > runtime_tools-1.13.2
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/runtime_tools-1.13.2
  |  applications:
  |      :kernel
  |      :stdlib
  |  includes: none
  |_____

  > stdlib-3.8.1
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/stdlib-3.8.1
  |  applications:
  |      :kernel
  |  includes: none
  |_____

  > crypto-4.4.2
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/crypto-4.4.2
  |  applications:
  |      :kernel
  |      :stdlib
  |  includes: none
  |_____

  > phoenix_pubsub-1.1.2
  |
  |  from: _build/prod/lib/phoenix_pubsub
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :crypto
  |  includes: none
  |_____

  > ecto-3.1.5
  |
  |  from: _build/prod/lib/ecto
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :crypto
  |      :decimal
  |  includes: none
  |_____

  > eex-1.8.2
  |
  |  from: /mnt/c/Program Files/Elixir/bin/../lib/eex
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |  includes: none
  |_____

  > phoenix_html-2.13.3
  |
  |  from: _build/prod/lib/phoenix_html
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :eex
  |      :logger
  |      :plug
  |  includes: none
  |_____

  > compiler-7.3.2
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/compiler-7.3.2
  |  applications:
  |      :kernel
  |      :stdlib
  |  includes: none
  |_____

  > jason-1.1.2
  |
  |  from: _build/prod/lib/jason
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |  includes: none
  |_____

  > sasl-3.3
  |
  |  from: /home/ymtszw/.asdf/installs/erlang/21.3.7/lib/sasl-3.3
  |  applications:
  |      :kernel
  |      :stdlib
  |  includes: none
  |_____

  > telemetry-0.4.0
  |
  |  from: _build/prod/lib/telemetry
  |  applications:
  |      :kernel
  |      :stdlib
  |  includes: none
  |_____

  > gettext-0.16.1
  |
  |  from: _build/prod/lib/gettext
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |  includes: none
  |_____

  > ecto_sql-3.1.4
  |
  |  from: _build/prod/lib/ecto_sql
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :telemetry
  |      :db_connection
  |      :ecto
  |  includes: none
  |_____

  > logger-1.8.2
  |
  |  from: /mnt/c/Program Files/Elixir/bin/../lib/logger
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |  includes: none
  |_____

  > disttest-0.1.0
  |
  |  from: _build/prod/lib/disttest
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |      :logger
  |      :runtime_tools
  |      :gettext
  |      :jason
  |      :phoenix_pubsub
  |      :postgrex
  |      :ecto_sql
  |      :phoenix_html
  |      :plug_cowboy
  |      :phoenix
  |      :phoenix_ecto
  |  includes: none
  |_____

  > iex-1.8.2
  |
  |  from: /mnt/c/Program Files/Elixir/bin/../lib/iex
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |  includes: none
  |_____

  > mix-1.8.2
  |
  |  from: /mnt/c/Program Files/Elixir/bin/../lib/mix
  |  applications:
  |      :kernel
  |      :stdlib
  |      :elixir
  |  includes: none
  |_____

  > elixir-1.8.2
  |
  |  from: /mnt/c/Program Files/Elixir/bin/../lib/elixir
  |  applications:
  |      :kernel
  |      :stdlib
  |      :compiler
  |  includes: none
  |_____

==> Running validation checks..
    > Mix.Releases.Checks.Erts * PASS
    > Mix.Releases.Checks.Cookie * PASS
    > Mix.Releases.Checks.LoadedOrphanedApps * PASS
==> Generated overlay vars:
    release_name=:disttest
    release_version="0.1.0"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=false
    include_erts=true
    include_src=false
    include_system_libs=true
    erl_opts=""
    run_erl_env=""
    erts_vsn="10.3.4"
    output_dir="_build/prod/rel/disttest"
==> Copying applications to _build/prod/rel/disttest
==> Generating start_erl.data
==> Generating vm.args from rel/vm.args
==> Generating sys.config from config/config.exs
==> Including ERTS 10.3.4 from /home/ymtszw/.asdf/installs/erlang/21.3.7/erts-10.3.4
==> Generating boot scripts
==> Generating RELEASES
==> Applying overlays
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks/pre_configure.d
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks/post_configure.d
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks/pre_start.d
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks/post_start.d
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks/pre_stop.d
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks/post_stop.d
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks/pre_upgrade.d
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks/post_upgrade.d
==> Applying copy overlay
    src: _build/prod/lib/distillery/priv/libexec
    dst: releases/0.1.0/libexec
==> Applying mkdir overlay
    dst: releases/0.1.0/commands
==> Packaging release..
==> Archiving disttest-0.1.0
==> Writing archive to /mnt/d/Users/Romancer/workspace/siiibo-app/disttest/_build/prod/rel/disttest/releases/0.1.0/disttest.tar.gz
==> Updating archive..
==> Including system libs from current Erlang installation
==> Saving archive..
==> Archive saved!
Release successfully built!
To start the release you have built, you can use one of the following tasks:

    # start a shell, like 'iex -S mix'
    > _build/prod/rel/disttest/bin/disttest console

    # start in the foreground, like 'mix run --no-halt'
    > _build/prod/rel/disttest/bin/disttest foreground

    # start in the background, must be stopped with the 'stop' command
    > _build/prod/rel/disttest/bin/disttest start

If you started a release elsewhere, and wish to connect to it:

    # connects a local shell to the running node
    > _build/prod/rel/disttest/bin/disttest remote_console

    # connects directly to the running node's console
    > _build/prod/rel/disttest/bin/disttest attach

For a complete listing of commands and their use:

    > _build/prod/rel/disttest/bin/disttest help

Description of issue

(I am kind of familiar with release handling myself through in-house experiences in access-company/antikythera framework)

My understanding is that distillery should leave :mix and :distillery itself (:artificery included) behind on building world, not to be included in releases as runtime applications.

Looking at --verbose logs, those two applications are somehow "discovered." But cat _build/prod/lib/disttest/ebin/disttest.app says:

{application,disttest,
             [{applications,[kernel,stdlib,elixir,logger,runtime_tools,
                             gettext,jason,phoenix_pubsub,postgrex,ecto_sql,
                             phoenix_html,plug_cowboy,phoenix,phoenix_ecto]},
              {description,"disttest"},
              {modules,['Elixir.Disttest','Elixir.Disttest.Application',
                        'Elixir.Disttest.Repo','Elixir.DisttestWeb',
                        'Elixir.DisttestWeb.Endpoint',
                        'Elixir.DisttestWeb.ErrorHelpers',
                        'Elixir.DisttestWeb.ErrorView',
                        'Elixir.DisttestWeb.Gettext',
                        'Elixir.DisttestWeb.LayoutView',
                        'Elixir.DisttestWeb.PageController',
                        'Elixir.DisttestWeb.PageView',
                        'Elixir.DisttestWeb.Router',
                        'Elixir.DisttestWeb.Router.Helpers',
                        'Elixir.DisttestWeb.UserSocket']},
              {registered,[]},
              {vsn,"0.1.0"},
              {mod,{'Elixir.Disttest.Application',[]}}]}.

So mix, distillery are not traceable from disttest, meaning it is added by distillery for some reasons.

Now, I know that iex must be included for Elixir-friendly remote console. Although iex itself does not depend on mix at runtime.

My question is two-fold:

Thanks in advance! :heart:

ymtszw commented 5 years ago

Digging issues, found this one: #534

https://github.com/bitwalker/distillery/blob/2.0.9/lib/mix/lib/releases/models/release.ex#L15-L30

            applications: [
              # required for elixir apps
              :elixir,
              # included so the elixir shell works
              :iex,
              # required for Mix config provider
              :mix,
              # required for upgrades
              :sasl,
              # required for some command tooling
              :runtime_tools,
              # needed for config provider API
              :distillery
              # can also use `app_name: type`, as in `some_dep: load`,
              # to only load the application, not start it
            ],

So basically, to provide config provider feature, distillery currently depends on :mix, and set itself up as a runtime application in release.

Config provider feature is lifted to stdlib in Elixir 1.9.0, and it is reframed as standalone module so that it will not depend on Mix anymore.

Thus, this problem should naturally resolve when we migrate to mix release in Elixir 1.9.0, I believe.