set include_erts: false results in cannot expand $ERTS_LIB_DIR in bootfile

raarts commented 7 years ago

Steps to reproduce

Linux VM, fresh Alpine Linux with erlang+elixir+phoenix added.
Use this VM to create a release of my phoenix project
Create a docker container based off p711/erlang (which builds erlang the same way as my VM)
When I choose not to include erts, I get the following output:

init terminating in do_boot (cannot expand $ERTS_LIB_DIR in bootfile)^M^M
^M^M
Crash dump is being written to: erl_crash.dump...done^M^M
./backend.sh: line 341: {"init terminating in do_boot",'cannot expand $ERTS_LIB_DIR in bootfile'}^M/bin/erl: not found^M
./backend.sh: line 918: {"init terminating in do_boot",'cannot expand $ERTS_LIB_DIR in bootfile'}^M/erts-/bin/erlexec: not found^M

This does not happen when I choose to include the runtime, and then my project runs normally.

Looking at the output it seems to be looking for /bin/erl on my system that's in /usr/bin because I compiled erlang with --prefix=/usr

Do I need to specify somewhere where erlang resides?

The bootfile is a binary file, but indeed seems to contain $ERTS_LIB_DIR.

Reading the docs I thought REPLACE_OS_VARS=true would help, but it made no difference

Verbose Logs

alpine35:~/backend# MIX_ENV=prod mix release --env=prod --verbose
==> Loading configuration..
==> Assembling release..
==> Building release backend:0.0.1 using environment prod
IMPORTANT: You have opted to *not* include the Erlang runtime system (ERTS).
You must ensure that the version of Erlang this release is built with matches
the version the release will be run with once deployed. It will fail to run otherwise.
==> One or more direct or transitive dependencies are missing from
    :applications or :included_applications, they will not be included
    in the release:

    :elixir_make

    This can cause your application to fail at runtime. If you are sure
    that this is not an issue, you may ignore this warning.

==> Discovered applications:
  poolboy-1.5.1
    from: _build/prod/lib/poolboy
    applications:
      :kernel
      :stdlib
    includes: none

  ecto-2.1.3
    from: _build/prod/lib/ecto
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :decimal
      :poolboy
    includes: none

  phoenix_ecto-3.2.2
    from: _build/prod/lib/phoenix_ecto
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :ecto
      :plug
    includes: none

  xain-0.6.0
    from: _build/prod/lib/xain
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
    includes: none

  distillery-1.3.0
    from: _build/prod/lib/distillery
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  decimal-1.3.1
    from: _build/prod/lib/decimal
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  connection-1.0.4
    from: _build/prod/lib/connection
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  db_connection-1.1.1
    from: _build/prod/lib/db_connection
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :connection
    includes: none

  postgrex-0.13.1
    from: _build/prod/lib/postgrex
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :db_connection
      :decimal
      :crypto
    includes: none

  eex-1.4.0
    from: /usr/local/lib/elixir/bin/../lib/eex
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  phoenix-1.2.1
    from: _build/prod/lib/phoenix
    applications:
      :kernel
      :stdlib
      :elixir
      :plug
      :poison
      :logger
      :eex
    includes: none

  ueberauth_facebook-0.6.0
    from: _build/prod/lib/ueberauth_facebook
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :oauth2
      :ueberauth
    includes: none

  ueberauth_google-0.5.0
    from: _build/prod/lib/ueberauth_google
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :oauth2
      :ueberauth
    includes: none

  oauth2-0.8.3
    from: _build/prod/lib/oauth2
    applications:
      :kernel
      :stdlib
      :elixir
      :hackney
    includes: none

  ueberauth_github-0.4.1
    from: _build/prod/lib/ueberauth_github
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :ueberauth
      :oauth2
    includes: none

  ueberauth-0.4.0
    from: _build/prod/lib/ueberauth
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
    includes: none

  ueberauth_identity-0.2.3
    from: _build/prod/lib/ueberauth_identity
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :ueberauth
    includes: none

  poison-2.2.0
    from: _build/prod/lib/poison
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  bamboo-0.8.0
    from: _build/prod/lib/bamboo
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :hackney
      :poison
    includes: none

  mime-1.1.0
    from: _build/prod/lib/mime
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  plug-1.3.0
    from: _build/prod/lib/plug
    applications:
      :kernel
      :stdlib
      :elixir
      :crypto
      :logger
      :mime
    includes: none

  phoenix_html-2.9.3
    from: _build/prod/lib/phoenix_html
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :plug
    includes: none

  cowlib-1.0.2
    from: _build/prod/lib/cowlib
    applications:
      :kernel
      :stdlib
      :crypto
    includes: none

  ranch-1.3.2
    from: _build/prod/lib/ranch
    applications:
      :kernel
      :stdlib
      :ssl
    includes: none

  cowboy-1.1.2
    from: _build/prod/lib/cowboy
    applications:
      :kernel
      :stdlib
      :ranch
      :cowlib
      :crypto
    includes: none

  phoenix_pubsub-1.0.1
    from: _build/prod/lib/phoenix_pubsub
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :crypto
    includes: none

  combine-0.9.6
    from: _build/prod/lib/combine
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  metrics-1.0.1
    from: _build/prod/lib/metrics
    applications:
      :kernel
      :stdlib
    includes: none

  ssl_verify_fun-1.1.1
    from: _build/prod/lib/ssl_verify_fun
    applications:
      :kernel
      :stdlib
      :ssl
    includes: none

  certifi-1.0.0
    from: _build/prod/lib/certifi
    applications:
      :kernel
      :stdlib
    includes: none

  mimerl-1.0.2
    from: _build/prod/lib/mimerl
    applications:
      :kernel
      :stdlib
    includes: none

  idna-4.0.0
    from: _build/prod/lib/idna
    applications:
      :kernel
      :stdlib
    includes: none

  ssl-8.0.2
    from: /usr/lib/erlang/lib/ssl-8.0.2
    applications:
      :crypto
      :public_key
      :kernel
      :stdlib
    includes: none

  public_key-1.2
    from: /usr/lib/erlang/lib/public_key-1.2
    applications:
      :asn1
      :crypto
      :kernel
      :stdlib
    includes: none

  asn1-4.0.4
    from: /usr/lib/erlang/lib/asn1-4.0.4
    applications:
      :kernel
      :stdlib
    includes: none

  hackney-1.6.6
    from: _build/prod/lib/hackney
    applications:
      :kernel
      :stdlib
      :crypto
      :asn1
      :public_key
      :ssl
      :idna
      :mimerl
      :certifi
      :ssl_verify_fun
      :metrics
    includes: none

  tzdata-0.5.11
    from: _build/prod/lib/tzdata
    applications:
      :kernel
      :stdlib
      :elixir
      :hackney
      :logger
    includes: none

  timex-3.1.13
    from: _build/prod/lib/timex
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :tzdata
      :gettext
      :combine
    includes: none

  crypto-3.7.1
    from: /usr/lib/erlang/lib/crypto-3.7.1
    applications:
      :kernel
      :stdlib
    includes: none

  comeonin-3.0.1
    from: _build/prod/lib/comeonin
    applications:
      :kernel
      :stdlib
      :elixir
      :crypto
      :logger
    includes: none

  gettext-0.13.1
    from: _build/prod/lib/gettext
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
    includes: none

  logger-1.4.0
    from: /usr/local/lib/elixir/bin/../lib/logger
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  compiler-7.0.2
    from: /usr/lib/erlang/lib/compiler-7.0.2
    applications:
      :kernel
      :stdlib
    includes: none

  elixir-1.4.0
    from: /usr/local/lib/elixir/bin/../lib/elixir
    applications:
      :kernel
      :stdlib
      :compiler
    includes: none

  stdlib-3.1
    from: /usr/lib/erlang/lib/stdlib-3.1
    applications:
      :kernel
    includes: none

  kernel-5.1
    from: /usr/lib/erlang/lib/kernel-5.1
    applications: none
    includes: none

  backend-0.0.1
    from: _build/prod/lib/backend
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :gettext
      :comeonin
      :timex
      :phoenix_pubsub
      :cowboy
      :phoenix_html
      :bamboo
      :ueberauth_identity
      :ueberauth_github
      :ueberauth_google
      :ueberauth_facebook
      :phoenix
      :postgrex
      :distillery
      :xain
      :phoenix_ecto
    includes: none

  iex-1.4.0
    from: /usr/local/lib/elixir/bin/../lib/iex
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  sasl-3.0.1
    from: /usr/lib/erlang/lib/sasl-3.0.1
    applications:
      :kernel
      :stdlib
    includes: none

  runtime_tools-1.10.1
    from: /usr/lib/erlang/lib/runtime_tools-1.10.1
    applications:
      :kernel
      :stdlib
    includes: none

==> Generated overlay vars:
    release_name=:backend
    release_version="0.0.1"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=false
    include_erts=false
    include_src=false
    include_system_libs=true
    erl_opts=""
    erts_vsn="8.1"
    output_dir="_build/prod/rel/backend"
==> Copying applications to _build/prod/rel/backend
==> Generating nodetool
==> Generating start_erl.data
==> Generating vm.args
==> Generating sys.config from config/config.exs
==> Generating boot script
==> Generating RELEASES
==> Generating start_clean.boot
==> Applying overlays
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks/pre_configure.d
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks/pre_start.d
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks/post_start.d
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks/pre_stop.d
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks/post_stop.d
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks/pre_upgrade.d
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks/post_upgrade.d
==> Applying mkdir overlay
    dst: releases/0.0.1/commands
==> Packaging release..
==> Archiving backend-0.0.1
==> Writing tarball to _build/prod/rel/backend/releases/0.0.1/backend.tar.gz
==> Updating tarball
==> Tarball updated!
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: _build/prod/rel/backend/bin/backend console
      Foreground: _build/prod/rel/backend/bin/backend foreground
      Daemon: _build/prod/rel/backend/bin/backend start

Description of issue

What are the expected results?

that it uses the erlang system in the container and runs.

What OS, Erlang/Elixir versions are you seeing this issue on?

Alpine Linux, erlang 19.3, elixir 1.4.0

If possible, also provide your rel/config.exs, as it is often my first troubleshooting question, and you'll save us both time :)

# Import all plugins from `rel/plugins`
# They can then be used by adding `plugin MyPlugin` to
# either an environment, or release definition, where
# `MyPlugin` is the name of the plugin module.
Path.join(["rel", "plugins", "*.exs"])
|> Path.wildcard()
|> Enum.map(&Code.eval_file(&1))

use Mix.Releases.Config,
    # This sets the default release built by `mix release`
    default_release: :default,
    # This sets the default environment used by `mix release`
    default_environment: Mix.env()

# For a full list of config options for both releases
# and environments, visit https://hexdocs.pm/distillery/configuration.html

# You may define one or more environments in this file,
# an environment's settings will override those of a release
# when building in that environment, this combination of release
# and environment configuration is called a profile

environment :dev do
  set dev_mode: true
  set include_erts: false
  set cookie: :"***************"
end

environment :prod do
  set include_erts: false
  set include_src: false
  set cookie: :"***************"
end

# You may define one or more releases in this file.
# If you have not set a default release, or selected one
# when running `mix release`, the first release in the file
# will be used by default

release :backend do
  set version: current_version(:backend)
  set applications: [
    :runtime_tools
  ]
end

Is there documentation that says one thing, but Distillery does another? If so, please link the doc here so it can be updated if it's a documentation issue, or so that the fix can be based around what's documented.

bitwalker commented 7 years ago

What do you see when you run bin/backend describe?

raarts commented 7 years ago

docker run -it -e PORT=80 -e REPLACE_OS_VARS=true p711/backend describe
init terminating in do_boot (cannot expand $ERTS_LIB_DIR in bootfile)

Crash dump is being written to: erl_crash.dump...done
/bin/erl: not found341: {"init terminating in do_boot",'cannot expand $ERTS_LIB_DIR in bootfile'}
backend-0.0.1\n
erts:
path:       /app/releases/0.0.1
sys.config: /app/var/sys.config
vm.args:    /app/var/vm.args
name:       backend@127.0.0.1
cookie:     **********************************
erl_opts:   none provided

hooks:
No custom hooks found.

commands:
No custom commands found.

bitwalker commented 7 years ago

What is the result of running which erl on that system?

raarts commented 7 years ago

/app/releases/0.0.1 # which erl
/usr/bin/erl

Trevoke commented 7 years ago

I ran into a similar problem by setting include_erts to the path of the prod machine's erts-8.3 directory and trying to do a hot upgrade. I thought maybe it was all related, but now I think this might be part of the same root cause.

Trevoke commented 7 years ago

Update - let's mark my comment as temporarily invalid until I can reproduce it solidly.

pjanis commented 7 years ago

I ran into the same problem using a Ubuntu 16.04 server and an OS X development machine. The problem only occurs when running something like ./my_app console from inside bin/, bin/my_app console works just fine.

The problem seems to be that start_clean.boot from the development machine is included in bin/. There are at least a couple ways to work around this, simply deleting start_clean.boot works, as does removing the -boot start_clean option, export ROOTDIR="$("$__erl" -boot start_clean -eval 'io:format("~s~n", [code:root_dir()]), halt().' -noshell)" and "$ROOTDIR"/bin/url -boot start_clean -eval 'Ver = erlang:system_info(version), io:format("~s~n", [Ver]), halt()' -noshell in the my_app.sh scripts.

I'm not sure exactly why you'd want the start_clean.boot from the development system with include_erts set to false, so it might make sense to remove it from the tarball.

bitwalker commented 7 years ago

So I'm unable to reproduce this with the following steps:

Build release in bitwalker/alpine-erlang:latest, and copy it to host
Run p711/erlang (basically a copy of the above image, but tried to stay true to the test), and copy the release tarball into the container
Extract the tarball and run bin/test console (and bin/test foreground for good measure)

You can try to reproduce as well by cloning https://github.com/bitwalker/distillery-test, and running make deploy-alpine in the project root.

@pjanis Regarding the start_clean boot file: both start_clean.boot and <yourapp>.boot are part of the release tarball, so if one fails, both should fail, which should only happen if the ERTS version is different (which is expected, as .beam files are compiled for a specific version of ERTS, but more importantly, your app depends on specific versions of the system applications, and those versions will not be the same with a different version of ERTS). If you omit the -boot flag, it uses the default start_clean present in the Erlang installation, which is similar to the one produced by distillery, except it doesn't include :elixir and :iex applications.

I suspect the problem here is that people are deploying from one version of ERTS (say 8.1) to another (say 8.3).

pjanis commented 7 years ago

I don't have docker setup on my development machine, but I'll set it up later tonight.

In the mean time, I did mange to reproduce the problem on just the development machine. Below is the console history.


bash-3.2$ git clone https://github.com/bitwalker/distillery-test
Cloning into 'distillery-test'...
remote: Counting objects: 239, done.
remote: Total 239 (delta 0), reused 0 (delta 0), pack-reused 238
Receiving objects: 100% (239/239), 28.82 KiB | 0 bytes/s, done.
Resolving deltas: 100% (103/103), done.
bash-3.2$ cd distillery-test/
bash-3.2$ mix do deps.get, compile
Running dependency resolution...
Dependency resolution completed:
  certifi 1.1.0
  combine 0.9.6
  gettext 0.13.1
  hackney 1.8.0
  idna 4.0.0
  metrics 1.0.1
  mimerl 1.0.2
  neotoma 1.7.3
  poison 3.1.0
  ssl_verify_fun 1.1.1
  timex 3.1.13
  tzdata 0.5.12
* Getting timex (Hex package)
  Checking package (https://repo.hex.pm/tarballs/timex-3.1.13.tar)
  Fetched package
* Getting poison (Hex package)
  Checking package (https://repo.hex.pm/tarballs/poison-3.1.0.tar)
  Fetched package
* Getting combine (Hex package)
  Checking package (https://repo.hex.pm/tarballs/combine-0.9.6.tar)
  Fetched package
* Getting gettext (Hex package)
  Checking package (https://repo.hex.pm/tarballs/gettext-0.13.1.tar)
  Fetched package
* Getting tzdata (Hex package)
  Checking package (https://repo.hex.pm/tarballs/tzdata-0.5.12.tar)
  Fetched package
* Getting hackney (Hex package)
  Checking package (https://repo.hex.pm/tarballs/hackney-1.8.0.tar)
  Fetched package
* Getting certifi (Hex package)
  Checking package (https://repo.hex.pm/tarballs/certifi-1.1.0.tar)
  Fetched package
* Getting idna (Hex package)
  Checking package (https://repo.hex.pm/tarballs/idna-4.0.0.tar)
  Fetched package
* Getting metrics (Hex package)
  Checking package (https://repo.hex.pm/tarballs/metrics-1.0.1.tar)
  Fetched package
* Getting mimerl (Hex package)
  Checking package (https://repo.hex.pm/tarballs/mimerl-1.0.2.tar)
  Fetched package
* Getting ssl_verify_fun (Hex package)
  Checking package (https://repo.hex.pm/tarballs/ssl_verify_fun-1.1.1.tar)
  Fetched package
* Getting neotoma (Hex package)
  Checking package (https://repo.hex.pm/tarballs/neotoma-1.7.3.tar)
  Fetched package
===> Compiling idna
WARN:  Missing plugins: [rebar3_hex]
==> neotoma (compile)
Compiled src/neotoma.erl
Compiled src/neotoma_parse.erl
===> Compiling mimerl
===> Compiling metrics
==> gettext
Compiling 1 file (.yrl)
Compiling 1 file (.erl)
Compiling 20 files (.ex)
Generated gettext app
==> poison
Compiling 4 files (.ex)
Generated poison app
==> ssl_verify_fun (compile)
Compiled src/ssl_verify_util.erl
Compiled src/ssl_verify_fingerprint.erl
Compiled src/ssl_verify_pk.erl
Compiled src/ssl_verify_hostname.erl
==> combine
Compiling 6 files (.ex)
Generated combine app
===> Compiling certifi
===> Compiling hackney
==> tzdata
Compiling 16 files (.ex)
Generated tzdata app
==> timex
Compiling 57 files (.ex)
Generated timex app
Compiling src/conform_parse.peg
PEG manifest updated
==> conform
Compiling 1 file (.erl)
Compiling 19 files (.ex)
Generated conform app
==> distillery
Compiling 18 files (.ex)
Generated distillery app
==> test
Compiling 5 files (.ex)
Generated test app
bash-3.2$ mix release --env=prod_without_erts
==> Assembling release..
==> Building release test:0.1.0 using environment prod_without_erts
IMPORTANT: You have opted to *not* include the Erlang runtime system (ERTS).
You must ensure that the version of Erlang this release is built with matches
the version the release will be run with once deployed. It will fail to run otherwise.
==> Packaging release..
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: _build/dev/rel/test/bin/test console
      Foreground: _build/dev/rel/test/bin/test foreground
      Daemon: _build/dev/rel/test/bin/test start
bash-3.2$ _build/dev/rel/test/bin/test console
Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.4.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(test@127.0.0.1)1>
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
^Cbash-3.2$ cd _build/dev/rel/test/bin/
bash-3.2$ ./test console
init terminating in do_boot (cannot expand $ERTS_LIB_DIR in bootfile)

Crash dump is being written to: erl_crash.dump...done
/Users/pjanis/temp/distillery-test/_build/dev/rel/test/releases/0.1.0/test.sh: line 223: {"init terminating in do_boot",'cannot ex/bin/erl: No such file or directory
/Users/pjanis/temp/distillery-test/_build/dev/rel/test/releases/0.1.0/hooks/pre_configure.d/00_conform_pre_configure.sh: line 34:/erts-/bin/escript: No such file or directory $ERTS_LIB_DIR in boot file'}

bitwalker commented 7 years ago

Found the issue, can you give master a shot and confirm?

pjanis commented 7 years ago

confirmed. changes on master fix the issue.

bitwalker commented 7 years ago

👍 Thanks for the quick response!

bitwalker commented 7 years ago

1.3.5 has been published to Hex, just FYI

mpranjic commented 6 years ago

Hi, I'm experiencing the same issue with 1.5.1.

Removing start_clean.boot fixes.

Ch4s3 commented 6 years ago

I'm seeing this on 1.5.3.

bitwalker / distillery