Release builds are broken under OTP 21.1 and 21.2.1

AlexanderKaraberov commented 5 years ago

Environment

Rebar3 report
 version 3.7.5
 generated at 2018-12-26T09:38:33+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: release
Entered as:
  release
-----------------
Operating System: x86_64-apple-darwin18.2.0
ERTS: Erlang/OTP 21 [erts-10.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]
Root Directory: /usr/local/Cellar/erlang/21.2.1/lib/erlang
Library directory: /usr/local/Cellar/erlang/21.2.1/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.6.0
certifi: 2.3.1
cf: 0.2.2
common_test: 1.16.1
compiler: 7.3
crypto: 4.4
cth_readable: 1.4.2
dialyzer: 3.3.1
edoc: 0.9.4
erlware_commons: 1.3.0
eunit: 2.3.7
eunit_formatters: 0.5.0
getopt: 1.0.1
hex_core: 0.2.0
hipe: 3.18.2
inets: 7.0.3
kernel: 6.2
providers: 1.7.0
public_key: 1.6.4
relx: 3.27.0
sasl: 3.3
snmp: 5.2.12
ssl_verify_fun: 1.1.3
stdlib: 3.7
syntax_tools: 2.1.6
tools: 3.0.2

-----------------
Escript path: /usr/local/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report repos shell state tar tree unlock update upgrade upgrade upgrade version xref

Current behaviour

After building release of the project under OTP 21.2.1 and running

DEBUG=1 rebar3 release

I see that the latest versions of kernel and stdlib are used:

===> release: dbstats_release-1.0.1
           erts-10.2.1, realized = true
     goals: 
          dbstats,
     applications: 
          {kernel,"6.2"},
          {stdlib,"3.7"},

But after I try to start an application via a script _build/<profile>/rel/<release name>/bin/<release name> output immediately explodes with the following error messages and I have to force stop a beam.smp process.

(no logger present) unexpected logger message: {log,error,"Error in process ~p with exit value:~n~p~n",[<0.0.0>,{function_clause,[{init,prepare_run_args,[{eval,[<<"io:format(\"~s\", [code:root_dir()]), halt().">>,<<"300">>]}],[]},{init,map,2,[]},{init,boot,1,[]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.0.0>,pid=><0.0.0>,time=>1545816063357251}}
(no logger present) unexpected logger message: {log,error,"Error in process ~p with exit value:~n~p~n",[<0.0.0>,{function_clause,[{init,prepare_run_args,[{eval,[<<"io:format(\"~s\", [code:root_dir()]), halt().">>,<<"300">>]}],[]},{init,map,2,[]},{init,boot,1,[]}]}],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.0.0>,pid=><0.0.0>,time=>1545816063357251}}

Launching an app with rebar3 shell apps works flawlessly.

I'm not sure whether this is directly related but I've checked a primLoad section in the generated boot script and I've noticed a deprecated error_logger specified there:

 {path,["$ERTS_LIB_DIR/kernel-6.2/ebin","$ERTS_LIB_DIR/stdlib-3.7/ebin"]},
     {primLoad,
         [error_handler,application,application_controller,application_master,
          code,code_server,erl_eval,erl_lint,erl_parse,error_logger,ets,file,

And OTP docs state explicitly:

error_logger is no longer started by default, but is automatically started when an event handler is added with error_logger:add_report_handler/1,2. The error_logger module is then also added as a handler to the new logger.

But again these are merely my speculations. I've also tried to rollback to OTP 21.1 and faced the same issue.

Expected behaviour

release builds created with rebar3 3.7.5 to work fine with OTP 21.x

ferd commented 5 years ago

https://gist.github.com/robertoaloi/1184144#gistcomment-48488 might be related to this. This happens in relx, and the error message is generated by the OTP release itself, not by either tool.

AlexanderKaraberov commented 5 years ago

@ferd Thank you for the hint! It helped me resolve the problem. Just in case I shall post details for posterity here. Feel free to close an issue.

Indeed there was a shell expansion problem in the generated extended script to start a node. Namely in the codepath which locates ERTS root directory if it's not included in the release bundle.

find_erts_dir() {
    __erts_dir="$RELEASE_ROOT_DIR/erts-$ERTS_VSN"
    if [ -d "$__erts_dir" ]; then
        ERTS_DIR="$__erts_dir";
        ROOTDIR="$RELEASE_ROOT_DIR"
    else
        __erl="$(which erl)"
        code="io:format(\"~s\", [code:root_dir()]), halt()."
        __erl_root="$("$__erl" -boot no_dot_erlang -sasl errlog_type error -noshell -eval "$code")"
        ERTS_DIR="$__erl_root/erts-$ERTS_VSN"
        ROOTDIR="$__erl_root"
    fi
}

Setting {extended_start_script, false} and {generate_start_script, false} and providing my own start script with the fix was a bad option for me. Therefore so that not to monkey patch a generated script I've decided to proceed with {include_erts, true} and always have ERTS in the RELEASE_ROOT_DIR to eschew these repercussions.

erlang / rebar3