console_clean doesn't setup library search path correctly

[hlieberman]

Running a project with console_clean doesn't properly setup the directories such that you can load modules. For example, a minimal project with only a single direct dependency, say, on jiffy and an extended start script:

-module(foo_app).
-behaviour(application).

-export([start/2]).
-export([stop/1]).

start(_Type, _Args) ->
    io:format("Where is Jiffy? ~p~n", [code:lib_dir(jiffy)]),
    foo_sup:start_link().

stop(_State) ->
    ok.

You get the following executions:

console:

Exec: /tmp/foo/_rel/foo/erts-7.3.1.1/bin/erlexec -boot /tmp/foo/_rel/foo/releases/1/foo -mode embedded -boot_var ERTS_LIB_DIR /tmp/foo/_rel/foo/erts-7.3.1.1/../lib -config /tmp/foo/_rel/foo/releases/1/sys.config -args_file /tmp/foo/_rel/foo/releases/1/vm.args -- console
Root: /tmp/foo/_rel/foo
/tmp/foo/_rel/foo
Erlang/OTP 18 [erts-7.3.1.1] [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false]

Where is Jiffy? "/tmp/foo/_rel/foo/lib/jiffy-git"
Eshell V7.3.1.1  (abort with ^G)
(foo@agartha)1> 
User switch command
 --> q

console_clean:

Exec: /tmp/foo/_rel/foo/erts-7.3.1.1/bin/erlexec -boot /tmp/foo/_rel/foo/bin/start_clean -mode embedded -boot_var ERTS_LIB_DIR /tmp/foo/_rel/foo/erts-7.3.1.1/../lib -config /tmp/foo/_rel/foo/releases/1/sys.config -args_file /tmp/foo/_rel/foo/releases/1/vm.args -- console_clean
Root: /tmp/foo/_rel/foo
/tmp/foo/_rel/foo
Erlang/OTP 18 [erts-7.3.1.1] [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V7.3.1.1  (abort with ^G)
(foo@agartha)1> l(jiffy).
{error,nofile}
(foo@agartha)2> code:get_path().
["/tmp/foo/_rel/foo/lib/kernel-4.2/ebin",
 "/tmp/foo/_rel/foo/lib/stdlib-2.8/ebin"]
(foo@agartha)3> 
User switch command
 --> q

[bitwalker]

@tsloughter I found a way to address this, by running a quick eval which takes in the full .rel file, and generates a list of paths to the libraries for the current release, and appending them to the code path with -pa. This way console_clean loads with access to all code specific to the current release version, but with only kernel and stdlib loaded/started. Looks like this:

_get_code_paths() {
    erl -noshell -boot start_clean -eval "{ok, [{release,_,_,Apps}]} = file:consult(\"$REL_DIR/$REL_NAME.rel\"),lists:foreach(fun(A) -> io:fwrite(\"-pa $ROOTDIR/lib/~p-~s/ebin \", [element(1, A), element(2, A)]) end, Apps),halt()."
}

Then in the handler for console|console_clean|console_boot:

        __code_paths=$(_get_code_paths)
        set -- "$BINDIR/erlexec" \
            ...snip..
            ${__code_paths} \
            ...snip..

Did some testing and it looks good. Want me to open a PR?

[hlieberman]

On August 13, 2016 10:43:20 PM EDT, Paul Schoenfelder notifications@github.com wrote:

@tsloughter I found a way to address this, by running a quick eval which takes in the full .rel file, and generates a list of paths to the libraries for the current release, and appending them to the code path with -pa. This way console_clean loads with access to all code specific to the current release version, but with only kernel and stdlib loaded/started. Looks like this:

_get_code_paths() {
erl -noshell -boot start_clean -eval "{ok, [{release,_,_,Apps}]} =
file:consult(\"$REL_DIR/$REL_NAME.rel\"),lists:foreach(fun(A) ->
io:fwrite(\"-pa $ROOTDIR/lib/~p-~s/ebin \", [element(1, A), element(2,
A)]) end, Apps),halt()."
}

Then in the handler for console|console_clean|console_boot:

       __code_paths=$(_get_code_paths)
       set -- "$BINDIR/erlexec" \
           ...snip..
           ${__code_paths} \
           ...snip..

Did some testing and it looks good. Want me to open a PR?

I don't know if it helps, but this is technically a regression. relx did the right thing circa 2014 or so.

Harlan Lieberman-Berg ~hlieberman

[bitwalker]

Without looking through the commits I can't say for sure, but I would guess that previously a wildcard path was being added for the lib directory, but that has a problem in that it picks up all .beams in the lib directory, not just those for a specific version of a release. Not a problem if you aren't doing hot upgrades/downgrades, but still broken, and likely was removed later for that reason. The behavior today is not technically broken though: the systools script only adds code paths for applications which are specified to systools when generating the script, and since console_clean only loads kernel and stdlib, only the code paths for those two applications get added. It's obviously not very useful for debugging, which is the whole point of console_clean, since you have to then manually add code paths for applications you want to load, and this is what my proposed change addresses.

[tsloughter]

Can't we just have the start script set ERL_LIBS to $RELEASE_ROOT_DIR/lib before running erl?

[lrascao]

@hlieberman could you please try out #531 to see if it fixes this issue?

[lrascao]

closing, please re-open if issue still persists