Error evaluating Rebar config script with Mix Compile of Elixir

[guitarmind]

Tested Environments

- Ubuntu 14.04 LTS 64-bit (a VM of 2 vCPU and 4GB RAM)
- Alpine Linux 3.4 64-bit (a Docker container of 1 vCPU and 1GB RAM)

Rebar Version

rebar 2.6.1 R16B03 20160304_070244 git 2.6.1-24-ga998f52

C++ Compiler Version

c++ (Ubuntu 4.8.4-2ubuntu1~14.04.1) 4.8.4

ErlangVersion

Erlang/OTP 18 [erts-7.2]

Elixir Version

IEx 1.3.2

How to reproduce

The rebar.config.script in tuncer/re2 is fine with rebar compile. However, when doing the compilation through mix compile in Elixir will get the following errors:

Error evaluating Rebar config script ./rebar.config.script:73: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependencies defined in the script won't be available unless you add them to your Mix project

Detail Logs

$ mix compile
==> re2
Error evaluating Rebar config script ./rebar.config.script:73: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependencies defined in the script won't be available unless you add them to your Mix project
==> re2 (compile)
Compiling c_src/re2_nif.cpp
==> pooler (compile)
==> semver (compile)
==> lz4 (compile)
==> recon (compile)
==> timex
Compiling 57 files (.ex)
Generated timex app
==> quickrand (compile)
==> uuid (compile)
==> lru_cache
Compiling 1 file (.ex)
Generated lru_cache app
===> Rebar3 detected a lock file from a newer version. It will be loaded in compatibility mode, but important information may be missing or lost. It is recommended to upgrade Rebar3.
===> Rebar3 detected a lock file from a newer version. It will be loaded in compatibility mode, but important information may be missing or lost. It is recommended to upgrade Rebar3.
===> Rebar3 detected a lock file from a newer version. It will be loaded in compatibility mode, but important information may be missing or lost. It is recommended to upgrade Rebar3.
===> Compiling relx
==> exrm
Compiling 11 files (.ex)
warning: function beam_imports/1 is unused
  lib/exrm/appups.ex:98

warning: variable v2_file is unused
  lib/exrm/appups.ex:84

Generated exrm app
==> src (compile)
==> exometer (compile)
==> snappy (compile)
make: Entering directory `/home/markpeng/Gitlab/timeseries_service/deps/snappy/c_src'
make: `/home/markpeng/Gitlab/timeseries_service/deps/snappy/c_src/../priv/snappy_nif.so' is up to date.
make: Leaving directory `/home/markpeng/Gitlab/timeseries_service/deps/snappy/c_src'
==> cqerl (compile)
==> re2
Error evaluating Rebar config script ./rebar.config.script:73: evaluation failed with reason error:{badmatch,undefined} and stacktrace [{erl_eval,expr,3,[]}]
Any dependencies defined in the script won't be available unless you add them to your Mix project
==> timeseries_service
Compiling 35 files (.ex)
Generated timeseries_service app

The errors are all pointed to this line of code in rebar.config.script:

Rebar2FixedPortCompiler = Rebar2Vsn >= "2.6.4".

In tuncer/re2

%% Only look for libc++ if rebar is older than 2.6.4. If it's new
%% enough, we can omit the extra LDFLAGS, because CXX will be used in
%% the link command.
{ok, Rebar2Vsn} = application:get_key(rebar, vsn).
Rebar2FixedPortCompiler = Rebar2Vsn >= "2.6.4".
LDFLAGS = case Rebar2FixedPortCompiler of
              true ->
                  [];
              false ->
                  [{"(linux|freebsd|solaris)",
                    "LDFLAGS", "$LDFLAGS -l" ++ WhichCPPLib()}]
end.

In matehat/re2

LDFLAGS = {"(linux|freebsd|solaris)",
"LDFLAGS", "$LDFLAGS -l" ++ WhichCPPLib()}.

Using an older version from matehat/re2 fixed the issue. Looks like Rebar2Vsn is undedined?

[ghost]

Filed upstream bug: https://github.com/elixir-lang/elixir/issues/5645

[ghost]

Closing, since it's tracked as a mix bug.

[ghost]

Result of the discussion with mix devs: mix implements a not quite correct evaluator for rebar.config.script which unfortunately fails to provide the right environment for scripts but works for many scripts. mix's evaluator is used to extract deps before actually running rebar proper on the project.

If we can assume that when mix is used to evaluate the script it also means rebar3 will be used to build, then this will be simple to work around.

[ghost]

TODO: implement and test mix support kludge in rebar.config.script for 2.7.2

[guitarmind]

Hi @tuncer,

Sorry about the late reply and many thanks to your follow-ups and tracing of the issue with mix devs. I'm relatively new to Erlang/Elixir therefore I'm afraid that I might not have correct answer to the internal details of mix manager in Elixir.

However, jus saw the comment from https://github.com/elixir-lang/elixir/issues/5645 mentioned that:

Mix defaults to using rebar3 unless the hex package specifies it should be compiled with rebar2 or if the mix project declares it with manager: :rebar.

In my environment I have both rebar and rebar3 installed, therefore I can switch to always use rebar3 if that is desired by the workaround your are going to provide.

Thank you again for the help.

Regards, Mark

[ghost]

If we can assume rebar3, then the kludge will easy to implement, so no need to configure rebar version, since rebar3 is already the default. I'll let you know when I have something ready to test.

[ghost]

@guitarmind, can you try the mix-kludge branch? If you run CONFIG_DEBUG=1 mix compile, it should print the resulting config term before the script's evaluation finishes.

[ghost]

Evaluating the script with no environment (just a plain Erlang repl) works for me, meaning it will in mix as well. @guitarmind, did you test the branch?

[guitarmind]

@tuncer Thanks a lot for the efforts, I can confirm that mix compile works after updating to use mix-kludge branch. I have created markpeng/mix-kludge branch in my forked cqerl project (link), and from my Elixir application project now mix compile can be passed with no errors. Note that I was testing by rebar 2 and it seems to work!

Here is the output of mix compile:

==> re2 (compile)
Cloning into 're2'...
Fetching origin
Note: checking out '2017-01-01'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 09fc9ce... Make SimplifyWalker::SimplifyRepeat() use Regexp::Concat().
make: Entering directory `/home/markpeng/Gitlab/pegasus_timeseries_service/deps/re2/c_src/re2'
c++ -m64 -c -o obj/util/rune.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG util/rune.cc
c++ -m64 -c -o obj/util/strutil.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG util/strutil.cc
c++ -m64 -c -o obj/re2/bitstate.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/bitstate.cc
c++ -m64 -c -o obj/re2/compile.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/compile.cc
c++ -m64 -c -o obj/re2/dfa.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/dfa.cc
c++ -m64 -c -o obj/re2/filtered_re2.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/filtered_re2.cc
c++ -m64 -c -o obj/re2/mimics_pcre.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/mimics_pcre.cc
c++ -m64 -c -o obj/re2/nfa.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/nfa.cc
c++ -m64 -c -o obj/re2/onepass.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/onepass.cc
c++ -m64 -c -o obj/re2/parse.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/parse.cc
c++ -m64 -c -o obj/re2/perl_groups.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/perl_groups.cc
c++ -m64 -c -o obj/re2/prefilter.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/prefilter.cc
c++ -m64 -c -o obj/re2/prefilter_tree.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/prefilter_tree.cc
c++ -m64 -c -o obj/re2/prog.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/prog.cc
c++ -m64 -c -o obj/re2/re2.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/re2.cc
c++ -m64 -c -o obj/re2/regexp.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/regexp.cc
c++ -m64 -c -o obj/re2/set.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/set.cc
c++ -m64 -c -o obj/re2/simplify.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/simplify.cc
c++ -m64 -c -o obj/re2/stringpiece.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/stringpiece.cc
c++ -m64 -c -o obj/re2/tostring.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/tostring.cc
c++ -m64 -c -o obj/re2/unicode_casefold.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/unicode_casefold.cc
c++ -m64 -c -o obj/re2/unicode_groups.o  -std=c++11 -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -I.   -Wall -O3 -fPIC -pthread -std=c++11 -m64 -DNDEBUG re2/unicode_groups.cc
ar rv obj/libre2.a obj/util/rune.o obj/util/strutil.o obj/re2/bitstate.o obj/re2/compile.o obj/re2/dfa.o obj/re2/filtered_re2.o obj/re2/mimics_pcre.o obj/re2/nfa.o obj/re2/onepass.o obj/re2/parse.o obj/re2/perl_groups.o obj/re2/prefilter.o obj/re2/prefilter_tree.o obj/re2/prog.o obj/re2/re2.o obj/re2/regexp.o obj/re2/set.o obj/re2/simplify.o obj/re2/stringpiece.o obj/re2/tostring.o obj/re2/unicode_casefold.o obj/re2/unicode_groups.o 
ar: 正在建立 obj/libre2.a
a - obj/util/rune.o
a - obj/util/strutil.o
a - obj/re2/bitstate.o
a - obj/re2/compile.o
a - obj/re2/dfa.o
a - obj/re2/filtered_re2.o
a - obj/re2/mimics_pcre.o
a - obj/re2/nfa.o
a - obj/re2/onepass.o
a - obj/re2/parse.o
a - obj/re2/perl_groups.o
a - obj/re2/prefilter.o
a - obj/re2/prefilter_tree.o
a - obj/re2/prog.o
a - obj/re2/re2.o
a - obj/re2/regexp.o
a - obj/re2/set.o
a - obj/re2/simplify.o
a - obj/re2/stringpiece.o
a - obj/re2/tostring.o
a - obj/re2/unicode_casefold.o
a - obj/re2/unicode_groups.o
make: Leaving directory `/home/markpeng/Gitlab/timeseries_service/deps/re2/c_src/re2'
Compiled src/re2.erl
Compiling c_src/re2_nif.cpp

[ghost]

Thanks for testing. Fix landed in 1.7.2.