Problems compiling (warnings_as_errors)

[Iulian7]

I am compiling on macOS Sierra, rebar 3.4.1 on Erlang/OTP 20 Erts 9.0 and Elixir 1.4.5.

I managed to compile but only after overriding some dependencies in mix.exs and manually removing warnings_as_errors from some rebar.config files (e.g. cuttlefish and a few others).

The relevant part of my mix.exs looks like this:

  defp deps do
    [
      {:riak_core, "~> 3.0", hex: :riak_core_ng},
      {:poolboy, "~> 1.5", override: true},
      {:cuttlefish, github: "basho/cuttlefish", tag: "2.0.11", manager: :rebar3, override: true},
      {:lager, github: "basho/lager", tag: "3.2.4", manager: :rebar3, override: true},
      {:goldrush, github: "basho/goldrush", tag: "0.1.9", manager: :rebar3, override: true}
    ]
  end

The warning I am getting from cuttlefish for example is export_all flag enabled - all functions will be exported. There are also quite a lot of deprication warnings (e.g. the 'random' module is deprecated; use the 'rand' module instead from basho_stats).

Are these problems due to an incompatibility with Erlang 20 or is there something else I should be doing? Editing the rebar config files of dependencies seems kind of wrong.

https://github.com/basho/cuttlefish/issues/234

[Licenser]

Hi @Iulian7, this is a bug in mix. warning_as_errors should not lead to dependencies failing to compile, it should only evaluate it on the top level project.

Aside from fixing mix, the alternative is making all the dependencies compile w/o warning in Erlang 20. Which probably will eventually happen, but if you like to speed it up I suggest making PR's towards the dependencies in question. From we can do inside this repository, is only updating to newer versions of dependencies as they come along.

[Licenser]

Oh I forgot to say, I'm not yet on Erlang 20 so I don't know what problems will arise from it.

[Iulian7]

I also realise the cuttlefish fork https://github.com/Kyorai/cuttlefish which fixes the warnings_as_errors does not compile with mix for another reason. Using {:cuttlefish, "~> 2.0", override: true} gives back the error Uncaught error: {badmatch,error}.

I am running DEBUG=1 mix deps.compile cuttlefish and getting this:

...
===> Building escript...
===> Creating escript file /Users/iulian/elixir/riak/deps/cuttlefish/_build/default/bin/cuttlefish
===> processing <<"cuttlefish">>
===> new deps of <<"cuttlefish">> found to be []
===> processing <<"getopt">>
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badmatch,error}
===> Stack trace to the error location:
[{rebar_prv_escriptize,find_deps_of_deps,3,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,246}]},
 {rebar_prv_escriptize,find_deps,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,240}]},
 {rebar_prv_escriptize,escriptize,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,104}]},
 {rebar_prv_escriptize,do,1,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,80}]},
 {rebar_core,do,2,
             [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar_hooks,run_provider_hooks_,6,
              [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,49}]},
 {rebar_hooks,run_all_hooks,6,
              [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,16}]},
 {rebar_prv_compile,compile,3,
                    [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_compile.erl"},
                     {line,129}]}]
...

It is worth noting that {:cuttlefish, github: "basho/cuttlefish", tag: "2.0.11", manager: :rebar3, override: true} compiles if I modify rebar.config accordingly.

More details about the environment (as per rebar3 report) for completeness:

-----------------
Operating System: x86_64-apple-darwin16.6.0
ERTS: Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:0] [hipe] [kernel-poll:false] [dtrace]
Root Directory: /usr/local/Cellar/erlang/20.0/lib/erlang
Library directory: /usr/local/Cellar/erlang/20.0/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.3.0
certifi: 0.4.0
cf: 0.2.2
common_test: 1.15
compiler: 7.1
crypto: 4.0
cth_readable: 1.2.6
dialyzer: 3.2
edoc: 0.9
erlware_commons: 1.0.0
eunit: 2.3.3
eunit_formatters: 0.3.1
getopt: 0.8.2
inets: 6.4
kernel: 5.3
providers: 1.6.0
public_key: 1.4.1
relx: 3.23.1
sasl: 3.0.4
snmp: 5.2.6
ssl_verify_fun: 1.1.1
stdlib: 3.4
syntax_tools: 2.1.2
tools: 2.10

-----------------
Escript path: /usr/local/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report shell state tar tree unlock update upg
rade upgrade upgrade version xref 

Also, if I simply run rebar3 compile in deps/cuttlefish with {:cuttlefish, "~> 2.0", override: true} it compiles fine:

===> Verifying dependencies...
===> Fetching getopt ({pkg,<<"getopt">>,<<"0.8.2">>})
===> Downloaded package, caching at /Users/iulian/.cache/rebar3/hex/default/packages/getopt-0.8.2.tar
===> Fetching lager ({pkg,<<"lager">>,<<"3.2.1">>})
===> Downloaded package, caching at /Users/iulian/.cache/rebar3/hex/default/packages/lager-3.2.1.tar
===> Fetching goldrush ({pkg,<<"goldrush">>,<<"0.1.8">>})
===> Downloaded package, caching at /Users/iulian/.cache/rebar3/hex/default/packages/goldrush-0.1.8.tar
===> Compiling goldrush
===> Compiling lager
===> Compiling getopt
===> Compiling cuttlefish
===> Building escript...

Any ideas what might be going on or how this can be addressed?

[Licenser]

@Iulian7 have you asked the mix folks about it? It seems the package itself compiles fine but only files when compiled with mix instead rebar.

My knowledge about mix ends about at knowing what it rougly is, I could imagine it calls some old version of rebar3 perhaps? Or uses an outdated plugin. I am really not sure here.

[Iulian7]

@Licenser Thanks for the pointers. It turns out the rebar3 version mix was using was slightly older. I fixed this but it was not the problem. I will write to the Elixir/mix guys and see if I get anything.

[Licenser]

Awesome, if there is anything we can do let us know!

[Iulian7]

I did some more experimentation. For some of these I had to remove warnings_as_errors from rebar.config because some have not been changed for Erlang/OTP 20.

Configuration 1 (compiles):

 {:riak_core, "~> 3.0", hex: :riak_core_ng},
 {:cuttlefish, github: "basho/cuttlefish", tag: "2.0.11", manager: :rebar3, override: true},
 {:lager, github: "basho/lager", tag: "3.2.4", manager: :rebar3, override: true},
 {:goldrush, github: "basho/goldrush", tag: "0.1.9", manager: :rebar3, override: true}

Configuration 2 (does not work):

      {:riak_core, "~> 3.0", hex: :riak_core_ng},
      {:cuttlefish, "~> 2.0", override: true}, 
      {:lager, github: "basho/lager", tag: "3.2.4", manager: :rebar3, override: true},
      {:goldrush, github: "basho/goldrush", tag: "0.1.9", manager: :rebar3, override: true}

Basically it works with the initial basho/cuttlefish release 2.0.11 but not with Kyorai/cuttlefish release 2.0.12 (which as far as I can tell is what the hex package cuttlefish 2.0.12 is currently linked to). The 2.0.12 release compiles on Erlang 20 without modification (only with rebar3). For the 2.0.11 release I had to remove warnings_as_errors as mentioned.

I find the whole thing extremely strange to say the least and I would like to find out what exactly is going on. I compared the rebar.config file for both versions and they do differ.

cuttlefish 2.0.11 rebar.config

{require_otp_vsn, "R16|17|18"}.

{erl_opts, [warnings_as_errors, {parse_transform, lager_transform}, debug_info, warn_untyped_record]}.

{eunit_opts, [verbose]}.
{cover_enabled, true}.

{escript_emu_args, "%%! -escript main cuttlefish_escript -smp disable +A 0\n"}.
{escript_incl_apps, [goldrush, getopt, lager]}.

{xref_checks, []}.
{xref_queries, [{"(XC - UC) || (XU - X - B - \"(rebar.*|mustache)\" : Mod)", []}]}.

{deps, [
    {getopt, ".*", {git, "https://github.com/basho/getopt.git", {tag, "v0.8.2"}}},
    {lager, ".*", {git, "https://github.com/basho/lager.git", {tag, "3.2.4"}}},
    {neotoma, ".*", {git, "https://github.com/basho/neotoma.git", {tag, "1.7.4"}}}
  ]}.

{post_hooks, [
    {"-win32", compile, "rebar escriptize"},
    {"^((?!-win32).)*$", compile, "./rebar escriptize"}
  ]}.

cuttlefish 2.0.12 rebar.config

{minimum_otp_vsn, "R16"}.

{erl_opts, [warnings_as_errors,
            {parse_transform, lager_transform},
            debug_info,
            warn_untyped_record]}.

{deps, [getopt, lager]}.

{escript_emu_args, "%%! -escript main cuttlefish_escript -smp disable +A 0\n"}.
{escript_incl_apps, [goldrush, getopt, lager, cuttlefish]}.
{escript_main_app, cuttlefish}.

{provider_hooks, [{post, [{compile, {default, escriptize}}]}]}.

{eunit_opts, [verbose]}.
{cover_enabled, true}.

{profiles, [{dev, [{deps, [neotoma]},
                   {plugins, [rebar3_neotoma_plugin]}]}]}.

There are some things that differ in these configurations. As I am not that familiar with Erlang I can't really say what implications these would have.

Perhaps some more seasoned Erlang devs can comment on this.

[Iulian7]

OK, the problem is with the post provider_hooks.

If I comment out {provider_hooks, [{post, [{compile, {default, escriptize}}]}]}. from rebar.config everything compiles fine. This escriptize hook doesn't seem to play well with mix. It's probably looking for some files in the wrong directories or something. Maybe...

I think this is related: https://github.com/erlang/rebar3/issues/1560

Does anyone think this is an issue that needs to be taken up to the main Elixir guys? Or is something else going on here?

[fire]

To compile riak on erlang 20:

Add this to mix.exs.

{:riak_core, github: "Kyorai/riak_core", branch: "fifo-merge"},

Modify rebar.conf in poolboy to remove warnings as errors:

{erl_opts, [debug_info,
            {platform_define, "^[0-9]+", namespaced_types}]}.
{eunit_opts, [verbose]}.
{cover_enabled, true}.

Modify rebar.conf in cuttlefish to remove warnings as errors and provider_hooks:

{minimum_otp_vsn, "R16"}.

{erl_opts, [{parse_transform, lager_transform},
            debug_info,
            warn_untyped_record]}.

{deps, [getopt, lager]}.

{escript_emu_args, "%%! -escript main cuttlefish_escript -smp disable +A 0\n"}.
{escript_incl_apps, [goldrush, getopt, lager, cuttlefish]}.
{escript_main_app, cuttlefish}.

{eunit_opts, [verbose]}.
{cover_enabled, true}.

{profiles, [{dev, [{deps, [neotoma]},
                   {plugins, [rebar3_neotoma_plugin]}]}]}.

Modify rebar.conf in riak_ensemble to remove warnings as errors:

{deps, [{lager, "~>3.2.1"}]}.

{erl_opts, [debug_info,
           warn_untyped_record,
           {platform_define, "^[0-9]+", namespaced_types},
           {platform_define, "^((1[8|9])|2)", rand_module},
           {parse_transform, lager_transform}]}.

{eunit_opts, [verbose]}.
{edoc_opts, [preprocess,
             {dir, "edoc"}]}.
{cover_enabled, true}.
{xref_checks, [undefined_function_calls]}.

{profiles, [{test, [{deps, [{riak_test, ".*", {git, "git://github.com/basho/riak_test", {branch, "develop"}}}]}]}]}.

Modify rebar.conf in riak_core to remove warnings as errors:

{erl_first_files, ["src/gen_nb_server.erl", "src/riak_core_gen_server.erl",
           "src/riak_core_stat_xform"]}.
%{cover_enabled, true}.
{erl_opts, [{parse_transform, lager_transform},
            debug_info, {platform_define, "^[0-9]+", namespaced_types},
            {platform_define, "18", old_rand},
            {platform_define, "17", old_rand},
            {platform_define, "^R15", old_hash}]}.
{edoc_opts, [{preprocess, true}]}.
{eunit_opts, [verbose]}.
{xref_checks, []}.
{xref_queries, [{"(XC - UC) || (XU - X - B - \"(cluster_info|dtrace)\" : Mod)", []}]}.

{deps, [
  %% Stuff we need to work with rebar3
  {goldrush, "~>0.1.8"},
  {cuttlefish, "~>2.0.12"},
  {clique, "~>0.3.9"},
  %% We now include folsom
  folsom,
  %% Normal deps
  {lager, "~>3.2.4"},
  {poolboy, "0.8.2", {pkg, basho_poolboy}},
  {basho_stats, "~>1.0.3"},
  {riak_sysmon, "~>2.1.3"},
  {riak_ensemble, "~>2.4.0", {pkg, riak_ensemble_ng}},
  {pbkdf2, "~>2.0.0"},
  {blume, "~>0.1.0"},
  {chash, "~>0.1.1"},
  {eleveldb, "~>2.2.20"},
  %%{eleveldb, {git, "https://github.com/project-fifo/eleveldb.git", {branch, "hex-2.2.19-diayzer"}}},
  {exometer_core, "~>1.0.0", {pkg, basho_exometer_core}}
]}.

%%-------------------------------------------------------------------
%% Profiles
%%-------------------------------------------------------------------

{profiles,
 [{test, [{deps,
           [{mustache, ".*", {git, "https://github.com/mojombo/mustache.erl.git", {tag, "v0.1.1"}}}]}]},
  {docs, [{deps, [{edown, "0.7.0"}]}]},
  {eqc, [{erl_opts, [{d, 'TEST'}]}, {deps, [meck]}, {plugins, [rebar_eqc]}]},
  {prod, [{relx, [{dev_mode, false}]}]},
  {lint,
   [{plugins,
     [{rebar3_lint,
       {git, "https://github.com/project-fifo/rebar3_lint.git",
        {tag, "0.1.4"}}}]}]}
 ]}.
{overrides,
 [
  %% R19
  {override, pooler,
   [{erl_opts,
     [{platform_define, "^[0-9]+", namespaced_types},
      debug_info,
      inline]}]},
  %% Normal
  {override, setup, [{post_hooks, []}]},
  {override, eleveldb,
   [{pre_hooks, [{compile, "c_src/build_deps.sh get-deps"},
                 {compile, "c_src/build_deps.sh"}]},

    {post_hooks, [{clean, "c_src/build_deps.sh clean"}]},

    {plugins,
     [pc]},

    {provider_hooks,
     [{post,
       [{compile, {pc, compile}},
        {clean, {pc, clean}}]}]}]}]}.

Make sure you have sudo apt-get install erlang-eunit erlang-os-mon.

I was unable to start riak though.

[fire]

Copying https://github.com/basho/riak_core/blob/develop/priv/riak_core.schema into priv/ I was able to get riak to run.

The other riak_core.schema didn't work.

[Licenser]

@fire what branch was that on? master or fifo-merge? If it's the fifo branch what was the error prompted?

Also, I think we really need a decent solution to deal with warn_on_errors, disabling them in all projects doesn't seem like a good solution :( of cause making them all error free would be the best :P

[facundoolano]

Oh I forgot to say, I'm not yet on Erlang 20 so I don't know what problems will arise from it.

@Licenser I'm trying it now with Erlang 20, and I have these errors:

• poolboy has warnings_as_errors and uses export_all
• riak_ensemble and riak_core have warnings_as_errors and uses gen_fsm

(I'm overriding erl_opts to work around them)

[Licenser]

removing warnings as errors is only a short-term remedy, those warnings will pop up as real errors eventually.

Sadly I'm a bit preoccupied with other things at the moment and don't have an erlang 20 installed, so unless someone wants to step up and properly fix the warnings and make a PR it will have to wait a bit until I get some air.

[bieniusa]

I would be happy to help in upgrading and giving back to this awesome community! But since all these forks are really hard to follow for me, could you point me out where I find the poolboy repo that I should clone, fix and do a pull request on?? The link on hex is broken 😕 -> Ah, educated guess: There is an "o" missing in the url!

[ghost]

Also, I think we really need a decent solution to deal with warn_on_errors, disabling them in all projects doesn't seem like a good solution :( of cause making them all error free would be the best :P

Just a thought - but rebar3 now has the means to override properties in all child projects - would be ideal if mix could do the same - will create an issue and link back to this.

[Licenser]

@bieniusa sorry for the wrong link, I see you found it already and thanks for updating poolboy!!!

[Licenser]

since @bieniusa was so kind to get the erlang 20 ball rolling I added a https://github.com/Kyorai/riak_core/tree/erl20 branch to track it :)

[Licenser]

See #33