dialyzer crash in freebsd when run with rebar3

[meike-hecker]

rebar3 as test dialyzer produces: =ERROR REPORT==== 16-Aug-2017::13:31:45 === Error in process <0.25819.0> with exit value: {{case_clause,map_type_assoc}, [{erl_types,'-from_form/5-PairsFromForm/3-0-',5, [{file,"erl_types.erl"},{line,4619}]}, {erl_types,from_form,5,[{file,"erl_types.erl"},{line,4612}]}, {erl_types,list_from_form,5,[{file,"erl_types.erl"},{line,4964}]}, {erl_types,list_from_form,5,[{file,"erl_types.erl"},{line,4965}]}, {erl_types,from_form,5,[{file,"erl_types.erl"},{line,4703}]}, {erl_types,fields_from_form,5,[{file,"erl_types.erl"},{line,4957}]}, {erl_types,fields_from_form,5,[{file,"erl_types.erl"},{line,4958}]}, {erl_types,'-record_from_form/6-anonymous-1-',7, [{file,"erl_types.erl"},{line,4899}]}]} ===> Error in dialyzing apps: Analysis failed with error: {{case_clause,map_type_assoc}, [{erl_types,'-from_form/5-PairsFromForm/3-0-',5, [{file,"erl_types.erl"},{line,4619}]}, {erl_types,from_form,5,[{file,"erl_types.erl"},{line,4612}]}, {erl_types,list_from_form,5,[{file,"erl_types.erl"},{line,4964}]}, {erl_types,list_from_form,5,[{file,"erl_types.erl"},{line,4965}]}, {erl_types,from_form,5,[{file,"erl_types.erl"},{line,4703}]}, {erl_types,fields_from_form,5,[{file,"erl_types.erl"},{line,4957}]}, {erl_types,fields_from_form,5,[{file,"erl_types.erl"},{line,4958}]}, {erl_types,'-record_from_form/6-anonymous-1-',7, [{file,"erl_types.erl"},{line,4899}]}]} Last messages in the log cache: Reading files and computing callgraph... done in 6.60 secs Removing edges... done in 0.13 secs *** Error code 1 dialyzer --no_native -r _build/default/lib/* -r _build/default/lib/lager --get_warnings runs without that error on the freebsd machine but finds a lot of dialyzer warnings.

on my ubuntu dev machine rebar3 as test dialyzer runs successfully.

I suspect the problem lies in one of my dependencies. But not sure though.

From Documentation it is not clear to me what is the difference between top_level_dep and all_deps in my case. I have an umbrella app with a bunch of local apps in app folder. I would like dialyzer to analyse my top level app and all apps in app folder but ignore any other dependency.

Are top_level_deps all dependencies listed in rebar.config of top level app? Is there a way to skip/exclude apps from dialyzer with rebar.config?

Environment

Rebar3 report version 3.4.2+build.3852.refffd0970 generated at 2017-08-16T13:33:03+00:00

Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)

Task: dialyzer Entered as: dialyzer

Operating System: amd64-portbld-freebsd10.3 ERTS: Erlang/OTP 19 [erts-8.2] [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] Root Directory: /usr/local/lib/erlang Library directory: /usr/local/lib/erlang/lib

Loaded Applications: bbmustache: 1.3.0 certifi: 2.0.0 cf: 0.2.2 common_test: 1.13 compiler: 7.0.3 crypto: 3.7.2 cth_readable: 1.3.0 dialyzer: 3.0.3 edoc: 0.8.1 erlware_commons: 1.0.0 eunit: 2.3.2 eunit_formatters: 0.3.1 getopt: 0.8.2 inets: 6.3.4 kernel: 5.1.1 providers: 1.6.0 public_key: 1.3 relx: 3.23.1 sasl: 3.0.2 snmp: 5.2.4 ssl_verify_fun: 1.1.2 stdlib: 3.2 syntax_tools: 2.1.1 tools: 2.9

---

Escript path: /root/jenkins/workspace/project_build/rebar3 Providers: app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report shell state tar tree unlock update upgrade upgrade upgrade version xref

[lrascao]

i usually get this kind of error when i'm switching around different OTP versions and forget to clean up stuff in between

[ferd]

Yeah I'd expect this error to be related to modifications in the underlying map implementation across versions, and running dialyzer from a version that does not understand the whole thing.

As for your questions:

• Are top_level_deps all dependencies listed in rebar.config of top level app?

Top-level deps are the direct dependencies of the app, i.e. no transitive ones.

• Is there a way to skip/exclude apps from dialyzer with rebar.config?

Yes. The config format allows an exclude_apps option where you can list app names you do not want to be used. Likely a tuple like {dialyzer, [{exclude_apps, [some_app]}]}. in your rebar.config if I recall correctly.

• I would like dialyzer to analyse my top level app and all apps in app folder but ignore any other dependency.

There is currently no way to ignore all dependencies, but I believe this is not something that would be too hard to add by looking at https://github.com/erlang/rebar3/blob/master/src/rebar_prv_dialyzer.erl#L208-L214

[meike-hecker]

Comparing dialyzer versions on both machines, dialyzer:module_info gives the same vsn hash. The freebsd machine which shows the error is our Jenkis build slave not messing around with different OTP versions and wiping out workspace for each build.

But I just found out that running ./rebar3 dialyzer succeeds. So somehow the issue arises with running dialyzer with test profile on freebsd.

When I compare _build/test/lib directories on both machines I find a difference. On the freebsd machine the top level app copy in _build/test/lib has a paratial copy of apps directory inside so some *.beam files are duplicated in _build/test/lib path which I suspect is wrong. If I run dialyzer from console with -r _build/test/lib/top_level_app I finally get the same error. Running dialyzer from consoler with -r _build/test/lib it complains about duplicates. And -r _build/default/lib succeeds as expected.

I will do further investigation on that later. If I find the time. For now I am happy if I leave out test suites/apps/deps from dialyzer

[ferd]

I'd be more ready to blame a problem with the environment than profile usage in FreeBSD to be honest; the only difference between between a non-default profile and the default one from the point of view of the compiler (aside from configuration merging) is the path where the libraries are being copied and/or symlinked to.

However, I believe the test profile will also compile modules from the test/ directory of an application, and will put them in ebin/ without adding them to the .app file, which, depending on the test structure, could cause odd problems (i.e. if one of your modules calls out to a test module through conditional compilation based on the TEST macro).

But partial copies of libraries are really odd since it would imply the profile logic as a whole is broken, or that file handling is broken on POSIX-compliant systems, which I also find hard to believe at this point.

[ferd]

Closing since I think the question has been answered. If you find the time to do further investigation and find an issue, please reopen this issue.

[fenollp]

Same here on ubuntu + otp20.3

===> Error in dialyzing apps: Analysis failed with error:
{function_clause,
    [{dialyzer_codeserver,get_callbacks,
         [{{case_clause,map_type_assoc},
           [{erl_types,'-from_form/5-PairsFromForm/3-0-',5,
                [{file,"erl_types.erl"},{line,4680}]},
            {erl_types,from_form,5,[{file,"erl_types.erl"},{line,4673}]},
            {erl_types,t_from_form2,5,[{file,"erl_types.erl"},{line,4542}]},
            {erl_types,t_from_form_check_remote,4,
                [{file,"erl_types.erl"},{line,4516}]},
            {dialyzer_utils,'-process_record_remote_types/1-fun-0-',7,
                [{file,"dialyzer_utils.erl"},{line,333}]},
            {lists,mapfoldl,3,[{file,"lists.erl"},{line,1354}]},
            {lists,mapfoldl,3,[{file,"lists.erl"},{line,1355}]},
            {dialyzer_utils,'-process_record_remote_types/1-fun-1-',7,
                [{file,"dialyzer_utils.erl"},{line,332}]}]}],
         [{file,"dialyzer_codeserver.erl"},{line,368}]},
     {dialyzer_plt,insert_callbacks,2,[{file,"dialyzer_plt.erl"},{line,149}]},
     {dialyzer_analysis_callgraph,analysis_start,3,
         [{file,"dialyzer_analysis_callgraph.erl"},{line,143}]}]}
Last messages in the log cache:
  Reading files and computing callgraph... done in 0.11 secs
  Removing edges... done in 0.01 secs

``` Rebar3 report version 3.5.0 generated at 2018-03-26T13:31:35+00:00 ================= Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any) ----------------- Task: dialyzer Entered as: dialyzer ----------------- Operating System: x86_64-unknown-linux-gnu ERTS: Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:0] [hipe] [kernel-poll:false] Root Directory: /home/pete/.bin/builds/otp/20.3 Library directory: /home/pete/.bin/builds/otp/20.3/lib ----------------- Loaded Applications: bbmustache: 1.3.0 certifi: 2.0.0 cf: 0.2.2 common_test: 1.15.4 compiler: 7.1.5 crypto: 4.2.1 cth_readable: 1.3.2 dialyzer: 3.2.4 edoc: 0.9.2 erlware_commons: 1.0.4 eunit: 2.3.5 eunit_formatters: 0.5.0 getopt: 1.0.1 inets: 6.5 kernel: 5.4.3 providers: 1.7.0 public_key: 1.5.2 relx: 3.24.3 sasl: 3.1.1 snmp: 5.2.10 ssl_verify_fun: 1.1.3 stdlib: 3.4.4 syntax_tools: 2.1.4 tools: 2.11.2 ----------------- Escript path: /home/pete/.bin/builds/bin/rebar3 Providers: app_discovery as auto clean compile compile config cover ct cut deps dialyzer do docs edoc escriptize eunit get-deps help info install install_deps key list lock new owner path pkgs publish release relup report search shell state tar tree unlock update upgrade upgrade upgrade user version xref ```

[ElectronicRU]

Same problem here, occurs only with newer rebar3 and only when _build/test is not empty (clearing it helps). Older rebar3 does not have this problem (using Erlang 19.3).

[ElectronicRU]

@ferd a similar issue was discovered in https://github.com/uwiger/parse_trans/issues/28 -- look into whether this can relate. I do use only lager_transform, for the record.

[ferd]

That's odd. I run Dialyzer daily on FreeBSD (11.1 and 11.2) and never get a problem even with _build/test having content in it.

I have used it with 19.3 (from packages), and 20 and 21 (from ports). I'm all on UFS though, if it could make a difference.

[ElectronicRU]

It's a very odd problem which seems to be triggered by combination of newer rebar3, #{type := type} typespec and existing plt. I use Mac OS X, which is basically FreeBSD probably.

[fenollp]

As referred above, the issue was fixed on latest otp20 and all otp21: https://github.com/uwiger/parse_trans/issues/28#issuecomment-395698470 This was a syntax_tools / erl_syntax.erl / OTP issue, not a rebar3 issue.

[ElectronicRU]

@fenollp why isn't the problem sufracing on older rebar3 then? Might be a combination of apps bundled in rebar + apps in native Erlang installation, but I get this problem both with bundled rebar3 and the one from rebar3 local install.

[fenollp]

You'll need to understand the tag which your OTP was built from. It must be one of the ones listed at https://github.com/erlang/otp/commit/d129131ee8ffda4713f807e6148b601c16f1b0bb: OTP-21.0.4 OTP-21.0.3 OTP-21.0.2 OTP-21.0.1 OTP-21.0

Then, git tags are re-mappable so there's the possibility that even though your tag seems right, the tag is now referencing an earlier commit. I'm not even sure of the above list as I am sure to have verified that the patch is on something like 20.3.8. Can't verify manually right now: cloning OTP takes forever.

So my advice here: recompile latest OTP20 or 21 and reinstall. If using kerl don't forget to kerl update releases to get latest checksums. export KERL_BUILD_BACKEND=git in the beginning may help: you'll get source right from github + with HTTPS.

If the issue isn't happening with older rebar3 versions it may be because they are not using too much of the map typing syntax. The linked commit shows the issue was about that.

[fenollp]

Finished cloning OTP:

0 otp.git maint λ git tag  --contains d129131ee8ffda4713f807e6148b601c16f1b0bb
OTP-21.0
OTP-21.0.1
OTP-21.0.2
OTP-21.0.3
OTP-21.0.4
0 otp.git maint λ git tag  --contains 3b3e2f46841e3e86c991be92d62cbb0360ca80e3
OTP-21.0
OTP-21.0-rc2
OTP-21.0.1
OTP-21.0.2
OTP-21.0.3
OTP-21.0.4