Closed KennethL closed 1 year ago
for projects not using rebar3
Would you mind providing some more details on the specific use-case here?
Background:
Rebar3 allows to get all the info in an easy way - https://github.com/WhatsApp/eqwalizer/blob/main/eqwalizer_rebar3/src/eqwalizer_build_info_prv.erl
It would not be hard to provide other integrations if a project model of "something not rebar" is well-defined.
but I don't understand why there is a dependency to rebar3 in runtime.
There is no real runtime dependency, - it's just "analysis-time" dependency.
eqwalizer:dynamic()
type - which is "bootstrapped" in https://github.com/WhatsApp/eqwalizer/blob/8c83a26b8a409614a80301c5f1d90ea4e4d60dd4/eqwalizer_support/src/eqwalizer.erl#L21 (see more details in https://github.com/WhatsApp/eqwalizer/blob/main/docs/reference/modes.md#type-eqwalizerdynamic)eqwalizer_support
library comes with fixes for some incorrect specs for OTP libraries. - see https://github.com/WhatsApp/eqwalizer/blob/main/eqwalizer_support/src/eqwalizer_specs.erlThis library is not needed at runtime - only during analysis time.
for projects not using rebar3
Would you mind providing some more details on the specific use-case here?
The most common use case would be projects which are build using make
, for example the otp applications. There are also a number of other very big projects which are built with make
. I also know projects built with Bazel.
If it is know what info eqWAlizer needs in a project definition it would be no big deal to create it and provide it in a suitable form. Probably the same form as rebar build_info
provides.
Background:
1. under the hood eqWAlizer needs a project definition in order to be able to read type definitions and specs from different modules - since in the wild specs and types are scattered across different modules. 2. It also distinguishes between 1st party code and 3rd-party dependencies (since there is little sense to type-check dependencies you don't own in the first place). 3. It also needs access to compiler options (includes and macro defines) for better parsing and error reporting (see the next item) 4. under the hood we use a parser with precise range information to be able to provide modern UX with user-friendly error messages - see the screenshot (see https://github.com/WhatsApp/eqwalizer/tree/main/mini-elp/parse_server)
Is the parse_server running in the background? How does it get info about what to parse?
Rebar3 allows to get all the info in an easy way - https://github.com/WhatsApp/eqwalizer/blob/main/eqwalizer_rebar3/src/eqwalizer_build_info_prv.erl
It would not be hard to provide other integrations if a project model of "something not rebar" is well-defined.
but I don't understand why there is a dependency to rebar3 in runtime.
There is no real runtime dependency, - it's just "analysis-time" dependency.
With runtime I mean runtime for eqwalizer. As I understand it rebar3 build_info
is called when running elp eqwalize ...
.
1. For gradual typing eqwalizer has `eqwalizer:dynamic()` type - which is "bootstrapped" in https://github.com/WhatsApp/eqwalizer/blob/8c83a26b8a409614a80301c5f1d90ea4e4d60dd4/eqwalizer_support/src/eqwalizer.erl#L21 (see more details in https://github.com/WhatsApp/eqwalizer/blob/main/docs/reference/modes.md#type-eqwalizerdynamic) 2. The `eqwalizer_support` library comes with fixes for some incorrect specs for OTP libraries. - see https://github.com/WhatsApp/eqwalizer/blob/main/eqwalizer_support/src/eqwalizer_specs.erl
This library is not needed at runtime - only during analysis time.
Analysis time is when elp eqwalizer
is running isn't it?
In summary. If I know what info Eqwalizer expects from 'rebar3 build_info' I am quite sure there are other means to provide the same info and give it as arguments to Eqwalizer (or in a single "config" file).
As of release v0.11.2, mini-elp now supports a JSON file as input as a substitute to rebar3
. This can be achieved using the --project
option, e.g., elp eqwalize-all --project path/to/build_info.json
, or elp eqwalize my_module --project path/to/build_info.json
.
The JSON file is structured in this way:
{
"apps": [app list],
"deps": [app list], // 3rd party dependencies (not type-checked), defaults to []
"root": "path/to/root" // Defaults to ""
}
where an app
is a map structured as such:
{
"name": "app_name",
"dir": "path/to/app", // Relative to project root
"src_dirs": ["path/to/src", ...], // Relative to app dir, defaults to ["src"]
"extra_src_dirs": ["path/to/extra_src", ...], // Relative to app dir, defaults to []
"ebin": "path/to/ebin", // Relative to app dir, defaults to "ebin"
"include_dirs": ["include", ...], // Relative to app dir, defaults to []
"macros": ["MACRO", ...], // Defaults to []
}
As an example, see the build_info.json file used to test mini-elp.
Note that currently, OTP is automatically detected and included in the build info passed to eqWAlizer using erl
. In the future, it should be possible to add a command line option or an additional field in the JSON file to specify a custom OTP version, depending on needs.
Of course, this is a first rough draft fix to solve this issue, please let us know how we can build on this to better support your use cases.
@VLanvin I am trying to get the example from mini-elp/test_projects/standard
to work but I fail.
Aren't we supposed to run elp eqwalize-all --project build_info.json
from the mini-elp/test_projects/standard
folder?
From the OTP is automatically detected and included in the build info passed to eqWAlizer
I assume that I do not need to do anything special to include OTP.
> ~/Downloads/elp-linux/elp eqwalize-all --project build_info.json
Loading JSON manifest
Loading applications ████████████████████ 3/3
Seeding database
Compiling dependencies
eqWAlizing ██████████░░░░░░░░░░ 5/10
Exception in thread "main" java.lang.AssertionError: assertion failed: could not find erlang:boolean/0 from app_a_lists
at scala.Predef$.assert(Predef.scala:279)
at com.whatsapp.eqwalizer.tc.Util.$anonfun$getTypeDeclBody$3(Util.scala:112)
at scala.Option.getOrElse(Option.scala:201)
at com.whatsapp.eqwalizer.tc.Util.getTypeDeclBody(Util.scala:109)
at com.whatsapp.eqwalizer.tc.Subtype.subTypePol(Subtype.scala:63)
at com.whatsapp.eqwalizer.tc.Subtype.subType(Subtype.scala:33)
at com.whatsapp.eqwalizer.tc.Elab.elabExpr(Elab.scala:329)
at com.whatsapp.eqwalizer.tc.Check.checkExpr(Check.scala:93)
at com.whatsapp.eqwalizer.tc.Check.checkBody(Check.scala:58)
at com.whatsapp.eqwalizer.tc.Check.checkClause(Check.scala:74)
at com.whatsapp.eqwalizer.tc.Check.$anonfun$checkFun$1(Check.scala:38)
at scala.collection.LazyZip2$$anon$1$$anon$2.next(LazyZipOps.scala:42)
at scala.collection.immutable.List.prependedAll(List.scala:153)
at scala.collection.immutable.List$.from(List.scala:684)
at scala.collection.immutable.List$.from(List.scala:681)
at scala.collection.BuildFromLowPriority2$$anon$11.fromSpecific(BuildFrom.scala:112)
at scala.collection.BuildFromLowPriority2$$anon$11.fromSpecific(BuildFrom.scala:109)
at scala.collection.LazyZip2.map(LazyZipOps.scala:37)
at com.whatsapp.eqwalizer.tc.Check.checkFun(Check.scala:38)
at com.whatsapp.eqwalizer.Pipeline$.tolerantCheckFun(Pipeline.scala:156)
at com.whatsapp.eqwalizer.Pipeline$.checkFun(Pipeline.scala:136)
at com.whatsapp.eqwalizer.Pipeline$.$anonfun$checkForms$2(Pipeline.scala:61)
at scala.collection.immutable.List.foreach(List.scala:333)
at com.whatsapp.eqwalizer.Pipeline$.checkForms(Pipeline.scala:44)
at com.whatsapp.eqwalizer.util.ELPDiagnostics$.getDiagnostics(ELPDiagnostics.scala:61)
at com.whatsapp.eqwalizer.util.ELPDiagnostics$.$anonfun$getDiagnosticsIpc$1(ELPDiagnostics.scala:50)
at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:37)
at com.whatsapp.eqwalizer.util.ELPDiagnostics$.getDiagnosticsIpc(ELPDiagnostics.scala:49)
at com.whatsapp.eqwalizer.Main$.ipc(Main.scala:70)
at com.whatsapp.eqwalizer.Main$.main(Main.scala:33)
at com.whatsapp.eqwalizer.Main.main(Main.scala)
thread 'main' panicked at 'failed to parse stdout from eqwalizer: Error("EOF while parsing a value", line: 1, column: 0)', crates/eqwalizer/src/ipc.rs:82:40
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
@kikofernandez - eqWAlizer requires OTP 25. - It looks like you are trying to run it with an older OTP (which doesn't have erlang:boolean()
type in erlang
module)
Is there any way to print the OTP location used by eqwalizer?
There is no way I know of to print it using elp/eqWAlizer, but it uses the command erl -noshell -eval "io:format('~s', [code:root_dir()])" -s erlang halt
internally to detect it, so you can just use this command in a terminal to get the same result.
Kenneth and I have tested with OTP 25.1.2 and we both get the following error:
eqwalizer/mini-elp/test_projects/standard$ ~/Downloads/elp eqwalize-all --project build_info.json
Loading JSON manifest
Loading applications ████████████████████ 3/3
Seeding database
Compiling dependencies
eqWAlizing ██████░░░░░░░░░░░░░░ 3/10
Exception in thread "main" java.lang.AssertionError: assertion failed: could not find erlang:boolean/0 from app_a_lists at scala.Predef$.assert(Predef.scala:279) at
com.whatsapp.eqwalizer.tc.Util.$anonfun$getTypeDeclBody$3(Util.scala:112) at
scala.Option.getOrElse(Option.scala:201) at
com.whatsapp.eqwalizer.tc.Util.getTypeDeclBody(Util.scala:109) at
com.whatsapp.eqwalizer.tc.Subtype.subTypePol(Subtype.scala:63) at
com.whatsapp.eqwalizer.tc.Subtype.subType(Subtype.scala:33) at
com.whatsapp.eqwalizer.tc.Elab.elabExpr(Elab.scala:329) at
com.whatsapp.eqwalizer.tc.Check.checkExpr(Check.scala:93) at
com.whatsapp.eqwalizer.tc.Check.checkBody(Check.scala:58) at
com.whatsapp.eqwalizer.tc.Check.checkClause(Check.scala:74) at
com.whatsapp.eqwalizer.tc.Check.$anonfun$checkFun$1(Check.scala:38) at
scala.collection.LazyZip2$$anon$1$$anon$2.next(LazyZipOps.scala:42) at
scala.collection.immutable.List.prependedAll(List.scala:153) at
scala.collection.immutable.List$.from(List.scala:684) at
scala.collection.immutable.List$.from(List.scala:681) at
scala.collection.BuildFromLowPriority2$$anon$11.fromSpecific(BuildFrom.scala:112) at
scala.collection.BuildFromLowPriority2$$anon$11.fromSpecific(BuildFrom.scala:109) at
scala.collection.LazyZip2.map(LazyZipOps.scala:37) at
com.whatsapp.eqwalizer.tc.Check.checkFun(Check.scala:38) at
com.whatsapp.eqwalizer.Pipeline$.tolerantCheckFun(Pipeline.scala:156) at
com.whatsapp.eqwalizer.Pipeline$.checkFun(Pipeline.scala:136) at
com.whatsapp.eqwalizer.Pipeline$.$anonfun$checkForms$2(Pipeline.scala:61) at
scala.collection.immutable.List.foreach(List.scala:333) at
com.whatsapp.eqwalizer.Pipeline$.checkForms(Pipeline.scala:44) at
com.whatsapp.eqwalizer.util.ELPDiagnostics$.getDiagnostics(ELPDiagnostics.scala:61) at
com.whatsapp.eqwalizer.util.ELPDiagnostics$.$anonfun$getDiagnosticsIpc$1(ELPDiagnostics.scala:50) at
scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100) at
scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87) at
scala.collection.mutable.ArraySeq.map(ArraySeq.scala:37) at
com.whatsapp.eqwalizer.util.ELPDiagnostics$.getDiagnosticsIpc(ELPDiagnostics.scala:49) at
com.whatsapp.eqwalizer.Main$.ipc(Main.scala:70) at
com.whatsapp.eqwalizer.Main$.main(Main.scala:33) at com.whatsapp.eqwalizer.Main.main(Main.scala)
thread 'main' panicked at 'failed to parse stdout from eqwalizer: Error("EOF while parsing a value", line: 1, column: 0)', crates/eqwalizer/src/ipc.rs:82:40
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
This was with OTP 25.1.2
How did you compile/install OTP?
I cannot reproduce the problem locally using ./otp_build all -a
followed by make install
, even on a fresh install.
It seems that it may come from OTP being built but not installed, since we detect and load OTP assuming a very precise directory structure.
@VLanvin Yes, I found the issue :) Thanks for the tip. For understanding why this happens, I was adding otp/bin
to our development path, as follows:
export PATH=/home/<user>/Code/otp/bin:$PATH
Even when the otp/bin
contains erl
with OTP 25 (same as the installed one in /usr/local/bin
) it will crash.
The solution is to use the default (installed) Erlang path.
I will continue testing if we can run eqwalizer on OTP code. Thanks!
I get another error, maybe you can help?
To make things local to the mini-elp
project I run the following commands from the folder eqwalizer/mini-elp/test_projects/standard
:
$ ln -s <path-to-local-otp>/otp otp # create a symlink to our dev OTP
Then, I have updated the build_info.json
as follows:
{
"apps": [
{
"name": "ftp",
"dir": "otp/lib/ftp",
"ebin": "ebin",
"macros": ["TEST"],
"src_dirs": ["src"]
},
{
"name": "eqwalizer",
"dir": "eqwalizer",
"ebin": "../_build/test/lib/eqwalizer/ebin",
"macros": ["TEST"],
"src_dirs": ["src"]
}
],
"deps": [
],
"root": ""
}
and I get the following error:
I have tested other projects (jesse
) with the symlink and it works, but the ftp
app from otp
crashes.
TL;DR: The latest release should solve the problem.
Long explanation: eqWAlizer was not meant to be used to type-check OTP modules initially. Since eqWAlizer needs info about OTP modules (types and specs) to eqWAlize anything, it was configured to always load this info from OTP BEAM files compiled with debug_info
.
Unfortunately, the abstract format used to represent Erlang terms does not contain enough location info for mini-elp to report errors (mini-elp works with ranges, whereas the abstract format only contains line/col info), which was causing a crash.
With the latest release, eqWAlizer now loads OTP modules from ETF instead if they belong to an app present in the apps:
field of the build info file.
closing, - since it is supported starting from 0.11.14
Can Eqwalizer be used without rebar3? If that was possible it would be much easier for projects not using rebar3 to try it out and use it.
I understand the use case for a rebar3 plugin but I don't understand why there is a dependency to rebar3 in runtime.
elp eqwalize m.erl