Allow mix to pass compiler options to erlang

[ewildgoose]

At present it's possible to compile Elixir to HiPE native code using something like the following command line:

ERL_COMPILER_OPTIONS="[native,{hipe, [o3]}]" mix compile

However, it's not clear how to do the same through the mix config file? Something which seems "close" is the project option: ":erlc_options", however, I can't figure out the syntax on how this passes through to the underlying compiler?

I tried adding the config option below, but it doesn't work as expected: erlc_options: [:native, {:hipe, [:verbose, :o3]}],

Can this be done through the existing :erlc_options feature (I think perhaps it can?) Failing that can we please add a new config option to pass this through?

(Justification: On my sample project invoking tight loops I get nearly a 2x speedup using the native compiler)

[josevalim]

@ewildgoose do you want to compile the whole project with native though? One option is for you to include @compile [:native, {:hipe, [:verbose, :o3]}] in the module with the tight loop. What do you think? Have you given something like that a try? If that doesn't work, try:

@compile :native
@compile {:hipe, [:verbose, :o3]}

[ewildgoose]

Hmm, that does set the compile options correctly for the modules I add this to. Interestingly enough, after adding this option to the two modules that make up the bulk of the code, my project now runs at nearly half speed though!

I think this highlights that hipe can be a performance detriment if you bounce between compiled and non compiled code in your tight loop?

Can you offer an opinion on whether my setting the ENV variable will also cause all the native dict/list, etc modules to be compiled in this project? That would certainly explain what I see (and argue for an option to set it at project level).

[josevalim]

The environment option will compile your project and all dependencies that are eventually compiled with that command. That's an interesting contrast though, if we support an option in your project, it won't affect your dependencies! So maybe the ENV option is the best way to go? Also, native takes more time to compile, so it is something you want to enable for builds + production but not locally?

I think this highlights that hipe can be a performance detriment if you bounce between compiled and non compiled code in your tight loop?

I haven't heard of it before so maybe something else is off (not confirming nor denying your initial suspicion). :)

[josevalim]

Ping.

[ewildgoose]

OK, after looking more closely I made a mistake on my previous observation:

a) I can observe specific speedups by adding @compile attributes to certain modules (ie performance stacks) b) I can't reproduce a slowdown by mixing compiled and non compiled code, not sure what happened there? c) If I add @compile to all modules I seem to get the same performance as building all with the ENV option. Previously I simply missed adding the @compile attribute to one module, hence why the speedup didn't match

I think it there is no disadvantage, then it would be extremely convenient to be able to set the ENV environment through mix? As you point out it has project level granularity, which I think means it will tend to match the logical unit of (performance) concern?

I'm not actually sure what disadvantages there are to HiPE? Perhaps moving towards having an option to enable it by default is a good direction?

FYI: The main speedup in my benchmark was in my "Queue" module. It's the inner loop of the calculation, so of course it makes sense, but it also highlights that such optimisations might be helpful more widely (the second module computes primes, but there is a lesser speed there curiously)

[josevalim]

I'm not actually sure what disadvantages there are to HiPE? Perhaps moving towards having an option to enable it by default is a good direction?

The disadvantage is that it takes way longer to compile. Take Ecto for example, compiling + running tests goes from 9s to 46s when native is enabled. That price is way too high to pay on development.

[wkhere]

2cents,

+1 for leaving hipe to env or to explicit module attribute. -1 for complicating the mix logic with extra checks if the whole set of beams is hipe or not (it probably should do it, as the mixture of hipe and non-hipe is sometimes [1] not preferred; this would raise a number of corner cases and decisions based on different preferences for different users).

I also haven't seen a practice in Erlang ecosystem to put hipe into rebar.config & family.

[1] Curious, are Erlang maintainers / hipe authors confirming this?

[ewildgoose]

I'm not sure why that's helpful? If there is EVER a requirement to build a component with HiPE, then you are forcing me to do that build manually as I can't put it into mix?

Jose seems to be confirming that want any use of hipe to be environment specific, ie deployment, not development. This implies that any such option would be better placed in the environment config, rather than using @compile directives?

+1 for an option to have mix compile a specific project with native compilation, but with control over whether to use it for dev/deploy environments

P.S. Does native compile make any noticeable differences to benchmarks for Ecto?

[wkhere]

Tough statements :) I'm not forcing you to do anything. Nor my lawyer. ;D

If I've driven my coding by theories on what ever can be my by possible req then I'd go nowhere, but that's me :D

Now short to the point. Use of hipe env-specific -> means they can be driven by env var. Can't see how it implies it's better placed in the config (I suppose you mean config.exs). It's just another option of doing this, better or not, depends on opinion.

[ewildgoose]

Apologies, but I'm not familiar enough with Elixir. Can you please suggest a suitable way to set the ENV value which can be used in an automated setup?

I guess to be quite specific, we have narrowed the hypothetical situation down to:

• I have identified that for some reason Ecto is my bottleneck, but magically by using the native compiler my performance is doubled...
• I don't want to use native compiler while I'm hacking on an Ecto bug fix (possibly because my Erlang hipe is broken...)
• But I have tested this on my deployment machine, so when I deploy my overarching project, I would like to ensure that Ecto is pulled in with native compilation, but the rest of my app is not (hypothetically because it breaks something)

So I guess first knock down my scenario, but assuming it stands up, perhaps we already have everything in Elixir? Could you please help me with a suggestion on how to use it?

(Just to compare with what I'm trying to do now: in my test app I compute some primes, but I get a substantial speedup if I use native compilation on the datatype module I use, a pairing heap. Now the pairing heap is/will become a separate dependency, so I now desire that I can indicate that the pairing heap is always natively compiled when including into my project. Does that seem reasonable?)

[wkhere]

Well, if you'd like to distinguish whether your app is hipe'd when you code it vs when is used as dep, then config.exs wouldn't help you as well I guess..

I'm not best at hypothetical situations, as I mentioned, but I can give you some real example of my existing setup.

I have an app using Ecto over a legacy database. There's also legacy Rails app running and parts of the old code are being superseded or extended by this Elixir app. Db config: database names and user credentials should be shared between old and new app, so they come from the script which sets env vars. Now the Elixir app reads env vars and that's it. This is similar to your case, because there's the :compiler app in the Erlang VM which also reads env vars - ERLC_COMPILER_OPTIONS.

To run tests (which go over another existing database), I use sth like ./myenv mix test where myenv is a few-liner script setting the env vars and at the end doing exec $@. I also run production app this way.

In the same way you could have your compilation for special cases just prepended by ./my_hipe or whatever. This seems a common sense for me.. I'm also pretty sure crafting such mini script takes much less time than reevaluating (I mean, even thinking of) all possible scenarios, but again that's my style of getting things done..

[bitwalker]

Just my two cents:

1. config.exs would be fine for setting this value for dev or prod only, using environment based configuration.

2.You can use aliases to approximate what your myenv script is doing. Why not set an alias for compiling your project natively?

Paul On Aug 12, 2014 6:56 PM, "Wojciech Kaczmarek" notifications@github.com wrote:

Well, if you'd like to distinguish whether your app is hipe'd when you code it vs when is used as dep, then config.exs wouldn't help you as well I guess..

I'm not best at hypothetical situations, as I mentioned, but I can give you some real example of my existing setup.

I have an app using Ecto over a legacy database. There's also legacy Rails app running and parts of the old code are being superseded or extended by this Elixir app. Db config: database names and user credentials should be shared between old and new app, so they come from the script which sets env vars. Now the Elixir app reads env vars and that's it.

To run tests (which go over another existing database), I use sth like ./myenv mix test where myenv is a few-liner script setting the env and then does exec $@

In the same way you could have your compilation for special cases just prepended by ./my_hipe or whatever. I'm also pretty sure crafting takes much less time then reevaluating all possible scenarios, but again that's my style of getting things done..

— Reply to this email directly or view it on GitHub https://github.com/elixir-lang/elixir/issues/2665#issuecomment-51993794.

[wkhere]

@bitwalker aliases may be even better for this! Curious, can you give some example? I only found Jose's initial proposal

[josevalim]

Mix aliases doesn't work because we need to set the environment variable before starting Mix. Basically we have three options here:

1. Mix project configuration - a bad idea because when used as a dependency the project should not necessarily force native
2. config.exs - solves the issue with Mix project configuration but it is per project. I.e., once you set it, all of your deps and projects will be compiled as native
3. Continue setting the ENV - gives specific control about when to compile and which dependencies to compile as native

For example, in production, you could do:

mix deps.get --only prod
ERL_COMPILER_OPTIONS="[native,{hipe, [o3]}]" mix deps.compile dep_that_requires_native
mix compile

That said, I believe the best option is what we have today: just set an environment variable. Thanks everyone for the discussion!

[mgwidmann]

Would be nice if ERL_COMPILER_OPTIONS="[native,{hipe,[o3]}]" were rolled into nice compact --native flag to mix compile --native or mix deps.compile --native some_project ...

Theres no way I'm going to remember ERL_COMPILER_OPTIONS="[native,{hipe,[o3]}]"...

[c0b]

but how about other compiler options, if I want to set for erl project wide? in my pure erlang project, I use a Makefile and erlc -I .../include -DHAVE_UNIT ... parse_transform like long options, to compile all erl files, if I want to switch to mix managed, how can I save the same set of options to mix.exs ?

[josevalim]

@c0b if your project has a makefile, you can tell Mix to use when it when compiling your project:

{:my_erl_dep, "~> 1.0.0", compile: "make compile"}

[c0b]

because its Makefile is using an old build framework (build from configure.ac and autoconf/automake) which we want to get rid of, and I don't see most elixir projects need a Makefile at all; one pain point in rebar2 and resolved in rebar3 is about the need of makefile; most rebar2 project is still relying on a Makefile and rebar3 doesn't need that.

I'm evaluating the way of building a pure erlang project with mix, just wonder is it possible to write erl_compile_options in mix.exs file ? like the erl_opts in rebar.config ? or will it be a new feature request to mix ?

1. http://kelly-mclaughlin.com/blog/2014/12/12/building-an-erlang-project-with-mix/
2. https://github.com/rebar/rebar/blob/master/rebar.config.sample#L26-L33

[josevalim]

Yes, it is definitely possible. :erlc_options in your mix.exs file. For more info, run mix help compile.erlang. :)

[c0b]

@josevalim could you post a gist for how to do that if you have an example? I'm with erlang for some years but still entry level with elixir/mix, can't figure out how to write there; I searched a little bit but couldn't find any good examples, the official doc is only a reference, I'm not seeing any good tutorial, if you know any good ones will be appreciated

1. http://featurebranch.com/using-mix-to-compile-your-erlang-projects/
2. http://elixir-lang.org/docs/stable/mix/Mix.Tasks.Deps.Compile.html

also, is there an command switch to turn on verbose mode for mix compile ? because right now, I can see mix is trying to compile my src/*.erl files and all failed, but I don't know what are the options in use, is there something like gcc -v to show the compilation internals like how does call erlc ?

is there a full reference for all switches of mix? even the mix -v to show a version I don't see anywhere is documenting it, I've tried mix help, mix -h, mix --help, ... all are listing available tasks, I assume there is another option to lookup help for all available command line switches?

$ mix -v
Mix 1.1.1

[josevalim]

I am sorry. I believe you should be able to run it by adding the following inside def projectin your mix.exs:

def project do
  [...,
  erlc_options:  [:no_debug_info, {:i, 'myinclude'}],
  ...]
end

[c0b]

is there some debugging command line switches for mix tool? like if a Makefile not working as expected we can call it by make --debug it will show a lot of verbose information really useful for debugging