Restore normal Makefile behavior for .erl files

[essen]

For .erl and similar files we group everything under one target in order to pass them all to erlc at once. A new Erlang/OTP PR would allow us to restore normal behavior with one file per target, in turn enabling more scenarios and making Erlang.mk less surprising.

https://github.com/erlang/otp/pull/2361

The PR is not yet in a release so this mode of operations should not be the default for quite some time or we'll end up slowing down everyone's builds.

We can most likely enable this erlc server by default since Erlang.mk uses the same options for all the files in a project. It would still be good to allow a project to disable this of course.

[mbj4668]

Now that OTP has implemented the compile server, is this behavior something you plan to change?

[essen]

Yes initially as an option and then possibly by default if it all goes well. But I am busy with other things at the moment.

[sstrollo]

That would be great! But even without that change, maybe it would be good to change the makedep.erl to use the erlc -M option(s) to generate the .d file (since it doesn't seem makedep.erl doesn't quite catch everything). Something along the lines of:

$(PROJECT).d:: $(ERL_FILES) $(call core_find,include/,*.hrl) $(MAKEFILE_LIST)
    erlc -M -MG $(if $(IS_DEP),$(filter-out -Werror,$(ERLC_OPTS)),$(ERLC_OPTS)) -o ebin/ \
        -pa ebin/ -I include/ $(filter-out $(ERLC_EXCLUDE_PATHS),$(COMPILE_FIRST_PATHS) $(1)) $(ERL_FILES) | sed -e 's|ebin/|src/|g' -e 's|.beam|.erl|g' | sed 's|[^\]$$| ; touch $$@|g'

Totally understand being busy :) let us know if you would like pull requests...

[essen]

IIRC erlc -M produces a file that cannot be used for the current compilation method. But it would be perfectly fine otherwise. If makedep.erl doesn't catch everything, please open a ticket/PR to fix it with a test if there isn't one already.

PRs are more than welcome otherwise. Feel free to do the entire compile server work because I'm going to be busy for a while.

[sstrollo]

Thanks @essen. To be more explicit about the "doesn't quite catch everything": if you add -I to ERLC_OPTS then makedep.erl won't catch it (maybe this is an outlier use case?). So for example:

ERLC_OPTS += -I foo
include erlang.mk

then any .hrl files in foo/ will not be included in the .d file. (Maybe adding explicit -I to ERLC_OPTS is the wrong thing to do? Should we add external directories to some other variable?)

You are right about -M and the current compilation method (I tried to compensate for it with the sed commands - but it probably isn't enough since it won't be able to figure out the order they need to be compiled in).

I and @mbj4668 spent an hour taking a stab at the compile server thing, but we realized that it takes quite a bit more re-architecting than we initially thought (on account of how the dependencies are setup currently). So we parked it for now, also too busy:) Cheers!

[essen]

Please open a separate ticket about -I, it sounds like I just didn't think about that. Just need to add the extra include directories to the right places in makedep.erl I think.

[sstrollo]

Okay, thanks! I'll try to make a minimal reproduction and open a separate ticket about that.

[mbj4668]

We spent some more time on the compile server, and managed to get it to work, so I created a PR with the code we came up with. The biggest (?) problem is that ERL_FILES contains both the original source files and generated files (from yrl etc).

[essen]

Plan is to get the compile server work in sometimes next year.