rebar3 creates source.dag files

[maxlapshin]

Environment

• Add the result of rebar3 report to your message:

macbook-pro:flussonic max$ ../rebar3/rebar3 compile 
===> "/Users/max/Sites/flussonic/apps/ad_injector/ebin/ad_injector.app" is missing description entry
===> "/Users/max/Sites/flussonic/apps/ad_injector/ebin/ad_injector.app" is missing kernel from applications list
...
===> "/Users/max/Sites/flussonic/_build/default/lib/snmp2/ebin/snmp2.app" is missing description entry
===> Compiling lager
macbook-pro:flussonic max$ find _build/ -name source.dag
_build//default/lib/demo/.rebar3/rebar_compiler_erl/source.dag
_build//default/lib/luerl/.rebar3/rebar_compiler_xrl/source.dag
...

Rebar is from git:

macbook-pro:rebar3 max$ git describe 3.11.1-3-g22b221d0

Current behaviour

Rebar creates .rebar3/rebar_compiler_xrl/source.dag

Expected behaviour

I do not expect unknown files that are not even greppable in sources.

What do these files mean? Why are they created and what for?

[ferd]

https://github.com/erlang/rebar3/blob/ec224b7921cda1f9a1ef4373e80e706c173bf75f/src/rebar_compiler.erl#L245-L254

They're created by the compiler provider in rebar3 when running against a given app or module, to track changes and dependencies across files. This allows, for example, to rebuild a module when one of its included files or parse transforms is modified, without having to rebuild the whole application each time.

The files are in _build, and we expect the ability to store things in there, because it would not necessarily be obvious where else they could be stored (maybe in a parent .dag directory still within _build?) We would consider it a bug if these files ended up in a release directory or a tarball, though.

[tsloughter]

I'd like to see if the performance improves (not that I know of it being an issue with these) by having just 1 dag and that would be stored at the top of each profile. But I've never gotten around to trying it.

[maxlapshin]

Ok, got the idea.

So we will just make our private fork of rebar with port compiling and without dag.

[tsloughter]

Curious how you are using the build artifacts in _build that you need to not have the dag. And are you planning to simply have rebar3 recompile all modules on each compile, accept it won't recompile related modules (like if a hrl or parse transform changes it won't recompile the modules that are using it) or planning something else?

[maxlapshin]

Right now I have a combination of Emakefile for fast recompile, rebar2 for port compile and .app.src and our own tool for building our production version (combination of rebar2 + relx).

I want to replace rebar2 + our tool with something more standard, but seems that I need to deeply patch rebar3 to make what I need.

It is ok to recompile everything some rarely (it takes less then a minute) and ok to use it with combination of Emakefile for make:all([load]).

[maxlapshin]

I've understood my misunderstanding.

I consider ./rebar3 as prod compile as a mechanism to build resulting output that I'm going to pack to production. This is why I do not want to see in _build/prod any intermediate files.

It seems to be not as rebar3 is designed to work and resulting output should be made via relx.

The same situation is with .o files when I compile nif. I like the idea to remove build artifacts from source tree, but where to put them? .o and .d files is the same thing as source.dag

They are intermediate build artifacts that are not required for result, but required for build.

[tsloughter]

Right, when creating a release or an escript those dags are not included in either. Or publishing a hex package, which publishes source anyway. So usage of a built app from _build is expected to copy out what is needed.

[maxlapshin]

I've made following changes in my rebar3 fork: I'm now compiling to _build/default+intermediate following files: .o .d source.dag and the rest of non-final artifacts.

Now all seems to be ok.

How do you think: is such merge request valuable?

[ferd]

I understand that you're copying the prod/ subdirectory and putting it directly to production, but since you're (seemingly) not using releases, you're currently falling a bit outside of the expected pattern we have for the tool. Currently, we do expect to be able to do whatever we need to do in _build/.

I think we first need to understand why these files are not acceptable as they are. The implication is that you don't expect us to ever store anything but final data in these directories whereas we consider that the release process is what does the required cleanup for a prod build. That has broader implications than just getting the digraphs out of there, since it implies we structure the tool's expectations differently.

Are the files not filterable, no white- or black-list possible? How are they getting copied?

[maxlapshin]

Yes, I do not expect to see anything new in output.

Rebar once changed its default from no_debug_info to debug_info and it was a VERY serious problem for us.

This is why it is unacceptable for me to see any kind of unexpected files in output build.

Yes, we do not use releases, we have much more complicated boot process than it is possible to do with release.

./rebar3 as prod compile works very good for me now after I made some hacks to add intermediate build and ported back port_compiler inside it. I want to run compilation only once, because it is rather expensive (by cpu and time) and we make packages and run tests for each push in CI. This is why I just copy _build/prod to deb package.

[ferd]

What you consider the output build, we consider rebar3's workspace. I think that's the problem here. You're suggesting that we fundamentally pick a different workspace to align with the one you want (old rebar did not even have such a thing, it just put everything in deps/ and in ebin/). I'm not sure it makes sense to openly make it a contract to support pristine library directories and decide to put rebar3's intermediary work elsewhere. That's the whole point of what is going in there already and chances are we'd just have to make one more copy of all the libs to accommodate that. It seems more straightforward from our point of view to make a copy for prod that only extracts what it needs, rather than making a copy for all the other use cases, in case someone wants to use the current working copy for prod while we also support releases for prod.

We'll also not be bringing port_compiler in, because neither Tristan or myself are willing to support C compilation (we don't personally make enough use of it), and nobody has stepped up to maintain it in its current form -- every plugin maintainer has stuck around for a few months at most, and that's not sustainable long term. It will remain a plugin for the foreseeable future as long as it can't be maintained and stable.

[maxlapshin]

Ok, got it.

In any case, thanks for the tool. It's recompiling speed is really cool.

[ferd]

That being said, if performance is an issue and we can have access to some sample applications to see how it's being used, it'd be useful for us to optimize that more directly. Unfortunately, most of the people who complained about performance in the past had private code with private uses cases we could never optimize for.