On-demand build of imported source files

[nordlow]

dub finds all *.d files and passes them to the compiler, whether any other code uses that module or not. This is in contrast to dmd -i, which only causes the compiler to look at a D source file if it's actually imported.

There might be cases where dub could benefit from such a behavior via a flag. At least when building applications,

[WebFreak001]

should probably only do this for executable and unittest targets

[Geod24]

Definitely not for unittests. We have modules that are just unittest modules and aren't imported by anyone.

[ljmf00]

What about files with static this ctors? Executables can rely on modules with those and as far as I'm understanding, this will make those modules not behave as intended.

[Geod24]

That's also a concern. Although this style of coding is not prevalent AFAIK. I think we should sample the existing ecosystem to see what people need / go to naturally.

In my case, we introduced the mainSourceFile change (that has drawn some controversy) because our main project is:

• An application (server);
• With some tools (a config file validator, a small client);
• Has a complex unittest structure (we have some "tools" that are generated to check our C++ bindings);
• And is used as a library by other programs;

This we solved with:

• Having the server configuration first and having some excludes (https://github.com/bosagora/agora/blob/ab158eadaf30aa6bb8e335a09921365af909cb98/dub.json#L41-L55)
• Having a ridiculously long exclusion list for the client (https://github.com/bosagora/agora/blob/ab158eadaf30aa6bb8e335a09921365af909cb98/dub.json#L56-L83)
• Using targetType: none for the "library": https://github.com/bosagora/agora/blob/ab158eadaf30aa6bb8e335a09921365af909cb98/dub.json#L102-L105

The last two solutions are hacks. The targetType, especially, forces us to essentially bypass dub when using it as a library.

So I fully support having a way to have the -i behavior. I'm wondering if, after a transition period, it should even be made the default. But as mentioned, we also have unittest-only modules which no one import.

[WebFreak001]

aren't static this ctors only called when their module is imported and something from the module is used in some way?

[nordlow]

aren't static this ctors only called when their module is imported and something from the module is used in some way?

According to https://dlang.org/spec/module.html#staticorder they run are run regardless of being imported or not.

[rikkimax]

Given that this has the potential to break people's code (i.e. self-registering routes), has anyone run a profiler on dub to see where it actually spends its time as part of slowing down compilation?

[ljmf00]

Given that this has the potential to break people's code (i.e. self-registering routes), has anyone run a profiler on dub to see where it actually spends its time as part of slowing down compilation?

This can benefit from huge libraries like vibe-d. If someone only uses 2 or 3 files and they are statically linked, there is no need to compile the whole thing as the unused symbols will get discarded anyway.

aren't static this ctors only called when their module is imported and something from the module is used in some way?

According to dlang.org/spec/module.html#staticorder they run are run regardless of being imported or not.

I'm not completely sure if that is the current behaviour, although.

[WebFreak001]

possible feature for dud?

[nordlow]

possible feature for dud?

Ping, @burner.

[burner]

I'm not quite sure if I really understand the purpose/benefit.

Could somebody please add an issue to dud's github https://github.com/symmetryinvestments/dud linking back to this one here

[nordlow]

Could somebody please add an issue to dud's github https://github.com/symmetryinvestments/dud linking back to this one here

dmd -i <set_of_initial_d_files> only compiles the files that set_of_initial_d_files actually imports. This cannot currently be expressed by dub. I'm not sure if such a behaviour would be relevant to have in dud.

[burner]

hm, you can exclude and include specific files in the dub config can't you

[Geod24]

You can but it's a pain, if you want to have multiple binaries in the same package. Using subpackages is a pain, so in my experience people tend to reach for configurations instead. Problem is, dub will compile everything in source, so if you add a new source file, you have to exclude it in all your configurations. Not really scalable. See my comment above for an example of such usage.

[burner]

playing devils advocate here.

sounds like you want variables and basic set operations to create an array of the files you want to exclude and then say something like sourceFiles / YOU_VARIABLE HERE

[Geod24]

I want recursive exclusion, so I can say "exclude source/myAppName/". Currently I need to say "exclude source/myAppName/* but also source/myAppName/*/*", etc... But even this has issues, because of when globbing is performed. See for example https://github.com/dlang/dub/issues/2142

Although, even if exclusion / inclusion was smarter (it should be), excluding the same pattern for every other configuration would still be annoying boilerplate. Acceptable boilerplate, but boilerplate nonetheless.

[burner]

Looking at all the special things I have come across in regards to build file configuration languages an approach similar to what reggae does makes sense, aka. write a D program that creates the actual "makefile".

Making this api a first class citizen in dud is likely the better idea. In the sense that when

1. dud init does no longer work
2. you edit du(b|d).json as currently defined
3. if that fails you write a program that calls libdud