Disable subparts of a project

[AllanBlanchard]

Desired Behavior

I try to replace a configure file with dune-configurator. One important feature of our configure file is the ability to disable parts of the complete system:

$ ./configure --disable-foo

It prevents feature foo to be compiled, tested, and installed. Which is useful for both users and (mostly) developers.

We have the following architecture:

- dune project
- dune
- src/plugins/
    - p1
        - dune-project
        - dune
    - p2
        ...
    ...

Currently, when at least one plugin is disabled, the configure generates a dune file in src/plugins, that contains:

(include_subdirs no)
(data_only_dirs <disabled-plugins-directories>)

But, as far as i understand, it is not possible to do that with dune-configurator since we cannot generate a dune file that would be immediately used by the current build process.

Example

My initial idea was to generate a .enabled file in each plugin directory (each containing true or false), and to modify plugins dune file like this:

(library
 (name p1)
  (optional)
  (public_name proj-p1.core)
  (enabled_if %{read:.enabled}) ; << added line
  (libraries proj.kernel))

However, it happens that dune does not like it, and asks me to file this issue:

File "src/plugins/p1/dune", line 38, characters 14-30:
38 |   (enabled_if %{read:.enabled})
                   ^^^^^^^^^^^^^^^^
Error: Only architecture, system, model, os_type, ccomp_type, profile,
ocaml_version and context_name variables are allowed in this 'enabled_if'
field. If you think that read should also be allowed, please file an issue
about it.

Is there something we can do about it? Did I miss a simpler solution?

(FYI @bobot)

[emillon]

Hi, I don't have a complete answer or solution at the moment but I have some questions and initial remarks.

• How does "modularity" work in this software? I mean what does it mean to disable a part? Is it an actual plugin (as in a cmxs file) that is not being built? or is this just some parts that are not built? what is the reason for not building everything? (too many dependencies? too large?)
• how do you expect the configure step to look like? you mention that it's not possible to generate and use a dune file in a single step. that's correct. but it's not possible to pass --disable-x to dune, so this needs to happen in a separate step

Then I have some remarks:

• even though you mention dune-configurator, I don't think it's central to what you're trying to achieve here. The general pattern is that it's possible to generate parts of dune files using generators and include them in some places, but this is achievable without it. dune-configurator is just a useful helper to use in the generator scripts to do things like run the C compiler or call pkg-config. But if you're only using it to read CLI arguments and generate bits of dune files, you won't need dune-configurator.
• you're using (optional) together with (enabled_if). these are somehow competing features: (optional) asks dune not to fail if something failed while building this stanza (sort of), while (enabled_if) filters this stanza depending on a static-ish boolean. so my advice would be to drop the (optional) - if the condition in (enabled_if) is true, dune should fail loudly if it fails to build that component.
• even though dune prompted you to ask for (enabled_if %{read:...}), I believe that it's a feature that's a bit too powerful to be included. I think that it could create dependency cycles that are difficult to detect.

The actual solution will depend on what you expect and how this works (hence my questions above 😃), but my hunch is that you could split this work into several opam packages, attach the various bits to the right packages, and use -p / --only-packages when you want to build just subparts. But it's common to always build everything when developing, except things that are actually not buildable and thus filtered with a (enabled_if). You can find that pattern in the eio repo for example.

[AllanBlanchard]

How does "modularity" work in this software? I mean what does it mean to disable a part? Is it an actual plugin (as in a cmxs file) that is not being built? or is this just some parts that are not built? what is the reason for not building everything? (too many dependencies? too large?)

To give more details: the software is Frama-C. In the original build process, one could say "ok, I do not want the WP plugin", or "I do not need the GUI" during the configure step. In this case, the CMXS is not built, the build process just behave as if the code of the plugin was not there, same for the tests of the plugin. More precisely, in the configure step, there was two ways for a plugin to be removed from the build:

• either a dependency is missing
• or the user explicitly asks for this feature to be disabled.

For Frama-C developers, the main reason for not building a plugin sometimes is too make the build process faster during development and being able to say OK just run all available tests (thus all tests that do not depend on a disabled plugin) thus making the test process (a lot) faster. For users, it can be because they want to be sure that some plugins are disabled since they consider them not mature enough (it is less common for users to disable plugins though).

how do you expect the configure step to look like? you mention that it's not possible to generate and use a dune file in a single step. that's correct. but it's not possible to pass --disable-x to dune, so this needs to happen in a separate step

I try to proceed in an incremental way, migrating Frama-C to Dune has been quite a crazy work, now it has been integrated into our main branch so I try not to start another huge work on that. So at first, I was OK with the idea of having a configuration file, read during the "configuration" step, with a fallback rule where everything is enabled. To be honest the current solution (with the dune file at plugins root) works, I just feel like it is very hackish.

even though you mention dune-configurator, I don't think it's central to what you're trying to achieve here. [...]

Yes, now that I have spent more time working with it, I undestand that I do not necessarily need dune-configurator, even though I find it useful for other configure tasks.

you're using (optional) together with (enabled_if). these are somehow competing features: [...] if the condition in (enabled_if) is true, dune should fail loudly if it fails to build that component.

I understand your point. But I feel like it would require more work in our "configure" step without this. It is related to the two reasons why a plugin could be disabled: either a dependency is missing (reason why we have the optional) or someone asks to disable the plugin (reason why I wanted to add the read). Thus: the plugin is built is two conditions are true: the user wants it (or at least does not want to disable it) and all dependencies are available.

even though dune prompted you to ask for (enabled_if %{read:...}), I believe that it's a feature that's a bit too powerful to be included. I think that it could create dependency cycles that are difficult to detect.

Interesting, I do not see how because I am not familiar enough with Dune, but I trust you on that point.

my hunch is that you could split this work into several opam packages, attach the various bits to the right packages, and use -p / --only-packages when you want to build just subparts.

In fact there are already opam packages for all plugins. I think I should have a look to this option. I remember that I already had a discussion about it with @bobot but I do not remember the details. Nevertheless, we would probably need some scripts or make to use it :thinking: .

[emillon]

Thanks for the answers. I'll let @bobot take on this one since he'll have the context on both the dune and frama-c sides :)

[ivg]

We need exactly the same feature for BAP (which is not surprising as BAP was modeled after FramaC). We use oasis with ocamlbuild and omake backends. And our current implementation allows us to specify the set of components using the configure script, e.g.,

./configure --enable-{x86,llvm,frontend}

Since we have several hundred components, we also have a sort ordered set language of our own, so we can do,

./configure --enable-everything --disable-arm

It would be nice if dune build will allows us specify the set of packages to build. In fact, it will even nicer if dune will utilize the dependency information from the dune-project file, so that if I enable a certain subproject it will automatically enable its dependencies. Maybe, each project could define an alias, so we can do dune build @project:llvm @project:x86 @project:fronted and this will build the designated subprojects along with their dependencies.