First steps to make the Makefile backend bootstrap again.

[samoht]

Support for dynamic inter-module dependency discovering. A bit hackish, but not too much actually. With that patch, the library bootstrap again, yay!

The current scheme to turn our pretty applicative DSL into a monadic monstruosity:

• add a <dir>.prepare step, which goal is to symlink all the source files into the build dir . This need some kind of convolutions and is done partly in the backend, as Assemblage has only access to the reverse information in the project description. But the current solution is not too bad and generic enough (I think).
• to compile a ml or mli your first need to build the corresponding ml-deps or mli-dep (very similar to ocamlbuild, ocp-build and other).
• to generate a ml-deps or mli-deps you must first prepare all the include directories (ie. .prepare). This ensure that ocamldep has all the relevant information to compute correct dynamic dependencies.
• And finally, the final trick is to -include all the ml-deps and mli-deps to lazily add the new depenencies infered by ocamldep.

This is all (relatively) standard in the Makefile world and it seems to work fine on my relatively limited testing.

[dbuenzli]

I'm very impressed I thought it would be much more painful than that. Thanks for tying the knot on the bootstrap. That being said we still see that our model doesn't really capture what is needed and I wonder if we should not embed a monadic build DSL in the applicative configuration DSL.

[dbuenzli]

Travis failure is due to opam not installing the latest version of cmdliner.

[samoht]

if we should not embed a monadic build DSL in the applicative configuration DSL.

yup, I was thinking to something similar. Would be nice to have a proper model to compile our applicate DSL too. Not sure we really want the full power of monadic, though, and not sure that's really useful for now on (as we have only one semi-working backend).

[samoht]

Travis failure is due to opam not installing the latest version of cmdliner.

Right I'll fix the Travis scripts to add the right dependency in an other PR.

[dbuenzli]

Would be nice to have a proper model to compile our applicate DSL too.

Note sure what you exactly mean here. Basically what we have is a statically known list of parts. The availability of parts and their build artefacts (inferred from the rules) depend on the configuration. The applicative DSL allows you to express in which configurations a part is available and with which rules (and hence which build artefacts exist in a configuration).

Note that the fact that the list of parts is static should be carefully considered (see #135), it does have advantages like being able to list the part without having a proper configuration in hand; but it also limits the system. For example suppose we have a module relational mapper that generates on compilation unit per table from a data base description, we can't express that. For this reason you also need to list all your compilation units by hand and can't just say look into that directory (which is quite handy when you are in the initial phases of a project where you rename everything every two minutes).

Not sure we really want the full power of monadic, though, and not sure that's really useful for now on (as we have only one semi-working backend).

Maybe not. But somehow I'd still like to be able to express the logic in assemblage itself, rather than having to hack around the problem. Meanwhile I'm curious to see if your model can cope with C deps, see #144.

[samoht]

For this reason you also need to list all your compilation units by hand and can't just say look into that directory (which is quite handy when you are in the initial phases of a project where you rename everything every two minutes).

I think we can still do that with our model, using something like:

open Assemblage
let lib_of_dir ?f dir =
  let f = match f with None -> (fun x -> Some (unit x)) | Some f -> f in
  Sys.readdir dir
  |> List.fold_left (fun units x -> match f x with None -> units | Some u -> u :: units) []
  |> lib dir

[samoht]

Would be nice to have a proper model to compile our applicate DSL too.

Note sure what you exactly mean here.

Note sure either :-) I agree that we need a model of the build system (that we don't have yet) and maybe that model is fully monadic. But I'd be more happy with a static knowledge of targets and a dynamic dependency discovery.

Regarding applicative vs. monadic. I still believe that our model is the right one :p we can already express a lot of different things (sometime it is a little bit convoluted) and we can do a lot of static pre-computation which is nice. But I agree that it's still a little bit too early to be completely sure ...

[dbuenzli]

I think we can still do that with our model, using something like: [...]

This is highly unsatisfactory ! E.g. it won't detect new additions or suppressions in the directory. You'll have to resetup which is both painful and retarded. We need to be able to express dependency on that directory contents. Again I'll point to tup here which does correctly handle these things.

There is something to untie or tie, we have a notion of (statically tracked) dependency in the configuration language and a notion of dependency on in the build model (and the build model is embedded in the configuration language). Now when we setup a configuration we generate a build system (under the form of a set of build actions), if I change a configuration key not all build actions change so only the products that depend on build actions that changed should actually be rebuild. So in some sense configuration keys are simply the roots of the dependency graph that includes build actions.

I agree that we need a model of the build system (that we don't have yet)

What what ? Of course we have a build system model it's described right here and it's unsatisfactory (basically ~inputs and ~outputs are statically determined at configuration time).

The more I think about this the more I think we have the wrong API...

[dbuenzli]

So in some sense configuration keys are simply the roots of the dependency graph that includes build actions.

@samoht, just to expand a little bit on that.

This is the reason why I think we should move away from the retarded Makefile backend/model and have a default driver that provides a tup-like experience via an assemblage build command (that could maybe initially, partially be implemented by a Makefile). For me the setup command is redundant. I want a single command which is build. Setting up a configuration is just building the configuration (by which I mean assigning values to configuration keys) there's no need to have two commands. If you directly issue an assemblage build after a checkout it simply uses the configuration key's default value to build.

So doing an assemblage setup --debug-key=true && assemblage build should be equivalent to assemblage build --debug-key=true. The semantics of the later is set the configuration key debug to true and rebuild whatever products depends on that configuration key (subsequent builds will have --debug=true set, until you change the value) which we know, thanks to the applicative configuration DSL that tracks those. From a backend implementation perspective by a file based build system the idea you can see assemblage --debug-key=true as touching a file _build/_config/debug file and build products that result from an action that uses the debug key need to depend on that file.

Once we have that, we should have assemblage switch that allows us to live with different build configurations without having to constantly rebuild everything from scratch...

[samoht]

This is highly unsatisfactory !

I knew you'd like it :-)

Of course we have a build system model it's described right here and it's unsatisfactory The more I think about this the more I think we have the wrong API...

I see your point and I kind of agree (although I really would like to keep a simple API for simple use-cases). I'll think about it more ...

[dbuenzli]

I see your point and I kind of agree (although I really would like to keep a simple API for simple use-cases).

I think API simplicity for simple use cases is orthogonal and can always be provided --- casual users should only use and compose parts. In any case I think that what we have now may not be the final word but is a first good step and we should try to see up to where we can live with it from the perspective of my last message; the cracks will maybe show us the way.