New build system API: collecting thoughts, ideas and wishes

[ghost]

We would like to come up with a new clean and well defined build system API. The main reason we want to do this now is to help the migration from Jenga to Dune inside Jane Street. Indeed, we would like to come up with a new API that we would implement in both Jenga and Dune, making it easier for the build system team at Jane Street to share rules between the two systems.

Moreover, we want to think of this API not just as helping for the migration but also as a new ideal API that we would eventually be happy to open to users when we are ready to make Dune extensible. At this point we should have enough experience with build systems to design a solid API that will both be easy to reason about and still give enough power to the build system to perform fast and correct builds.

This ticket is to collect thoughts, ideas and wishes on such a new API. Please throw in any idea you have, or anything you always wished a build system API would provide!

[smondet]

Maybe all of this is out of scope:

I hacked together and use https://opam.ocaml.org/packages/febusy/ — I'd be happy for it to be obsoleted by a dune-library with a nice API :)

• I've read about the applicative Vs monad part; but it would be nice if that library would still have a monadic API to treat build-artifacts directly in OCaml (maybe with degraded performance?)
• It'd be nice to build other things than unix files, like arbitrary OCaml values (marshaled for inter-run persistence but still faster / in-memory within a given process).
• Provinding special “ways of being a dependency” would be cool too: deps that are always rebuilt, deps that are built once-per-process, deps that are built at most once-per-Time.Span.t, etc.

[ghost]

Thanks @smondet! This looks interesting, we'll have a look at febusy. The second point is looking similar to the memoization system we have, though we don't have persistence (yet).

[mjambon]

From my recent experience:

• Generating dune files directly is a breeze.
• I see some features becoming deprecated in favor of new ones. I imagine it would be handy to have a tool for migrating a whole project from an older version of dune to a newer one automatically (wherever possible, otherwise emit some sort of to-do list for the user).
• As you happen to know, dune-deps uses a crude interpretation of dune files using sexplib for parsing. This works well enough given the scope of this application. It would be nice to have a library that scans a project and returns a typed representation of the dune project, as long as browsing that tree is straightforward.

I'm not sure what it says about a public dune API... I just wanted to share some feedback.

[ghost]

Thanks for the feedback @mjambon :) Regarding migration, we have dune upgrade at the moment which helps migrating from jbuilder or dune 1.x to dune 2.x.

Regarding extracting info from dune projects for other applications, the direction we are currently following is to add a versioned RPC to Dune. That will allow to continue evolving Dune features while keeping a stable and simple interface for third-party applications.

[rgrinberg]

As you happen to know, dune-deps uses a crude interpretation of dune files using sexplib for parsing. This works well enough given the scope of this application. It would be nice to have a library that scans a project and returns a typed representation of the dune project, as long as browsing that tree is straightforward.

I believe we have something like that with $ dune describe