A radical proposal: Unify CommonML and obuild

[jordwalke]

Hi!

I started building CommonML, a build system for ML programs a while ago - before I even knew that obuild existed:

They are similar in spirit: Create a build system that that is opinionated (doesn't aim to build every kind of project under the sun), automatically namespaces, and generally "just works" if you structure your project in a sane way.

I do like when various competing implementations/ideas can thrive and explore their own directions independently, but I find that that works best when there is already many developer resources available. Since so few people are actually making attempts (obuild and CommonML) and neither of these are our full time jobs, I'm curious if there's a way we could unify our efforts. There are some things I like about obuild and some things I like about CommonML. I'll list some of them.

• obuild builds c stubs.
• obuild has a really nice command line interface clean/install etc.
• obuild has the notion of building a "library" - whereas CommonML rebuilds every dependency into your result statically.
• obuild is written in OCaml which is generally better than JavaScript, especially for high performance build tooling. CommonML is written in JS, which is really easy to hack quickly, but the result doesn't perform as well.
• CommonML has a --compileJS=true flag making js_of_ocaml compilation dirt simple.
• CommonML uses npm package.json as both the way to model dependencies on sublibraries as well as create namespaces. This is the biggest advantage because npm is so incredibly popular and easy to use for local development.
• CommonML automatically creates your .merlin autocomplete files for your project and all your dependencies.
• CommonML will be getting some really beautiful error pretty printing soon.

But there's some things both of these projects lack. For example, the ability begin debugging your program in ocamldebug with "one click". ocamldebug is actually really great, but no one uses it because it's too hard to get set up with module paths etc. This is the job of a build system! However, I'd rather not have two projects invent that same wheel - and a bunch of other wheels that are to come.

Any thoughts on a long term strategy? If obuild implements the features of CommonML, I would gladly switch to obuild and redirect people to it. I don't care about how it actually happens - CommonML was merely built out of necessity - it's not the end goal in and of itself, so I would happily lay it down if something better comes along. Could some of the contributors to obuild check out CommonML to see how close it is to their vision?

[jeromemaloberti]

Hi,

There is definitely a lot of overlap on both projects, but it seems to me that the end goal is different. I may change my mind later, but I see obuild as a single project builder that integrate in a larger distribution like opam. Among the common features that I want to implement:

• js_of_ocaml support, I don't know how yet, though.
• .merlin generation is almost trivial, and .opam file generation is almost complete.
• if I can understand what need to be generated, I definitely want on top of my priority the ocamldebug file. If you have an example, that would be very helpful.
• There are new features in ocaml that I want implemented soon enough, ppx, namespace.
• From an implementation point of view, I want obuild to become a library + front-end, so other programs/converters can be trivially written (rebuild triggered by inotify, bridge with assemblage).
• support for popular external tools, menhir, protobuffer ?, thrift?, etc.
• generation of interfaces (ocaml -i)
• finish the implementation of all features (doc generation, etc.)
• and obviously, a nice cli + good error messages. I don't know npm package.json, but I would expect it to be relatively easy to convert to obuild files. When obuild will be a library, it may be just another front-end though. What I don't plan to implement (but on this too I can change my mind) is the downloading of the sources (via git or any other mean). It seems to me that CommonML is a builder and deployment tool that could be built on top of obuild (as a library or a binary), by implementing the parsing of packages.json, and downloading the sources before calling obuild. I will definitely check out CommonML soon.

Cheers

[jordwalke]

You should check it out. CommonML could probably be built on obuild if obuild let you control the namespacing of modules. For example, CommonML packages up namespaces not by directory but by package. So each package.json defines a single compilation package that can depend on other packages. Everything inside of its src directory is in one package namespace (as opposed to nested folders creating packs). Also, CommonML doesn't use packs, it uses module aliases to perform namespacing which I'm told benefits compilation time - and also is the only thing that merlin will really work well with when locating modules (packs confuse it).

CommonML is a hyper granular, sandboxed focused dependency manager - where globals are discouraged. I do think that once you're done with a CommonML build, it would be awesome to automatically turn it into an OPAM install/package - sounds like obuild will help out there.

[jordwalke]

There are new features in ocaml that I want implemented soon enough, ppx, namespace.

To me, the most important feature that CommonML supports is not the ppx feature, but the -pp feature. It's also kind of nuanced because you need to also supply the correct flags for -intf-suffix, -intf and -impl. You can look at CommonML to see how that is done.

[jordwalke]

Another feature that is really important, is the ability to pass the flag --forDebug=true and have it ensure that all the dependencies are rebuilt using the debug flag so that stack traces (and js source maps) are supported for every file.

[jordwalke]

What I don't plan to implement (but on this too I can change my mind) is the downloading of the sources (via git or any other mean). It seems to me that CommonML is a builder and deployment tool that could be built on top of obuild (as a library or a binary),

On that note, npm is hands down the most popular and easiest way to specify what exactly you want downloaded (either via npm or git) for your rapid, local, sandboxed development. Nothing else even comes close. That is, in fact, the only purpose of CommonML - to specify what you want - and then make sure that no matter what you depend on, everything is perfectly namespaced, autocomplete always works out of the box, and you can "one-click" compile it either to native, bytecode, or JS with source maps. Developers should only need to know how to use package.json specifications for dependencies, and then everything should just work. obuild might be a great way to build those dependencies once npm downloads them into your local sandbox (it just needs to support custom namespacing rules and generate the .merlin configs + docs).

[jordwalke]

Are you opposed to having obuild read build parameters from a package.json file? That would be a nice step towards making obuild and CommonML more similar so that there are fewer parts and would make a move to obuild more feasible.

[jordwalke]

Another important CommonML feature I forgot to mention is the fact that the package.json is used to form module visibility. So inside the package.json, there is an exports list where you can declare which modules are visible to other packages that depend on you. You can see every module internally, but your dependents can only see the ones that you declare exported. Similarly, you can see your dependencies' exported modules, but you can not see their dependencies' exported modules. You can't even see what they depend on, thankfully. This is accomplished via module aliases, and generating separate versions of the module aliases file. The one that is chosen depends on the visibility rule. You can search CommonML for -open to see how this is done.

[UnixJunkie]

.merlin automatic generation is interesting.

[jeromemaloberti]

Ok, let's try to answer to all the points:

• -pp is already supported for camlp4 in obuild, unless I'm misunderstanding your point.
• namespaces are only available in recent enough ocaml version, so packs are still useful. I don't know enough about namespaces, so can they really replace completely packs ? It is interesting anyway.
• I'm kind of worried about the integration of a CommonML package in opam. Mostly because packaging potentially multiple times your dependencies (even namespaced) may not play well with ocamlfind. Also, CommonML and OPAM are contradictory in term of goals. Opam shares the packages, CommonML isolates them.
• --forDebug=true is also implemented in obulid in -g.
• I am not opposed to read build parameters from a package.json, however it would add complexity and dependencies (unless you include a json library). Right now the parser is entangled with the project management, and that is not good. Obuild should be librarified, and other parsers such as package.json or assemblage, should be added externally to create easily obuild project structures. Are you opposed to use obuild as a library ?
• I'm not sure about the module visibility ... it doesn't seems too hard to implement though.

[UnixJunkie]

I'm closing this. If Facebook wants to bring manpower to the project, they are very welcome.