Add support for project templates to dune init

[shonfeder]

This was part of the "level 2" plan for dune init, as noted in discussion around the originating issue https://github.com/ocaml/dune/issues/159#issuecomment-408586053

The continued desirability for this feature came up on discourse, most clearly articulated by mjambon. It was also discussed in the meeting earlier this week.

Some ideas:

1. A template can just be a directory with the intended project skeleton.
2. Templates should support variable substitution (for project names etc.). Probably using the same convention as dune subst?
3. Templates should work with existing command line arguments (E.g., dune init proj foo --inline-tests should add the inline_tests stanza to library components even if they are not in the template).
4. There should be a sensible default template for projects. This default is both a convenience and a form of "documentation as automation" suggesting minimal best practices.
5. It should be possible to use templates for the library, test and executable components.
6. It should be possible for users to set their own default templates via ~/.config/dune/config.
7. It should be possible to override any default templates via a command line argument --template foo (which could search for a directory named foo, and then fall back to a base name of a directory in the directory ~/.config/dune/templates.)

Proposed initial course of action:

1. Replace the existing hard-coded configuration for dune init proj with the default template. (Ideas (1) to (4))
2. Add support for overriding the default template. (Ideas (5) to (6))

Questions:

1. Do we need default templates for library, test, and executable components?
2. Where is the boarder between where we would want templates and where we would want plugins? (Initial thought: plugins are for dynamic behavior that musts be computed, a template is for automating static configuration boilerplate).

[rgrinberg]

It should be possible for users to set their own default templates via ~/.config/dune/config.

Could you expand on this a little more? Is the idea for the user to set preferences for existing templates? or is the goal to define new templates?

If it's the latter, then I would prefer that we first implement for defining public/private templates first. In other words, the same semantics we have for executables & libraries. I think only then we can start talking about allowing the user to have a store for private templates.

[shonfeder]

The latter. (I have not considered preferences for templates. Are preferences needed when you can just make a variation of a template to suite your needs?) The idea is, a user may always want projects to use base or containers or certain ppxs, or they may always want to two libraries, or certain kinds of testing boilerplate (or anything really), and they should be able to achieve all of this with a custom template.

I'm actually not clear on the public/private semantics for libraries and executables (which should help me understand). In the documentation, I can find two mentions that might be relevant. The first is about scope:

scope: a scope determines where private items are visible. Private items include libraries or binaries that will not be installed.

But I haven't found anything that explains how to declare an executable as private.

The other instance is in the private_modules field of the library stanza, but I don't see a corresponding field for executables.

Could you explain what would be the difference between a private template and a public template? In my understanding so far, a template is just a component (e.g., library, executable, project) skeleton, i.e. is some files with placeholders for string substitution -- it's not clear to me what a public/private semantics of such a template would be.

[rgrinberg]

But I haven't found anything that explains how to declare an executable as private.

A private executable is one where the public_name is omitted.

I haven't yet written down anything regarding defining/sharing templates. Let me write out some ideas I've had and they should answer most of your questions.

First, a template should be definable using the same construct we define other entities dune knows about - using stanzas. For example:

(template
 (name foo) ;; in conjunction with $ dune init foo
 (synopsis "short description here")
 (source dir/)) ;; dir/ would contain the template files to instantiate this template.

The above would define a private template. Such a template remains visible only in its scope, so to prevent identically named templates from colliding. I suppose we could add a way to qualify the scopes in such a situation.

For templates to be installed & shared, they need to have a public_name, and possibly be attached to a package. The idea is natural if you want templates to be distributed just like anything else. For example:

$ opam install opium
$ dune init opium.starter --name helloworld

[shonfeder]

Oh dope! Let me unpack a bit, to be sure I understand what you're proposing:

• (a) Templates would be first-class components[1]: they end up just like libraries and executables, except instead of building to object code in the _build directory they "build" to source code within the current scope (creating that scope if needed) -- of course with the caveat that they "build" via initialization.
• (b) Private templates would only be available within the scope they are defined in, whereas public templates get installed as part of a package.
• (c) We need a way of configuring packages so that they can register templates with dune when they are installed (probably just installing the templates in an appropriate directory?), presumably this happens via dune @install?
• (d) The inclusion of meta-data like the synopsis field suggests we'll need to extend the init command (or perhaps add a template command) to be able to query available templates.

I like the general shape of this a lot. Particularly, the idea that installed packages can be template-bearing: this opens the way for many "getting started" tutorials and "example" directories to be replaced with easy template generation. I think this would be a tremendous gain for accessibility in the ecosystem. :+1:

Assuming my understanding is correct, I have two followup questions:

1. The user story motivating private templates is not totally obvious to me -- I can vaguely guess at some possible scenarios, but a concrete example would help me here. Could you sketch out a case where a user would want a private template?
2. It seems like this plan envisions templates as a kind of sub-package. This seems to be the kind of thing a user of a package may want to only include optionally during install (I may want opium, but have no need for it's templates). So I wonder if we shouldn't require templates to be specified as a package, which can then be included in the depopts? Otherwise, might we risk having to implement an embedded package manager in dune? (Delegating this to the package manager might also save us the trouble of having to implement (c) ).

[1]: Is there a better term we use for a "configurable unit", i.e., that which is to be built according to the configuration of a stanza?

[rgrinberg]

The user story motivating private templates is not totally obvious to me -- I can vaguely guess at some possible scenarios, but a concrete example would help me here. Could you sketch out a case where a user would want a private template?

I agree that private templates aren't really useful. They're really more of an intermediate step to the full blown feature. In my mind, this can be implemented as a first step, and everything else can come in successive PR's.

It seems like this plan envisions templates as a kind of sub-package. This seems to be the kind of thing a user of a package may want to only include optionally during install (I may want opium, but have no need for it's templates)

Well this problem isn't unique to templates unfortunately. Packages are already magic boxes that can easily surprise you with extra libraries and executables that you don't really need or expect. I would say that we should just keep things uniform and address this problem down the road. At the very least, templates don't introduce dependencies on other packages so the footprint is minor in comparison to libraries & executables.

Otherwise, might we risk having to implement an embedded package manager in dune

Yeah, see the duniverse project everyone is aching for so badly :)

Is there a better term we use for a "configurable unit", i.e., that which is to be built according to the configuration of a stanza?

I'm not aware of one.

[shonfeder]

Thanks @rgrinberg. iiuc, the concrete steps can now be refined thus:

• [ ] Introduce a stanza allowing configuration of template components
• [ ] Add logic to instantiate defined templates
• [ ] Add logic to register installed templates with dune
• [ ] Add default templates for library, executable, test, and project components to the dune package, using the new stanza
• [ ] Adapt the init subcommand to initialize components using the above
• [ ] Add support for users configuring and specifying their own templates (or templates installed form other packages).

I should be able to get started on the first action item by next week.

[shonfeder]

Welp, 2020 went into a dark hole for me, and I'm only now emerging :)

In the interm, two quite fully featured approaches to project formatting have come on the scene:

• drom
• spin

I believe we decided that we're ok with offloading this task to external tools.

I'm closing this under that understanding. But if someone thinks this should still be open, please make some noise.

[tmattio]

I'm the author of Spin, the project generator mentioned above.

I had been following the development of dune init and was actually hoping that it would one day make tools like Spin obsolete. I think project generation is extremely important for newcomers and although I'm happy with the visibility Spin has had, it's clear to me it won't be able to bring as much value as Cargo or Mix's own project generation.

If it's not a priority, I'm happy to contribute it: There are a lot of overlaps between Spin's design and the design laid out in #159, so I think I'm in a good position to help on this.