[RFC] generating opam files

[ghost]

This ticket describe a proposal to let dune generate the opam files. They are several goals for such a project:

1. one less kinds of configuration files to write and maintain for users
2. so that we can reuse the same information for various packaging systems
3. so that we don't need to specify package dependencies manually, since all the necessary information is already encoded in dune files

While 3 is feasible, it is a lot of work and can be considered as a long term goal. This proposal only covers 1 and 2, which is already an improvement, especially as many fields of opam fields are now always the same.

Example

Let's take the current opam file of utop:

opam-version: "1.2"
maintainer: "jeremie@dimino.org"
authors: ["Jérémie Dimino"]
license: "BSD3"
homepage: "https://github.com/ocaml-community/utop"
bug-reports: "https://github.com/ocaml-community/utop/issues"
dev-repo: "https://github.com/ocaml-community/utop.git"
build: [
  ["jbuilder" "subst"] {pinned}
  ["jbuilder" "build" "-p" name "-j" jobs]
]
depends: [
  "base-unix"
  "base-threads"
  "ocamlfind"    {>= "1.7.2"}
  "lambda-term"  {>= "1.2"}
  "lwt"
  "lwt_react"
  "camomile"
  "react"        {>= "1.0.0"}
  "cppo"         {build & >= "1.1.2"}
  "jbuilder"     {build & >= "1.0+beta9"}
]
build-test: [["jbu`git add` ilder" "runtest" "-p" name "-j" jobs]]
available: [ocaml-version >= "4.02.3"]

and utop.descr file:

Universal toplevel for OCaml

utop is an improved toplevel (i.e., Read-Eval-Print Loop or REPL) for
OCaml.  It can run in a terminal or in Emacs. It supports line
edition, history, real-time and context sensitive completion, colors,
and more.  It integrates with the Tuareg mode in Emacs.

The goal is to have only this dune-project file instead:

(lang dune 1.X)
(using opam 1.0)
(github_project ocaml-community/utop)
(license BSD3)
(authors ("Jérémie Dimino" jeremie@dimino.org))

(description "\
Universal toplevel for OCaml

utop is an improved toplevel (i.e., Read-Eval-Print Loop or REPL) for
OCaml.  It can run in a terminal or in Emacs. It supports line
edition, history, real-time and context sensitive completion, colors,
and more.  It integrates with the Tuareg mode in Emacs.
")

(depends
 base-unix
 base-threads
 (ocamlfind (>= 1.7.2))
 (lambda-term (>= 1.7.2))
 lwt
 lwt_react
 camomile
 (react (>= "1.0.0"))
 (cppo build (>=1.1.2))
 (ocaml (>= 4.02.3)))

Then, because of (using opam 1.0), dune would automatically generate the utop.opam and utop.descr files in the source tree, and possibly even commit them.

Notes

A couple of the fields in the dune-project file above such as license and description are pure informative fields, however for all the other ones, dune could reuse the information. For instance the constraint on ocaml could be used to automatically decide what opam switches to test against when a special command line flag is passed. The name of the project, homepage, ... would be derived from (github_project ocaml-community/utop).

[emillon]

Dune is able to deal with several opam packages, with different metadata, so I'm not sure that dune-project is the best place to handle this (for single-opam projects, it's OK of course).

[ghost]

For multiple package we can simply group things inside (package ...) stanzas:

(package foo
 (depends ...))

(package bar
 (depends ...)
 (description ...))

Missing fields would be inherited from the global ones.

[avsm]

I think that putting this information in dune-project is quite important, since it means that we can adopt a different generation strategy for other non-opam packaging formats. For example, creating deb packages would require partitioning out the bytecode and native code files separately, and creating a different "cut" of packages (e.g. lwt subpackages might be folded in).

In the longer term, I can see dune-project being the place where larger projects can package up codebases in a way that is suitable for different upstream requirements. So it might be interesting to also write down a syntax for one other format (perhaps esy or rpm or deb) to see what the metadata differences look like. We could immediately generate Ubuntu PPAs if we generated Deb's easily, and interoperate with system packages...

[jberdine]

My understanding of the current proposal is that the opam build entry is hardcoded to essentially "dune" "build" "-p" name "-j" jobs, right? Is that intended? I don't know what fraction of packages can get away with just that, but it's certainly not all of them.

[avsm]

There are indeed a few packages that need to run a manual configure phase first. I'm pretty sure we can do away with those in a future revision of Dune, but it's a non-zero number at the moment.

[ghost]

That's intended yes. When these steps are necessary, it tends to break the composability aspect of Dune, which isn't good. I also believe that we can extend Dune so that such extra steps are not necessary. it just means that in the meantime such packages won't be able to use (using opam 1.0) in their dune-project file. If that's a problem we can always add a mechanism to keep a few fields under the control of the user.

[emillon]

So it might be interesting to also write down a syntax for one other format (perhaps esy or rpm or deb) to see what the metadata differences look like.

In the context of Debian, it seems difficult to use the same metadata, at least for official packages. The version numbers might be different, there might be a distinction between source and binary packages, etc. Probably this will be handled by the distribution itself using dh_ocaml or similar tools.

[jordwalke]

I think that putting this information in dune-project is quite important, since it means that we can adopt a different generation strategy for other non-opam packaging formats.

I do agree. But I think it is slightly better to have a build-system agnostic configuration at the top level, but one that can be extended by build systems and any other tool imaginable. Just as an example, we have package.json at the top of our projects, but we can store any fields there that build systems or dev tools (like merlin) could look for. I wouldn't want to call that file a merlin-project.json, or dune.json - it made more sense for it to be called package.json and I would have rather it been even more generic (project.json). Is the dune-project proposed here kind of like opam but in sexp? If so would it make more sense to think of it as sexp_of_opam?

[ghost]

Just as an example, we have package.json at the top of our projects, but we can store any fields there that build systems or dev tools (like merlin) could look for.

That's basically the idea of dune-project: personally, I want dune-project to be the only configuration file I have to write and then let dune generate everything else. For instance I was thinking of adding support for generating a package.json file so that we can use dune projects with esy out of the box.

[andreypopp]

For instance I was thinking of adding support for generating a package.json file so that we can use dune projects with esy out of the box.

The common workflow with esy and dune is to define dune package as a dependency and then make dune available in the esy environment. This means dune isn't available w/o esy in such case.

I think it makes more sense to add to esy an ability to read project metadata directly from dune-project files (similarly how it does read it from package.json or *.opam files).

[ghost]

I think it makes more sense to add to esy an ability to read project metadata directly from dune-project files (similarly how it does read it from package.json or *.opam files).

That would be awesome! Dune can provide an API to easily read dune-project files

[jordwalke]

Yeah, those are cool ideas too. I would mention though that the ideal project descriptor that we encourage people to use would not sound as if it's specific to any one technology. I liked that npm chose the package.json name because technically it's not only used by the package manager but also build tooling to configure things. That's one the reasons (not all the reasons) why we encourage a package.json file. If you wanted something similar, I would suggest naming it something more generic like project.sexp (even npm doesn't include the word "npm" in package.json). Sad thing about node_modules though!

[andreypopp]

Agree about choice of the name.

Also it would be nice if dependencies declared there can be fully qualified as from which source they come:

(depends
 (opam lwt)
 (opam lwt_react >= 1.0)
 (github user/repo beta)
  ...
)

[rgrinberg]

I'm not sure the above belongs to dune project files. Specifying the dependencies which a dev used to build a package on his own box is a property of the workspace rather than the project. Users of the project are quite likely to want to fetch dependencies from different sources. The resolution of dependencies isn't something that we should hard code in a project file.

[andreypopp]

It's not resolution of dependencies but an ability to specify dependency which is hosted on github. Instead of github it could be (npm reason >=3.0.0) and so on.

My point was that it shouldn't be opam specific if dune-package is going to support various packaging systems as stated in the motivation for this issue.

[ghost]

The idea is that the names we write in the dune-project file are abstract package names. These abstract names can them be mapped to real packages by using one or more package managers such as opam, npm, debian, ...

For instance, if package lwt exists in both opam and npm, is there any reason to prefer one of the two? If not, it seems to me that the precise source of the package shouldn't appear in the dune-project file.

That said, it would indeed be nice to allow to provide a precise source for projects that are not yet released.

[rgrinberg]

This will be released in 1.10. There are still some features missing, such as validating the depends fields, but the proposal is mostly implemented.