Multiple public libraries in a package

[ezyang]

Motivation. A common pattern with large scale Haskell projects is to have a large number of tighty-coupled packages that are released in lockstep. One notable example is amazonka; as pointed out in https://github.com/haskell/cabal/issues/4155#issuecomment-270126748 every release involves the lockstep release of 89 packages. Here, the tension between the two uses of packages are clearly on display:

1. A package is a unit of code, that can be built independently. amazonka is split into lots of small packages instead of one monolithic package so that end-users can pick and choose what code they actually depend on, rather than bringing one gigantic, mega-library as a dependency of the library.

2. A package is the mechanism for distribution, something that is ascribed a version, author, etc. amazonka is a tightly coupled series of libraries with a common author, and so it makes sense that they want to be distributed together.

The concerns of (1) have overriden the concerns of (2): amazonka is split into small packages which is nice for end-users, but means that the package maintainer needs to upload 89 packages whenever they need to do a new version.

The way to solve this problem is to split apart (1) and (2) into different units. The package should remain the mechanism for distribution, but a package itself should contain multiple libraries, which are independent units of code that can be built separately.

In the Cabal 1.25 cycle, we've added two features which have steadily moved in the direction of making multiple public libraries possible:

• Convenience libraries (#269) mean that we already have Cabal-file level syntax support for defining multiple libraries. Note that these libraries are only accessible inside the package, so to a certain extent, the only thing that would need to be changed is making it possible to refer to these libraries.

• Per-component build (#3064) makes it easy to build each internal library of a package separately, without having to reconfigure a package and then rebuild. This means that, from the perspective of new-build, building multiple libraries from a package is "just as separate" as building multiple packages, and I imagine Stack would be interested in taking advantage of this Setup.hs feature.

So the time is ripe for multiple public libraries.

Proposal. First off, I want to say that for the 2.0 release cycle, I do not think we should add support for the feature below. However, what I do want to do is make sure that the design for convenience libraries (which is new) is forwards compatible with this (see also #4155)

We propose the following syntactic extensions to a Cabal file:

1. First, build-depends shall accept the form pkgname:libname whereever pkgname was previously accepted. Thus, the following syntax is now supported:

build-depends: amazonka:appstream, amazonka

amazonka refers to the "public" library of amazonka (the contents of the library stanza with no name), while amazonka:appstream refers to library appstream inside amazonka. A version range associated with sub-library dependency is a version constraint on the package containing that dependency; e.g., amazonka:appstream >= 2.0 will force us to pick a version of the amazonka package that is greater than or equal to 2.0.

2. We add a new field to library stanzas, public, which indicates whether or not the library is available to be depended upon. By default, sub-libraries are NOT public.

NEXT, we need the following modifications to the Setup.hs interface:

The --dependency flag previously took pkgname=componentid; we now augment this to accept strings of the form pkgname:libname=componentid, specifying what component should be used for the libname of pkgname.

Explanation. The primary problem is determining a syntax and semantics for dependencies on sub-libraries of a package. This is actually a bit of a tricky problem, because the build-depends field historically serves two purposes: (1) it specifies what libraries are brought into scope, and (2) it specifies version constraints on the packages that we want to bring in. The obvious syntax (using a colon separator between package name and library name) is something like this:

build-depends: amazonka:appstream, amazonka:elb

But we now have to consider: what is the semantics of a version-range applied to one of these internal libraries? E.g., as in:

build-depends: amazonka:appstream >= 2.0,
               amazonka:elb >= 3.0

Does having separate version ranges even make sense? Because the point of putting all libraries in the same package is to ensure that they are tightly coupled, it doesn't make sense to consider the version range on a library; only on a package. So the build-depends above should be considered as levying the combined constraint >= 2.0 && >= 3.0 to the amazonka package as a whole.

This causes a "syntax" problem where, if you want to depend only on sub-libraries of a package, there is no obvious place to put the version bound on the entire package itself. One way to solve this problem is to add support for the following syntax amazonka:{appstream,elb} >= 2.0; now there is an obvious place to put the version range.

Downsides. Prior to solving #3732, there will be some loss of expressivity if a number of packages are combined into a single package: cabal-install's dependency solver will solve for the dependencies of ALL the libraries (even if you're only actually interested in using some of them.) This is because the solver always solves for all the components of a package, whereas it won't solve for the dependencies of a package that you don't depend on.

Prior art. This is a redux of https://github.com/haskell/cabal/issues/2716 Here is what has changed since then:

1. The motivation has been substantially improved; last time the motivation involved some Backpack/internal code readability hand-waving; now we specifically identify some existing, lockstep packages which would benefit from this. The proposal has nothing to do with Backpack and stands alone.

2. The previous proposal suggested use of dashes for namespace separation; this proposal uses colons, and the fact that the package must be explicitly specified in build-depends means that it is easy to translate a new-style build-depends into an old-style list of Dependency, which means tooling keeps working.

3. The previous proposal attempted to be backwards compatible. This proposal is not: you'll need a sufficiently recent version of Cabal library to work with it.

CC @mgsloan, @snoyberg, @hvr, @Ericson2314, @23Skidoo, @dcoutts, @edsko

[fgaz]

5526 is in! :tada:

, but please remember that in the final 3.0 release, sublibraries will NOT be public by default: you won't be able to depend on them from outside of the package unless you put visibility: public in the sublibrary stanza.