Package variants: a trivial cross-compilation solution

[whitequark]

So I know there's https://github.com/ocaml/opam/issues/1536, however, that issue has too much bikeshedding for my taste. I've implemented cross-compilation for two platforms already (https://github.com/whitequark/opam-cross-windows and https://github.com/whitequark/opam-cross-android) and I think we don't need all that complexity (cross-switch dependencies etc). I have a much simpler proposal.

Let's define a notion of variant packages. A variant package differs from a regular package by a suffix in its name, e.g. re-cross:windows-x86. The suffix is available via the variable %{variant}%. This variable, as a special case, is resolved in version constraints as well as %{version}% (https://github.com/ocaml/opam/issues/1993). Since it is a constant string that can include no evaluation of its own, I don't think that presents any practical problems, and a violation of abstraction levels in opam that it entails is more a reflection of an opam's design shortcoming than any true problem.

Given a package name re-cross:VARIANT, it is resolved to an opam file as follows. If a repository has a package with the verbatim name re-cross:VARIANT, then that package's opam file is used. If not, but it has a package re-cross whose opam file includes flags: [ variant ], that package's opam file is used. Otherwise the re-cross:VARIANT package is considered nonexistent.

I'm using re-cross here since the variant mechanism is fairly generic and it may be useful for more than cross-compilation; on top of that the opam files for cross-compiled packages often differ substantially and it is not immediately obvious whether merging them with the non-cross-compiled opam file is wise. Of course, we could decide that the only use for the variant mechanism should be cross-compilation and call it something like "platform".

This is essentially the only thing that is required to support cross-compilation as a first-class citizen. Concepts like a dependency on a build component are expressed elegantly. Similarly, dependency trees are rooted in packages such as ocaml:windows-x86, which do not include flags: [ variant ]; this prevents installation of nonsensical variants.

[lefessan]

Please, could you avoid the use of : in filenames in your proposal ?

As you said, you have already implemented two cross-compilation repositories without variant, so what's the point of this new proposal ? If it is to be able to re-use opam files for both a package and its variant, could you elaborate on how you pick the dependencies of a variant packages ? Is the variant tag propagated transitively ?

[whitequark]

Please, could you avoid the use of : in filenames in your proposal ?

The symbol is entirely unimportant. @ fits maybe even better: rs-cross running at Windows.

As you said, you have already implemented two cross-compilation repositories without variant, so what's the point of this new proposal ?

Because right now it's only possible to use one architecture per switch. This gets very tedious very quickly once you want to deploy to e.g. 32-bit and 64-bit Windows simultaneously, or even worse with iOS, where you need 32/64-bit ARM to make fat binaries for the devices and 32/64-bit x86 to make fat binaries for the emulator, or even worse than that for Android, where for a wide enough reach one has to build for three architectures (ARM, MIPS, x86) and potentially more than two variants within each (e.g. armv7 vs arvm6 for 32-bit alone...).

If it is to be able to re-use opam files for both a package and its variant

This is to re-use opam files for several variants. Although it is possible in principle to re-use the opam file for native installation and for cross-compilation, sometimes it is inconvenient: e.g. for packages that build and run syntax extensions, the cross-compiled package should never try to build syntax extensions, but rather it should depend on the native package and amend the build recipe so that the syntax extension from the native package is used.

This means that the opam file essentially becomes two intertwined opam files with largely different build recipes and dependencies and completely different test recipes, and often different maintainers, at which point I start to doubt the rationality of keeping it all in the single opam file. In the case where maintainers do indeed wish to maintain the cross-compiled variant it should be easy enough to factor out common fields and set up a release process that duplicates them.

Conversely, a typical package made to cross-compile once can be cross-compiled to nearly any architecture, and neatly tucked away into its respective ocamlfind toolchain. The fact that for the majority (actually, I think every single one I wrote so far) of cross-compiled packages, the opam file only differs in the value of OCAMLFIND_TOOLCHAIN is naturally suggestive of sharing them.

could you elaborate on how you pick the dependencies of a variant packages ?

As a baseline proposal, all variant packages are treated as completely separate for all intents and purposes. I think this greatly simplifies implementation.

Often, it will be useful to have variant packages upgraded in lockstep. E.g. if I have ctypes@ios-arm32 and ctypes@ios-arm64 most likely I just want a fat binary and it makes no sense to have different versions. In this case a metapackage ctypes@ios-arm can be added, that depends on both and specifies a version in the dependency. One could even add ctypes@ios that depends on ctypes@ios-arm and ctypes@ios-x86.

As an enhanced proposal, to reduce proliferation of these metapackages, the solver could automatically enforce all variant packages with the same base name (i.e. the part of the name before @) to have the same version.

I prefer the enhanced proposal, but even the baseline proposal is a dramatic improvement in usability of cross-compilation.

Is the variant tag propagated transitively ?

The variant tag will be specified explicitly in the dependency, since it makes sense for a package to e.g. depend on a version of itself without the tag, or else it is also useful to specify metapackages that depend on other packages in variants different than the variant of the metapackage.

[AltGr]

I think I understand your proposal a little bit more now. Incidentally, I have been thinking about how we could implement a provides: field recently, and — putting aside all the problems to resolve for that — the resulting feature seems fairly close; maybe close enough that we wouldn't need both.

The general idea is that dependencies are expressed on features rather than packages, with packages providing their own name as a feature, but possibly others as well; a feature foo can then be provided by a number of packages, possibly including a package foo, and depending on foo should be understood as depending on the disjunction of these packages. (We probably don't just want this with a random choice of the actual package chosen, and there may be some ways to resolve that, but I won't delve into the details there).

So in your case, you would be able to specify that re-cross_windows-x86 has provides: "re-cross"; and you can add available: constraints to control its availability. Then other packages can depend on either re-cross or re-cross_windows-x86 explicitely. Note that in the example you give, it would be closer to have re-cross provide any variants (flags: variant), but that could be inconvenient if there are many.

Do you think something like that could be adapted to your use-case ? It's extremely useful if I can have an idea of a new use-case while I am still designing the feature.

[whitequark]

a feature foo can then be provided by a number of packages, possibly including a package foo, and depending on foo should be understood as depending on the disjunction of these packages

So in your case, you would be able to specify that re-cross_windows-x86 has provides: "re-cross"; and you can add available: constraints to control its availability.

Unless I'm missing something, this is the exact inverse of what I need. The whole point of my proposal is to have multiple co-installed cross-compiled dependency graphs in the same switch; there is never a case where I want to depend on a disjunction of packages because cross-compiled packages aren't replaceable, that is, they expressly provide different features! And yes, it will not help at all with deduplication. See the list above: to support the most moderate list of platforms I'm aiming to for the project I'm working on, I'll need seven opam files (windows-32, windows-64, android-arm32, ios-x86-32, ios-x86-64, ios-arm-32, ios-arm-64)! This is a maintenance nightmare without efficient deduplication.

[AltGr]

Ah, the part that I missed may be that a re-cross package with a flags: variant would never actually be installed as re-cross (providing any variant), but would need to be instanciated for every required variant, acting like a template ?

Did I get it right this time ?

If yes, then indeed, it wouldn't be as complex as I first thought. Having in advance a list of possible variants would help to do the expansion, but that could also be deduced from the set of variant packages, dependencies, and the command-line.

[whitequark]

Ah, the part that I missed may be that a re-cross package with a flags: variant would never actually be installed as re-cross (providing any variant), but would need to be instanciated for every required variant, acting like a template ?

Correct.

Having in advance a list of possible variants would help to do the expansion, but that could also be deduced from the set of variant packages, dependencies, and the command-line.

Yes, that's the idea. E.g. the amount of variants for Android targets is nearly unbounded: you have armv5, armv6, armv7, armv7s, arm64, mips32, mips64, x86, x86_64 (which may or may not have various processor extensions...) now multiply that by soft/hardfloat and Android API level. Of course, not all of them are useful at the same time, but there is no reason to arbitrarily limit this list or, worse, embed it in every package.

[dbuenzli]

I have to admit that I still don't really get this proposal and what it means in practice (e.g. where do we find build and host compilers, cross binaries etc.)

In any case while I appreciate that it may not be possible at the moment to have a single opam file for cross and non-cross, I still think that it should be an eventual goal. So that all the work that goes into curating the opam-repository can be reused for cross-compiled switches without a hitch. I'm sure this should be possible in a user friendly way provided with build systems that do understand cross situations.

So the question is, does this proposal still enables to reach that eventual goal ?

[whitequark]

I have to admit that I still don't really get this proposal and what it means in practice (e.g. where do we find build and host compilers, cross binaries etc.)

Please look at my opam-cross-windows, opam-cross-android, opam-cross-ios etc. This proposal is a formalization of the conventions I use, and it enables nothing less or more than reuse of opam files between platforms and installation of multiple versions of the same package for different platforms side-by-side.

So the question is, does this proposal still enables to reach that eventual goal ?

Yes, there is no technical or conceptual problem with it.

The only practical issue is that the packages will have to handle a large diversity of platforms, many of which require special casing. E.g. most (though by far not all) buildsystems have a graceful way to turn off compilation of native dynlink, but few, topkg not among them, allow to gracefully turn off compilation of dll*.so stubs. So, I foresee these parts of the opam files bitrotting with maintainers who do not have much time or desire to test their software on unsual platforms.

[jordwalke]

I proposed something that sounds similar for yarn.