OPAM repo support

[dra27]

Three features added:

1. Support in the repository for pinning the flexdll and flexlink packages in OPAM
2. Makefile targets for generating OPAM metadata files for opam-repository
3. Fully centralised infrastructure for version number (including a Makefile target for bumping it after release)

[dra27]

OPAM 2.0 (especially the Windows version) is of course not yet finalised, so I wouldn't necessarily merge this yet... but it's here for interest/comment/review now!

[alainfrisch]

Can you describe a bit more the OPAM support? In particular, why are sources of flexlink installed? Same questions for runtime support files? I understand one might need to recompile them for a given version of msvc, but in the context of OPAM, couldn't we safely asssume that a given switch is tied to a specific version of msvc and compile the runtime support upon installation? Or is the idea to use opam to deploy a more generic version of flexlink with the source files of its runtime support, for use outside opam?

[dra27]

For newer versions of FlexDLL (the 0.29-0.34 packages do it differently - see https://github.com/dra27/opam-repository/tree/next-windows), the porting I've been doing uses the bootstrapping mode for flexdll. So the flexdll OPAM package installs the sources so that the OCaml package can copy them to flexdll/ (rather than downloading them).

So the flow for installing OCaml 3.11.0--4.02.3 in OPAM works as follows:

1. Install flexdll package (it's a dependency). This means compiling the required flexdll and flexdll initer object files and installing them and flexlink.exe from the binary release zip file to the OPAM switch bin dir. flexdll.h is copied is the packages lib dir.
2. Install ocaml package. The OPAM package ensures that flexdll.h can be picked up (https://github.com/dra27/opam-repository/blob/next-windows/packages/ocaml/ocaml.4.03.0/files/win.sh#L64) and flexlink.exe, etc. is in PATH, so it just works "as normal"
3. Install flexlink package. This recompiles flexlink.exe using the new ocamlopt and replaces the flexlink.exe which was extracted from the binary release zip. At this stage, a purist would relink all the OCaml executable files (i.e. repeat everything which used flexlink before), but even my evangelism knows some bounds :smile:

For installing OCaml 4.03.0 and later (which means flexdll >= 0.35) it's a little simpler:

1. Install flexdll, but this time nothing is actually compiled - the package just installs flexdll.h to lib and all the sources to share
2. Install ocaml package. The preparation script for that copies the flexdll sources from the share directory to flexdll/ in the build tree (see https://github.com/dra27/opam-repository/blob/next-windows/packages/ocaml/ocaml.4.03.0/files/win.sh#L97-L101)
3. Install flexlink package. This is just a dummy package - there's nothing to compile because OCaml's make -f Makefile.nt world.opt stage already built and installed flexlink.exe.

[dra27]

Related to this, have you seen MPR#7373?

[alainfrisch]

For OCaml 4.03.0: it seems a bit weird to me that the flexdll package installs sources which are compiled by the ocaml package. Why not include everything in the ocaml package itself (either provide a custom .tar.gz for Windows with a copy of flexdll sources; or add these files in the OPAM package)?

For older OCaml, the ocaml package could include the precompiled flexlink.exe.

Basically, I think I'm suggesting merging the flexdll and ocaml packages under Windows (but keep flexlink, in particular for people who could use OPAM to set up a development environment for working on flexlink itself).

What do you think of this approach?

[dra27]

I agree it's a little weird - but it's only an implementation detail. It would be difficult to put it as a single .tar.gz for ocaml, unless upstream did that (because effectively it would be a fork). However, OPAM 2.0 does also have an extra-sources field which is mainly used for downloading patches but could also be used to retrieve the sources directly.

My reason for splitting it this way is that it uncouples FlexDLL and OCaml at the OPAM level - so you can, for example, switch to OCaml 3.11.0 and upgrade the flexdll package to 0.35 (which obviously recompiles OCaml) - with your approach, OCaml and FlexDLL releases are fixed together. It also means that already - niche as it may be - you can pin flexdll in order to work on the C code and have OPAM automatically test OCaml with it.

I should add (as in the screenshots I sent earlier this year) that the process is transparent for the casual Windows user (if there is such a thing for OCaml!!) - those packages are all automatically selected and dealt with just by specifying ocaml.

[dra27]

Another option, which would reduce the weird copying of sources, would be to alter the bootstrapping process in OCaml - so have make -f Makefile.nt flexdll bootstrap flexlink only and still require the C objects to have been compiled separately beforehand? If that were changed, it'd be no problem at all back-porting it to 4.03 and 4.04 for OPAM's benefit (see https://github.com/metastack/ocaml-legacy!)

[dra27]

Actually, I thought even more (various cycling journeys...) and maybe for OPAM it would be better to have a version of the OCaml bootstrap which only creates the bytecode flexlink. So:

1. OPAM installs flexdll which, for all versions, compiles the C support objects and installs them to lib/flexdll
2. OPAM installs ocaml which, for versions before 4.03.0, itself downloads and installs flexlink.exe from the binary zip and uses that and for versions after 4.03.0, uses something like make -f Makefile.nt flexlink world world.opt which only builds a bytecode image of flexlink and doesn't compile the native code version or install flexlink.exe at all
3. OPAM installs flexlink which, for all versions, compiles and installs the native code version of flexlink.exe

I still like for convenience the make -f Makefile.nt flexdll world world.opt (but then I was always likely to) for testing, so I'd be tempted to keep it... but that'd be a matter for discussion on the OCaml GPR.