Make configuration lookup aware of the --toolchain option

[whitequark]

Before this commit, the .install file was generated using the ocamlc configuration for the build system. When cross-compiling, this causes it to contain incorrect extensions, e.g. dllfoo.so for Windows, though this did not actually cause installation failures because of the way opam-installer works.

This commit is larger than it looks, because unfortunately I had to move code around a bunch. The solution with an optional argument to Topkg_conf.tool is also somewhat ugly, but making it mandatory resulted in a cascade of API changes, some of which are probably breaking (I don't know what is the topkg API stability policy) and some I do not know how to fix (the invocations in Topkg_care_ipc, for example).

Ultimately it's just passing -toolchain to the ocamlfind ocamlc invocation.

Also, once this PR is merged, topkg will become the first packager tool that has first-class support for cross-compilation. I will start looking at changes to opam that will allow to completely reuse opam files...

[whitequark]

Depends on https://github.com/ocaml/ocamlbuild/issues/181 being fixed, though.

[dbuenzli]

Before this commit, the .install file was generated using the ocamlc configuration for the build system.

To be precise it was generated using the ocamlc configuration found by Conf.tool "ocamlc" Host_os which has plenty of ways to be made to point to the right ocamlc (especially with opam v2.0 I assume).

I'm still a bit surprised you didn't try to explore that route as I think that for the future it would be better if we find ways of achieving this without relying on ocamlfind and the -toolchain option which I hope you have in mind for your opam investigations.

(Nothing against merging this though).

[whitequark]

I'm still a bit surprised you didn't try to explore that route as I think that for the future it would be better if we find ways of achieving this without relying on ocamlfind and the -toolchain option which I hope you have in mind for your opam investigations.

So far, ocamlfind has been the single most crucial component in achieving practical cross-compilation. Without it, I wouldn't have cross-builds with oasis, omake, omakefile, or handwritten makefiles. (Which is to say I would have to partially or completely rewrite target buildsystems. Right now that only happens with ocp-build, which is thankfully very unpopular.) The motivation for having first-class support for ocamlfind toolchains in topkg (or any other build tool), I think, doesn't need explanation.

As for investigating other avenues, to put it bluntly, what is to be gained? It is an uphill battle that very few people are interested in (topkg has the distinction of being the first OCaml tool to even try), and even if I/we manage to push something upstream, this means one more configuration to support, because none of the ocamlfind-using build systems are going away any time soon, and most of them ever. As a packager I would like to see every ocamlc invocation to be wrapped in ocamlfind, and I have to a degree succeeded in that, pushing this upstream into every maintained package that I cross-compiled.

Of course, you could always integrate ocamlfind into topkg itself, and autogenerate META and _tags files based on pkg/pkg.ml, which would actually be pretty sweet, but it seems like a lot of effort for avoiding a few forks. I'm not sure if this is what you have in mind.

[whitequark]

As for investigating other avenues, to put it bluntly, what is to be gained?

To put it another way: to see meaningful improvement in OCaml packaging one would have to start at the compiler, to address the issues with ppx extensions at least, (not that there's any consensus on what, or even whether something should be changed) and work their way up the stack, while remaining compatible with all legacy code out there, and afterwards one would still have to support everything that was supported before, and now this other mode.

On top of that, the effort required to produce the build tool you wish is immense. I am not even sure if the OCaml community has available the sort of manpower required. Rust has this kind of build tool, Cargo, and it took them three rewrites from scratch to get it right, somewhere around two years to make it minimally usable, and a lot more will have to be done before it's polished. That's with someone working full-time just on Cargo for a large period of time, more people working full-time to get the compiler and language support there, and no legacy ecosystem to speak of. I do not have so much hubris (or so much time) as to imagine I can get this project done, nor such a strong interest in OCaml packaging anyway.

[dbuenzli]

As for investigating other avenues, to put it bluntly, what is to be gained?

Eventual simplification of the build eco-system. I have the feeling that you are only interested in now and quick, I happen to be interested in the long term perspective and simplification and constantly need to deal with entropy generating people with a lack of long term build vision and simplification.

ocamlfind's META files are an absurd part of a build bureaucracy (and obscurity) that we should eventually try to eliminate if we once want to get to a reasonable build eco-system. You may want to check out the recent discussion on https://github.com/ocaml/ocaml/pull/1065 for more details about these claims. Of course this won't happen in a fortnight but it's a goal we should keep in mind when deal with the current system and try to evolve it towards a better one.

In this particular case you are basically using ocamlfind -toolchain as an indirection. topkg gives you other ways to perform these indirections. Fundamentally this PR is absolutely not needed in an opam cross-compilation switch in which the HOST_OS_OCAMLC environment variable is made to point to the right binary (or in which the HOST_OS_XBIN variable is made to point to a bin directory which has all of the cross tools). Something you only need to set once in the switch to deal with all topkg packages. Similarly you could have pointed HOST_OS_OCAMLBUILD an appropriate binary rather than having introduced the Conf.toolchain configuration variable.

You may not like the impliciteness of these environment variables but I argue they eventually make the system more explicit since your build logs have the full paths to the binaries that are being used rather than you having to understand the layer of ocamlfind obscurity and have to peek for whichever findlib.conf is being used to eventually find out what tool is really being invoked.

and autogenerate META and _tags files based on pkg/pkg.ml,

This is unlikely to happen for now. People in the autogeneration business are only introducing indirections to deal with poorly designed systems which enshrines them and leads to even more baroque and poorly designed systems. Do not autogenerate, fix the systems themselves. For this particular case eliminate the need for META files altogether (in any case for people interested in META generation, the build system itself is in the best position to perform this since this is where ocamlfind packages are specified) or provide me a good API to _tags files that can be readily reused for topkg's business.

[whitequark]

I have the feeling that you are only interested in now and quick, I happen to be interested in the long term perspective and simplification and constantly need to deal with entropy generating people with a lack of long term build vision and simplification.

It seems that you're under impression that you will be solving OCaml package management once and for all. While I wish that was true, the practical reality is you will be adding just yet another way of building and packaging code. I wouldn't call that "entropy generating" if it is not badly done but I do not see the simplification in any such plan either.

Fundamentally this PR is absolutely not needed in an opam cross-compilation switch in which the HOST_OS_OCAMLC environment variable is made to point to the right binary

This is true but otherwise the behavior of --toolchain is wrong.

Similarly you could have pointed HOST_OS_OCAMLBUILD an appropriate binary rather than having introduced the Conf.toolchain configuration variable.

This is not true. There is no "host ocamlbuild" as ocamlbuild does not generate code itself. It has to know about both the build and host systems (which is what I've done), or alternatively the system invoking it must know about those, and correctly invoke it with -just-plugin/-no-plugin (which you've rejected). Of course, adding -toolchain to ocamlbuild helps everyone and not just topkg users.

Anyway, these are mere nitpicks. Topkg is rather inconvenient to use as this layer of indirection yet, but it's probably possible. Again, the answer to your question is that pushing some more structured package management in OCaml will never, ever, result in any benefit I will see, and has the drawback of having to spend my finite lifetime on it. Not that I really use my toolchain work much either anymore. For that matter, if a large amount of packages stops using ocamlfind without migrating to a solution that natively supports cross-compilation like topkg does, I will rather just deprecate the opam-cross-* repositories because it is not worth the hassle trying to maintain them in this situation.

[whitequark]

Ah, and about removing META files. I have no affection to them and if you can come up with and implement a solution that reliably removes META files, I will be glad to use and promote it. There are still serious unsolved problems though with that plan, such as ppx.

[dbuenzli]

It seems that you're under impression that you will be solving OCaml package management once and for all.

I won't, we will. It's a common effort that involves many people out there across different projects including the compiler. It's only that when people get to know suffer to understand something it sometimes becomes difficult to make them see, think and lean towards other ways.

I wouldn't call that "entropy generating" if it is not badly done but I do not see the simplification in any such plan either.

There will be a transition period that's why, I said eventual simplification. However the ocamlfind system can perfectly live in harmony with other plans without interference. There shouldn't be much more entropy since these other plans require very little from the end-users, in contrast to the ocamlfind workflow.

I'm not fine with the status quo and I don't see a way out through ocamlfind, this is why I try to entice people not to solve problems through it (a natural and easy problem solver though since it's an indirection layer).

This is true but otherwise the behavior of --toolchain is wrong.

Yes.

This is not true. There is no "host ocamlbuild" as ocamlbuild does not generate code itself. It has to know about both the build and host systems (which is what I've done), or alternatively the system invoking it must know about those, and correctly invoke it with -just-plugin/-no-plugin (which you've rejected). Of course, adding -toolchain to ocamlbuild helps everyone and not just topkg users.

Note I didn't say that ocamlbuild didn't need the toolchain option. I claimed that formally topkg didn't need it. You can have a "host ocamlbuild" that invokes the cross compilers (e.g. via -toolchain) and a "build ocamlbuild" that invokes the build compilers. IIRC a technique you pioneered yourself. This is for example what I tried to do in the tgls distribution massage hook which generate bindings through a build-os program [1] [2]).

I will rather just deprecate the opam-cross-* repositories because it is not worth the hassle trying to maintain them in this situation.

It should be our goal that you eventually stop maintaining them. Not because of the hassle, but because they are no longer needed.

I have no affection to them and if you can come up with and implement a solution that reliably removes META files, I will be glad to use and promote it. There are still serious unsolved problems though with that plan, such as ppx.

Some experimental steps have already been done in the odig project (see e.g. the toplevel loaders) but it will take time until this matures fully in conjunction with changes in the OCaml compilation model. But again this is not for tomorrow exactly and we'll continue to ocamfind for some time.

[dbuenzli]

The patch is in as 9117310b5acb7c15c387b3bcb79