Cross Compilation!

[jaredly]

In the Overview docs page cross-compilation is touted as a competitive advantage.... with the caveat that it's not actually implemented.

I didn't see an issue tracking this feature, so I thought I'd add one! feel free to close if it's just noise

[pirbo]

Please don't close as it is not noise ;-)

https://github.com/whitequark/opam-cross-windows (and similar for IOS and android) is an effort to make cross compilation easy for platforms where compilation can be painful.

9/39 of its packages have now a more recent version released built using jbuilder (and all the packages I dramatically depend upon btw :-) ). That means that opam-cross-* projects are basically agonising with jbuilder not being cross compilation ready.

This is super frustrating as cross compilation using ocamfind was relatively smooth and the non existing feature of cross compilation with jbuilder is advertised as a super exciting novelty...

[dra27]

Could you elaborate on what's broken - is it that those packages now generate incorrect META files, or is it that they can no longer be compiled for cross-compilation because their build systems have been switched to jbuilder?

[dra27]

(or possibly both)

[pirbo]

Sorry, maybe nothing is broken and it is just a documentation issue. My problem is "I have absolutely no idea how to cross compile a project that uses jbuilder" I explain a bit more ignorance at https://discuss.ocaml.org/t/to-be-solved-hopefully-cross-compiling-project-using-jbuilder/572

[pirbo]

First concrete question would be: how do I "say" to jbuilder to use the cross compiler instead of the compiler. Can I set $(OCAMLC) and $(OCAMLOPT)?

[ghost]

Note that the way I envisioned cross-compilation in jbuider is not the same approach as the one used by https://github.com/whitequark/opam-cross-windows. I believe that you could update the ...-windows packages that switched to jbuilder by extending the PATH with %{prefix}/windows-sysroot/bin. However, I don't know how to do that in opam files, you might need a wrapper script. Additionally, this won't work when the compilation of a package involves building an executable and then running it, but I don't think this is supported by opam-cross-windows either.

Regarding cross-compilation support in jbuilder itself, since jbuilder is already able to build against multiple configurations (build contexts) in parallel, what I had in mind was to simply have one build context setup for the host and another one for the target. Then when jbuilder needs to run an executable during the build, it would simply resolve it to the host build context. With opam it would simply require setting up 2 opam switches and would require no additional package repository.

[pirbo]

There is indeed no package that builds an executable and executes it later during its own build. There are packages that produce executable to by run by the host during compilation of packages that depend upon them (lwt with ppx_lwt and atdgen). You're right, they are handled in a fragile dirty way. We make the -windows package (the cross compiled one) depending upon the regular one (lwt-windows.3.0.0 depends upon lwt.3.0.0) and we create symlinks for the host executable in the "cross-compiled environment". I'll be more than happy to get a more robust solution.

Similarly, I'll be more than happy to avoid to have to "repackage" all the package I want to cross-compile one by one.

If I've understood you correctly, the preliminarily step is to make the cross-compiler (which is for now a package only available in the "correct" opam switch (the exact same ocaml version)) a proper opam switch. I'm not fully competent here but I think it will quite challenging. For now, compiling OCaml with the correct options so that you obtain a cross compiler does not build an ocamlrun for the host. But, the built cross compilers do depend upon an ocamlrun for the host... Could you make an opam switch depend upon on other one? Do you want to do that? Sounds ugly... Maybe we can imagine to compile OCaml twice in a cross compiler switch first time with normal option to get the host executable and second time to get the cross compilers. I've never create a switch, I don't know how custom you can go...

Your second paragraph is beyond my competence and I'm therefore completely lost. Are you saying that a package will be built in 2 opam (somehow entangled) switches at the same time? Do you insinuate that somehow jbuilder will recognise a cross compiler switch and when you want to install a package in it, it will require the corresponding non cross-compiler switch to exists, will check that the to-be-cross-compiled package is installed there (or at least its dependency) and (without any special configuration in the opam file for that) it will compile some part of the package using this switch? Sorry, I stop here my complete non sens fable but I really don't see what you have in mind (even if I look forward to it if it can make my life easier).

[hcarty]

It would be potentially very useful to have switches which depend on/build on one another. For example it could allow sharing of the compiler, jbuilder, merlin and other tools between switches which use the same compiler version. And for the cross-compilation use described here.

[ghost]

Following are a few more details about the way I see cross-compilation with jbuilder.

Build contexts (present)

First of all, let's look at build contexts. Without special configuration, jbuilder will use whatever ocaml compiler is available in the PATH and store the build artifacts in _build/default. Now, if you write a file jbuild-workspace containing:

(context ((switch 4.04.0)))
(contest ((switch 4.05.0)))

then jbuilder will instead keep two parallel trees of artifacts in _build/4.04.0 and _build/4.05.0. The contents of _build/4.04.0 will be the same as what you would have got without special configuration in _build/default if you did eval $(opam config env --switch 4.04.0) first. The same goes for _build/4.05.0.

Cross-compilation (future)

Now, let's assume that we extend the configuration of build contexts and allow to write this in the jbuild-workspace file:

(context ((switch 4.05.0)))
(context ((switch 4.05.0+mingw) (host 4.05.0)))

This would have two consequences:

1. to compile OCaml code inside _build/4.05.0+mingw, jbuilder would use the mingw-ocamlopt binary in the 4.05.0 opam switch rather than the ocamlopt binary in 4.05.0+mingw switch

2. let's assume that you have the following src/jbuild file:

(executable ((name foo)))
(rule (with-stdout-to blah (run ./foo.exe)))

When building _build/4.05.0/src/blah, jbuilder will resolve ./foo.exe to _build/4.05.0/src/foo.exe as expected. However, for _build/4.05.0+mingw/src/blah jbuilder will resolve ./foo.exe to _build/4.05.0/src/foo.exe since it knows 4.05.0 is the host context for 4.05.0+mingw.

We should never end up trying to call mingw-ocamlrun, so there is no need for it.

What does that give us?

Assuming that the right packages are installed in both opam switches, jbuilder will be able to cross-compile a given package without doing anything special.

Some packages might still have to be updated to support cross-compilation. For instance if the foo.exe program in the previous example was using Sys.os_type, it should instead take it as a command line argument:

(rule (with-stdout-to blah (run ./foo.exe -os-type ${os_type})))

However, such changes could be easily upstreamed.

What's missing?

This gives us a good story for cross-compiling OCaml code in general, but that doesn't tell us how the user can install a cross-compiled package with opam install .... That's a question for opam, but it seems to me that a simple solution would be to simply install everything twice, i.e. once in both switches. That's a gross over-approximation, but that would avoid having to specify where dependencies should come from, which could be tedious.

And if all your dependencies are using jbuilder, you can also build everything at once and jbuilder would build exactly what is necessary.

[toots]

Would love to see some progress on this. I've been trying to help with the effort at whitequark/opam-cross-windows and jbuilder is currently staling the whole effort. While the plan for cross-compilation looks awesome, there is currently no way, it seems, to force jbuilder to pick-up <host->-ocaml* binaries, leaving it only with dirty options such as editing the $PATH..

[rgrinberg]

Stay tuned. This is pretty much the highest priority item at the moment.

On Tue, Oct 24, 2017, 7:29 AM Romain Beauxis notifications@github.com wrote:

Would love to see some progress on this. I've been trying to help with the effort whitequark/opam-cross-windows and jbuilder is currently staling the whole effort. While the plan for cross-compilation looks awesome, there is currently no way, it seems, to force jbuilder to pick-up -ocaml* binaries, leaving it only with dirty options such as editing the $PATH..

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/janestreet/jbuilder/issues/144#issuecomment-338825878, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIe-wp5Ert-l4hoJ1xFGYoMOmcCUL2qks5svSFSgaJpZM4N3-jg .

[toots]

[rgrinberg]

Initial support is present via findlib toolchains and the -x option.

[srikanthkyatham]

Hi all Any progress on cross compilation story, any ticket/issue open which I could follow. Thanks!

[dra27]

There is support for cross-compilation planned for the compiler itself in two phases for OCaml 4.10 later this year and 4.11 early next year, but there are no issues/PRs to track at this stage.

[Jesse-Millwood]

There is support for cross-compilation planned for the compiler itself in two phases for OCaml 4.10 later this year and 4.11 early next year, but there are no issues/PRs to track at this stage.

I see there are some options like the custom runtime for 4.10 but is there anything specific you remember for 4.11? I see that 4.11 features are pretty much frozen and I didn't see anything cross compilation related when I skimmed the change log.