Controlling link order of `ocaml_lib` libraries ?

[dbuenzli]

I have a few declared ocaml_lib libraries and to compile the executable of my test suite I need to link them in a particular order. I there a way to achieve this ?

[gasche]

I'm not sure, but it should be the case currently that the order in which flags are added to the command-line corresponds to the order in which those flags were declared (... or maybe the reverse order, in which case I would be willing to consider it a bug and fix it). In particular it means that the order of libraries should (anti-)correspond to the order of ocaml_lib declaration. Does that match your experience? If you need different orders for different executables, a work-around would be to associate different tag names to the same library -- ugh.

[dbuenzli]

Does that match your experience?

No. I'll do a repro.

[dbuenzli]

So there a gist here https://gist.github.com/dbuenzli/b257be8c1eec8bf8c3a7c53a6ee2190f

cd /tmp
git clone https://gist.github.com/dbuenzli/b257be8c1eec8bf8c3a7c53a6ee2190f test
cd test 
ocamlbuild -classic-display test.native

Tweak the orders in the _tags file you'll always get the order:

ocamlopt.opt c.cmxa b.cmxa a.cmxa test.cmx -o test.native

If you have a work around it would be really appreciated I have been banging my head on the table with this for three hours now.

[dbuenzli]

Tweak the orders in the _tags file you'll always get the order:

And so does if you try to tweak the order in myocamlbuild.ml

[gasche]

So my impression seems to hold for the case where you use ~extern:true. When ~extern:true holds, the library is assumed to live outside the project, the libraries are added to the link command rather late in the pipeline, and processed by the tag engine which inserts things in rule declaration order¹. When ~extern:false holds, the library is actually picked before that by the dependency-discovery logic, which behaves different with respect to ordering -- I'm having a deeper look.

¹: note that the rule declaration order is the order of calls to ocaml_lib in myocamlbuild.ml, not the ordering in _tags. The tags active on a given file are represented internally as set of strings, so we know for sure their ordering is not preserved today. Changing to a list of strings to preserve the ordering might be possible (and from a user's perspective that would probably be the most convenient way to tweak this behavior), but it's an invasive change.

[gasche]

So the offending code is libs_of_use_lib, which takes the tags of a file and returns the list of library archives coming from ocaml_lib-declared tags to link with.

let libs_of_use_lib tags =
  Tags.fold begin fun tag acc ->
    try let libpath, extern = Hashtbl.find info_libraries tag in
        if extern then acc else libpath :: acc
    with Not_found -> acc
  end tags []

Because it's defined as a fold over the tags of the rule (instead of, as the other code-path, a fold over the flag declarations, matching each declaration against the tags), it is order-agnostic as the tags structure is. On the other hand, we could easily store some "order information" in the info_libraries tables and sort the output list according to this order.

So the current answer is "I don't think there is a quick workaround for non-extern libraries right now". On the other hand, this exploration of the sources leaves me with three ideas of changes that could allow this:

1. store the rule-declaration order and use it as suggest above; easy
2. add a way for some use_lib declarations to announce an explicit dependency from one to another; unknown difficulty
3. preserve the order of tags in _tags file: invasive change, but still possibly workable (modularity/encapsulation in theory allow to replace core structures easily)

So the question could be of what sounds like the most convenient way for users to specify dependencies:

• the library declaration order is the easiest to implement, and would be consistent with other parts of the system, but it's also a bit inflexible; is that a problem?
• the _tags order is maybe the most natural for users, but I think that in general encouraging people to rely on this may be a bit fragile
• specifying inter-dependencies between libraries is more robust than tag orders, but also less convenient

[dbuenzli]

the library declaration order is the easiest to implement, and would be consistent with other parts of the system, but it's also a bit inflexible; is that a problem?

I guess that would be ok.

[dbuenzli]

(and thanks for the analysis)