Define a C library that should be statically linked to a C-binding compilation

[vthemelis]

I have an OCaml library that has a C binding. The C binding has a dependency on a native component that I would like to also build with dune. At the moment, the dune file looks like:

(rule
 (deps
  (source_tree vendor))
 (targets libexample.a)
 (action
  (no-infer
   (progn
    (chdir
     vendor
     (progn (* RUN A BUILD STEP HERE *))
    (copy vendor/libexample.a libexample.a)))))

(library
 (name exampleWithNativeDep)
 (package examplePackage)
 (flags -linkall)
 (libraries base)
 (wrapped false)
 (foreign_stubs
  (language c)
  (names binding)
  ;; Note that there's a separate C binding file that relies on Dune to be built.
  ;;  This needs the libexample.a produced above but libexample is not directly targeted by OCaml.
  (flags :standard -Ivendor/Include -I.))
 (c_library_flags -lutil -lpthread)
 (foreign_archives example)

This works fine when running OCaml native builds but fails when I want to run a bytecode build. The issue is that the bytecode build wants dllexample.so to also exist.

Given that:

• I only need the artefacts of the rule to statically link them in my OCaml consumed C binding;
• I still want to rely on Dune to build the C binding;
• I need the binding to be statically linked to libexample.a both when the binding is built as a static library for consumption from native builds and dynamic library for consumption from bytecode builds.

is there a way to express this kind of behaviour with in dune?

This may be related to https://github.com/ocaml/dune/issues/4409

[Alizter]

cc @nojb

[vthemelis]

A very concrete example is here: https://github.com/grievejia/pyre-ast/pull/7/files#r1284319463

Effectively, I want to bring the rule outputs (which include a lib<something>.a) into the context of the C build for the library binding but not in the OCaml build as I'm not going to be FFI'ing directly into lib<something>.a.

This seems to work fine in the context of the dune-project above but it doesn't work correctly if I import this library in another library through opam. It looks like the linker is called again and it gets confused by the relative path in https://github.com/grievejia/pyre-ast/pull/7/files#diff-245cbcb675c7970f9f730daf5855169c2107e2fa08aed275c22ac704545c4cf4R39

Getting an error like:

File "dune", line 389, characters 7-11:   
389 |  (name main)
             ^^^^
ld: warning: directory not found for option '-L../../_build/default/lib/taglessFinal/'
ld: library not found for -lpython
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking (exit code 1)

[nojb]

is there a way to express this kind of behaviour with in dune?

Perhaps you can try using extra_deps in your foreign_stubs stanza to declare a dependency of your binding to libexample.a and pass it to the linker somehow using c_library_flags (the right invocation is probably going to depend on your particular compiler and system).

[vthemelis]

What are the semantics of extra_deps? Does it bundle the dependencies with my library? How is it going to work once the library needs to be linked again when building a different library/executable that depends on this library?

[vthemelis]

@nojb, I tried adding changing my foreign_stubs to:

 (foreign_stubs
  (language c)
  (names binding)
  (extra_deps %{project_root}/_build/default/lib/taglessFinal/libpython.a)
  (flags :standard -Ivendor/Include -I.))

Then, once I pin this library in opam, a client library that has bytecode mode breaks with:

⨯ dune build
Error: Error during linking (exit code 1)
File "interprocedural_analyses/taint/test/dune", line 35, characters 2-23:
35 |   sanitizeTransformTest
       ^^^^^^^^^^^^^^^^^^^^^
ld: warning: directory not found for option '-L../../_build/default/lib/taglessFinal/'
ld: library not found for -lpython
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking (exit code 1)

The only way I have to predictably fix this is to replace -L%{project_root}/_build/default/lib/taglessFinal/ into an absolute path (which of course doesn't work for a public library).

[vthemelis]

@nojb, do you know if it's somehow possible to opt-in to the same linking command in both the bytecode build and the exe build? This would allow me to just use foreign_archives and get on with it.

At the moment, dune will complain that it want dllpython.so for the bytecode build.

[nojb]

@nojb, do you know if it's somehow possible to opt-in to the same linking command in both the bytecode build and the exe build? This would allow me to just use foreign_archives and get on with it.

Not sure I understand exactly, but if you do not need the bytecode build, you can disable it by setting (modes native) in your (executable) stanza.

[vthemelis]

It seems that at the moment, dune is trying to link to my foreign library via libpython.a in exe mode and via dllpython.so in byte mode. Is it possible to use libpython.a for both modes?

I do need the byte mode to build in this case.

[nojb]

It seems that at the moment, dune is trying to link to my foreign library via libpython.a in exe mode and via dllpython.so in byte mode.

This is expected: pure bytecode executables cannot contain native code, so any native code must go into a shared library.

Is it possible to use libpython.a for both modes?

You can try (modes native byte_complete). The byte_complete mode means that Dune will embed the bytecode into a native executable that will link any native code statically. However, bytecode executables built this way do not support debugging information so you won't have exception backtraces and you won't be able to use ocamldebug with it.

[vthemelis]

This is expected: pure bytecode executables cannot contain native code, so any native code must go into a shared library.

Thanks for clarifying! In my case, libpython.a is only needed to be linked against the C stub of my OCaml library, not against the OCaml library itself.

So I want libpython.a to be linked against both my stub's lib.a and dll.so which should be possible irrespective of the mode of my OCaml executable.

Is it possible to have libpython.a available during both the stub so and lib compilation?

I do need the byte mode for debugging.

[nojb]

So I want libpython.a to be linked against both my stub's lib.a and dll.so which should be possible irrespective of the mode of my OCaml executable.

I am not an expert, but merging a static library into an existing static and/or shared library is highly system-dependent so Dune (and the OCaml toolchain) do not support this out-of-the-box as far as I know.

I would suggest that you first find out the exact linker invocation that would accomplish what you want and only later try to encode that as a Dune recipe.

[vthemelis]

I already have a dune hack that allows me to achieve exactly this but is very brittle due to the way in which I tell dune how to find the libpython.a static library:

https://github.com/grievejia/pyre-ast/pull/7/files#diff-245cbcb675c7970f9f730daf5855169c2107e2fa08aed275c22ac704545c4cf4L36-R39

Here, I just use -lpython on the C flags to link statically but I need to specify the location of the produced libpython.a library in a brittle way.

Note on the diff above how I had to remove the (foreign_archives python) from my stanza because that ended up requiring dllpython.so when building the bytecode which shouldn't be needed for building the stubs (it should only needed for building OCaml).

I think that what I want is effectively a way to add foreign_archives directly to the stub compilation, not both the stub compilation and the OCaml compilation as it happens now. Given that the stub compilation is just a C compilation, I should be able to specify if I want to link my foreign_archives statically or dynamically. So in that hypothetical dune language I'd be able to write:

(library
 (name taglessFinal)
 (package pyre-ast)
 (libraries base)
 (foreign_stubs
  (language c)
  (names binding)
  ;; Have foreign_archives directive inside the foreign_stubs.
  ;; Be able to specify if I want to statically link the archive or not.
  ;; If I don't opt-in for static linking then dune should require the dlls but not otherwise.
  (foreign_archives (link static) python)
  (flags :standard -Ivendor/Include -I.))
 (c_library_flags -lutil -lpthread)
 ;; no longer use the foreign_archives directive for the whole library
 ;; (foreign_archives python))

Alternatively, just having a dune variable that gives me the absolute path to the lib*.a file would serve just fine for a stopgap solution.

[nojb]

I already have a dune hack that allows me to achieve exactly this but is very brittle due to the way in which I tell dune how to find the libpython.a static library:

You mean this? It looks a little fishy to me: normally %{project_root} is the relative path to the root of the project directory inside the build directory, so I would have expected something like %{project_root}/lib/taglessFinal instead.

 (c_library_flags
  -lpthread
  -lpython
  -lutil
  -L%{project_root}/_build/default/lib/taglessFinal/))

But more generally, I would think that a solution along these lines is perfectly acceptable if it works; not sure it is worth it to investigate more elaborate fixes which will require more work to put in place.

[vthemelis]

Thanks, indeed, if -L%{project_root}/lib/taglessFinal worked, that would be an acceptable solution from my side but unfortunately it doesn't seem to work as dune only copies the lib*.a file inside the _build directory from my rule

[nojb]

unfortunately it doesn't seem to work as dune only copies the lib*.a file inside the _build directory from my rule

Sorry, but I cannot parse this sentence... could you clarify what is the issue you are facing?