Lupus
commented
1 year ago I've came up with one idea, and it seems to be working and solves this issue. Basically it boils down to piggyback on cargo vendoring and versioning capabilities to distribute OCaml code. I'll explain this idea on ocaml-lwt-interop example.
Project structure overview
├── src /* Rust sources */
│ ├── lib.rs
│ ├── local_executor.rs
│ ├── notification.rs
│ ├── promise.rs
│ ├── ptr.rs
│ ├── stubs.rs /* All of [ocaml::sig] and [ocaml::func] */
│ └── util.rs
├── test /* Tests */
│ ├── dune
│ └── test.ml
├── vendor /* Cargo vendor dir */
│ └── .............
├── build.rs /* Build script to parse [ocaml::sig] */
├── Cargo.lock /* Lock file for local development/CI */
├── Cargo.toml /* Rust crate definition */
├── dune /* OCaml bindings library is defined here */
├── dune-project /* OCaml dune project definition */
├── LICENSE
├── NOTICE
├── README.md
├── Rust_async.ml /* Hand-written higher level wrapping over stubs */
├── Rust_async.mli
├── rust-async.opam
├── stubs.ml /* Generated from stubs.rs */
└── stubs.mliCargo.toml
Cargo.toml defines a library of type ["staticlib", "cdylib", "lib"], so that it can be used both from Rust and from OCaml. Otherwise Cargo.toml does not have anything noteworthy.
dune
dune defines a rule to build rust code, and defines a library:
; Include .cargo as dune ignores .* by default
(dirs :standard .cargo)
(rule
(targets libocaml_lwt_interop.a dllocaml_lwt_interop.so)
(deps
(glob_files_rec *.toml)
(glob_files_rec *.rs)
.cargo/config
(source_tree vendor))
(locks cargo-build)
(action
(no-infer
(progn
(chdir
%{workspace_root}
(run cargo build --target-dir %{workspace_root}/_rust --release --offline --package ocaml-lwt-interop))
(copy %{workspace_root}/_rust/release/libocaml_lwt_interop.a libocaml_lwt_interop.a)
(copy %{workspace_root}/_rust/release/libocaml_lwt_interop.so dllocaml_lwt_interop.so)))))
(library
(name rust_async)
(public_name rust-async)
(libraries lwt.unix)
(foreign_archives ocaml_lwt_interop)
(c_library_flags
(-lpthread -lc -lm))
(preprocess
(pps lwt_ppx)))cargo is executed from %{workspace_root} (see (chdir ...)), which is basically _build/default in your project source tree. This rule depends on all *.rs and *.toml files, .cargo/config, and all of the vendor source tree to properly rebuild Rust code on any changes.
So while being in %{workspace_root}, cargo is requested to perform an offline release build only for our package and use %{workspace_root}/_rust as target directory. After that's done, the rule copies Rust artifacts into current folder (as chdir scopes only around cargo build). The no-infer stanza tells dune to not try to figure out where are the command arguments coming from, which allows to use copy instead of resorting to shell and mv.
(locks cargo-build) is a small optimization to avoid having multiple cargo processes running at the same time, as each is spawning a lot of parallel jobs assuming it has all the resources available exclusively, and total build time is a bit slower.
Rest of dune file is pretty straightforward.
Cargo build script
Build script looks like this:
use std::env;
fn main() -> std::io::Result<()> {
println!("cargo:rerun-if-changed=build.rs");
println!("cargo:rerun-if-changed=src/stubs.rs");
let current_dir = env::current_dir()?;
let current_dir = current_dir.to_str().unwrap();
let out_filename = "stubs.ml";
if current_dir.ends_with("/_build/default") {
println!(
"cargo:warning=[ocaml-lwt-interop/build.rs] Not generating {} as launched from dune build dir",
out_filename
);
return Ok(());
}
ocaml_build::Sigs::new(out_filename).generate()
}The generated stubs.ml(i) are committed to the repository, so we add an ugly hack to avoid overwriting of source files in dune build dir, as they are write-protected and build script fails to overwrite them, failing the whole build as a result. One is expected to run cargo build directly to regenerate them (or wait till your IDE does that behind your back!). See use cases below for the explanation of this choice.
Use cases
Local builds
When building the project locally, cargo is being launched from _build/default, and is requested to build a specific package, that is locally available in current directory. Cargo picks up the config, vendor dir, Cargo.toml, and builds the package as expected.
Vendoring to other project
Vendoring is the only supported mode to distribute OCaml/Rust bindings for consumption of other libraries or applications. We rely on the fact that cargo brings all the assets in the original repository along with Rust sources - in our case the most valuable assets are OCaml sources and Dune files.
[dependencies]
ocaml-lwt-interop = { git = "https://github.com/Lupus/ocaml-lwt-interop.git", branch = "build-from-vendor" }When cargo vendors a package, it removes its Cargo.lock and vendor dir, which solves the problem of having multiple Rust package versions in one project, and that also naturally solves the same problem for OCaml part of the bindings.
When Dune builds the project, it finds the vendored Dune projects inside cargo vendor dir, which allows current project to depend on those libraries, and those libraries can also depend on each other, and as long as those dependencies are reflected in corresponding Cargo.toml files, we essentially use cargo to distribute our OCaml code! And more importantly we have a guarantee that OCaml code will consistently use the same version of Rust libraries, and getting some opaque pointer to Rust type from one OCaml binding library, and passing it to another binding library would result in the same memory representation of that type and same functions which will work with it, despite the actual binary code being duplicated in actual .so/.a that are linked by OCaml.
Building of vendored OCaml/Rust library out of cargo vendor dir works, because of (chdir %{workspace_root} ....) for corresponding cargo invocation, and specificication of concrete package that we want to build. If we were to build from the same directory as vendored Cargo.toml, cargo will complain that we try to build a package that is supposed to be in worspace, but it not included in the workspace (YMMV, my top level project uses cargo workspace to define several packages in one folder).
I'm not sure about cargo vendoring and build scripts, it seems that it should invoke them when vendoring the code, I wasn't able to get the build script to run in the vendored library, so I just committed the artifacts. Probably this aspect can be improved. Distributing the [ocaml::sig] generator as a separate binary crate and just calling the binary from dune would simplify a lot of things in this setup.
Having all Rust builds run against single %{workspace_root}/_rust target folder ensures that cargo does not build anything multiple times, which might be the case when using per-project Rust building sandboxes. %{workspace_root} works stable even when Dune libraries are vendored arbitrarily in the project tree, %{workspace_root} will be the same for all those libraries.
Evaluation
So far I created an experimental branch with follows the design described above. I'm able to build the project from it's home directory, and depend on it via git url in another project's Cargo.toml, cargo vendor it and build other project, both using cargo build and dune build.
I have not yet tried this approach to depend on some library A, that depends on some library B, both being OCaml/Rust binding libraries, but I belive it should work as expected, as cargo will vendor both A and B only once in vendor/ dir, Dune will spot them, and resolve the dependencies at OCaml level accordingly.
zshipko
tizoc
I'm thinking about some sound approach to distribute bindings for
ocaml-rsas separate libraries.I have a project, ocaml-lwt-interop, which provides some OCaml C stubs, that are wrapped by some OCaml library, and also it provides some library types and functions for Rust. Some other project might be willing to depend on
ocaml-lwt-interopand provide more OCaml and Rust primitives.How it's best to distribute it?
One idea that I currently have is to separate Rust stubs part and Rust library part into separate crates. Crate with stubs will "embedded" into OCaml library, that wraps those stubs with higher level API (yet exposing raw Rust opaque types and getters for them so that they can be used in other binding libraries). And Rust library part should be a normal Rust crate.
Let's say someone wants to write bindings to
hyper, this might look like this (Rust libs in red, OCaml libs in yellow):But how to solve the version constraints? If final OCaml application installs OCaml library
rust-hyper, that vendors all its Rust dependencies, andrust-hyperpullsrust-async, which also vendors its Rust dependencies, won't we end up with conflicting versions ofocaml-lwt-interopbeing used at the same time?Probably such libraries with bindings should not vendor Rust deps, and should not be published to opam, but need to all be vendored in the final OCaml application, where Rust dependencies need to be vendored? 🤯