pkg: Zarith doesn't build

[Alizter]

Building Zarith fails with:

ocamlfind install -destdir "/home/ali/HoTT/outputs/out/lib/ocaml/4.14.1/site-lib/" zarith META zarith.cma libzarith.a z.cmi q.cmi big_int_Z.cmi zarith_top.cma z.mli zarith.cmxa zarith_version.cmx z.cmx q.cmx big_int_Z.cmx zarith.cmxs zarith.h q.mli big_int_Z.mli zarith.a z.cmti q.cmti big_int_Z.cmti -optional dllzarith.so
ocamlfind: Bad configuration: Cannot mkdir /home/ali/HoTT/outputs/out/lib/ocaml/4.14.1/site-lib/zarith because a path component does not exist or is not a directory

This is using DUNE_PKG_OVERRIDE_OCAML=1 for the lock and build. And the following dune-project:

(lang dune 3.11)

(package
 (name foo)
 (dependencies
  (zarith (= 1.13))))

[Alizter]

The issue here is really that ocamlfind install doesn't work. Because it has been configured incorrectly.

[rgrinberg]

Yeah, that's a shame. There's a few packages like that we're going to add workarounds for. I think the simplest thing to do for now is to introduce our own overlay system like duniverse. We will need very few overlays in comparison to duniverse.

Also, it would be nice if we had a way to mark packages as being incompatible with dune's package management.

cc @Leonidas-from-XIV who worked with something like this in duniverse already.

[Alizter]

Zarith is able to be packaged up for Nix, but obviously that doesn't use the opam file.

https://github.com/NixOS/nixpkgs/blob/397aa97b0eb578f15534580461f7d3e9e2eb0c93/pkgs/development/ocaml-modules/zarith/default.nix#L7

They configure with the correct paths which is perhaps something we could do. Or at least modify the OPAM file for Zarith to allow. I think that is a non controversial change.

Duniverse had a stronger constraint of using dune to build everything whereas we don't really have that. Therefore I am not yet convinced an overlay systems is needed yet.

[rgrinberg]

Or we can use the dune build system Hugo write. Any of those options will require an overlay until the upstream package is updated.

[Leonidas-from-XIV]

cc @Leonidas-from-XIV who worked with something like this in duniverse already.

Overall I'd say the UX for this is pretty bad at large.

What we would need is

1. A way to mark packages as not buildable in the dune package management system (in opam-monorepo we required that the build instructions use dune, but here we'd need to know through some other means)
2. Maintain a repository with fixes which is a pretty annoying thing to do since you need to keep it up to date and we just don't really have the resources for it. dune-overlays hasn't been updated for a long time.

The other UX hiccup is when users fail to add the repository locking fails in annoying ways, so if we wanted to support this I would strongly suggest :standard to contain the overlay repo by default.

[Alizter]

        make default/.pkg/zarith/target [_private] (exit 2)
(cd _build/.sandbox/ab70976d60ff442b9d5c27cbea74d853/_private/default/.pkg/zarith/source && /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install)
ocamlfind install -destdir "" zarith META zarith.cma libzarith.a z.cmi q.cmi big_int_Z.cmi zarith_top.cma z.mli zarith.cmxa zarith_version.cmx z.cmx q.cmx big_int_Z.cmx zarith.cmxs zarith.h q.mli big_int_Z.mli zarith.a z.cmti q.cmti big_int_Z.cmti -optional dllzarith.so
ocamlfind: Config file not found - neither ../target/lib/findlib.conf nor the directory ../target/lib/findlib.conf.d
make: *** [project.mak:119: install] Error 2
-> required by _build/_private/default/.pkg/zarith/target/cookie

Is the latest error after fixing ocamlfind with the solution in #9013. Seems that ocamlfind is configured incorrectly or we are not passing the correct options.

[gridbugs]

I can't reproduce this issue anymore. My experiment is to take this empty dune project that depends on zarith: https://github.com/gridbugs/dune-pkg-dashboard/tree/main/bonsai-deps/zarith ...and run dune pkg lock && dune build and that works for me (aarch64 macos).

There's a chance that you'll get this error:

Error:
_build/.sandbox/8632755805c6a19305d7e278cabc9c3f/_private/default/.pkg/ocamlfind/source/configure:
No such file or directory
-> required by _build/_private/default/.pkg/ocamlfind/target/cookie
-> required by - package zarith

...which is https://github.com/ocaml/dune/issues/10080 and will be fixed in https://github.com/ocaml/dune/pull/10122 and affects all packages (seemingly non-deterministically), not just zarith.

[Alizter]

Here is the error I get when trying to build:

ocamlfind: Config file not found - neither ../target/lib/findlib.conf nor the directory ../target/lib/findlib.conf.d
        make default/.pkg/zarith/target [_private]
(cd _build/.sandbox/d7a39a761def65104092b1b8d88d232c/_private/default/.pkg/zarith/source && /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make)
(echo "let"; grep "version" META | head -1) > zarith_version.ml
ocamldep  zarith_version.ml z.ml q.ml big_int_Z.ml z.mli q.mli big_int_Z.mli > depend
ocamlc -g -I +compiler-libs -bin-annot  -c zarith_version.ml
ocamlc -g -I +compiler-libs -bin-annot  -c z.mli
ocamlc -g -I +compiler-libs -bin-annot  -c z.ml
ocamlc -g -I +compiler-libs -bin-annot  -c q.mli
ocamlc -g -I +compiler-libs -bin-annot  -c q.ml
ocamlc -g -I +compiler-libs -bin-annot  -c big_int_Z.mli
ocamlc -g -I +compiler-libs -bin-annot  -c big_int_Z.ml
ocamlmklib -g -failsafe -o zarith zarith_version.cmo z.cmo q.cmo big_int_Z.cmo -L/nix/store/44bqlp05nq9ymdjvngjx8fdjrfmgj63h-gmp-with-cxx-6.3.0/lib -lgmp 
ocamlc -ccopt "-I/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml  -I/nix/store/1gpyzgxz77m4jvmbd9nn3kf8qdlf4hcf-gmp-with-cxx-6.3.0-dev/include -DHAS_GMP  " -c caml_z.c
ocamlmklib -g -failsafe -o zarith caml_z.o -L/nix/store/44bqlp05nq9ymdjvngjx8fdjrfmgj63h-gmp-with-cxx-6.3.0/lib -lgmp 
ocamlc -g -I +compiler-libs -bin-annot  -c zarith_top.ml
ocamlc -g -o zarith_top.cma -a zarith_top.cmo
ocamlopt -g -I +compiler-libs  -c zarith_version.ml
ocamlopt -g -I +compiler-libs  -c z.ml
ocamlopt -g -I +compiler-libs  -c q.ml
ocamlopt -g -I +compiler-libs  -c big_int_Z.ml
ocamlmklib -g -failsafe -o zarith zarith_version.cmx z.cmx q.cmx big_int_Z.cmx -L/nix/store/44bqlp05nq9ymdjvngjx8fdjrfmgj63h-gmp-with-cxx-6.3.0/lib -lgmp 
ocamlopt -shared -o zarith.cmxs -I . zarith.cmxa -linkall
        make default/.pkg/zarith/target [_private] (exit 2)
(cd _build/.sandbox/d7a39a761def65104092b1b8d88d232c/_private/default/.pkg/zarith/source && /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install)
ocamlfind install -destdir "" zarith META zarith.cma libzarith.a z.cmi q.cmi big_int_Z.cmi zarith_top.cma z.mli zarith.cmxa zarith_version.cmx z.cmx q.cmx big_int_Z.cmx zarith.cmxs zarith.h q.mli big_int_Z.mli zarith.a z.cmti q.cmti big_int_Z.cmti -optional dllzarith.so
ocamlfind: Config file not found - neither ../target/lib/findlib.conf nor the directory ../target/lib/findlib.conf.d
make: *** [project.mak:119: install] Error 2
-> required by _build/_private/default/.pkg/zarith/target

[Alizter]

You have to build zarith _build/_private/default/.pkg/zarith/target direclty for some reason as the rules don't seem to include it in @all.

[Alizter]

Another way to trigger this is to try to build Coq with PM.

[gridbugs]

Oh interesting, so it seems like it works when it's built as a dependency in some cases and not others, and it doesn't build when referred to as the target? Any idea why zarith is special in this regard?

[gridbugs]

If I make an empty project (with (allow_empty)) that depends on the zarith package I can see from the output of dune build that zarith is being built and it exits without error. However if I add an executable that depends on the zarith library (ie. (libraries zarith)) then I get the error you reported.

[Leonidas-from-XIV]

@Alizter W.r.t to the ocamlfind error, this PR is aimed to improve the situation: https://github.com/ocaml/ocamlfind/pull/72 where it tries to do a bit better about determining the location where ocamlfind is installed and determine paths from there. I will try and see if it helps.

[gridbugs]

To help understand this issue I made a small library that installs with bare ocamlfind and a dependent app that builds with dune pkg. It fails in the same way as zarith so I think it's safe to say that this problem is not specific to zarith. Here's the project: https://github.com/gridbugs/oneshot-webserver-app