pkg: ocamlfind does not build

Alizter commented 11 months ago

ocamlfind generates a weird install instruction:

(install
 (progn
  (run %{make} install)
  (run install -m 0755 ocaml-stub %{bin}/ocaml)))

which installs over the ocaml installation. This is a no-no in pkg causing errors such as

Error: Target _build/_private/default/.pkg/ocamlfind/target of kind
"directory" already exists in the build directory
-> required by _build/_private/default/.pkg/ocamlfind/target/cookie
Hint: delete this file manually or check the permissions of the parent
directory of this file

Which still happen even when things are manually deleted.

We probably need to prepare an overlay for this package.

Alizter commented 11 months ago

When also translating filters we have this error:

File "dune.lock/ocamlfind.pkg", line 26, characters 10-35:
26 |      (not %{pkg:ocaml:preinstalled})
               ^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Undefined package variable "preinstalled"

Not sure what this variable is. Could it be user defined?

Edit: This appears to be an artifact of bypassing the ocaml installation since ocaml.config is no longer available. That is where these variables would be defined.

emillon commented 9 months ago

ocamlfind installs things in the ocaml directory because before 5.0, the compiler did not provide META files by itself.
ocaml:preinstalled correspond to whether a system compiler is being used.

Alizter commented 9 months ago

I believe we are correctly ignoring that step since with --experimental-translate-opam-filters we turn this build step into a when.

The more up to date error, after building using #9253 is this:

        make default/.pkg/ocamlfind/target [_private] (exit 2)
(cd _build/.sandbox/7beb9933e56ba265be05835348f0b1d4/_private/default/.pkg/ocamlfind/source && /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install)
if [ "0" -eq 1 ]; then \
    for x in camlp4 dbm graphics labltk num ocamlbuild; do \
      if [ -f "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/$x/META" ]; then \
        if ! grep -Fq '[distributed with Ocaml]' "//home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/$x/META"; then \
          rm -f site-lib-src/$x/META; \
        fi; \
      fi; \
    done; \
    test -f "site-lib-src/num/META" || rm -f "site-lib-src/num-top/META"; \
  fi
echo 'SITELIB_META =' > Makefile.packages.in
for x in `ls site-lib-src`; do test ! -f "site-lib-src/$x/META" || echo $x >> Makefile.packages.in; done
tr '\n' ' ' < Makefile.packages.in > Makefile.packages
rm Makefile.packages.in
install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/bin"
install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man"
/nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install-config
make[1]: Entering directory '/home/ali/test/_build/.sandbox/7beb9933e56ba265be05835348f0b1d4/_private/default/.pkg/ocamlfind/source'
install -d "`dirname \"/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf\"`"
test -f "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf" || install -c findlib.conf "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf"
make[1]: Leaving directory '/home/ali/test/_build/.sandbox/7beb9933e56ba265be05835348f0b1d4/_private/default/.pkg/ocamlfind/source'
for p in findlib; do ( cd src/$p; /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install ); done
make[1]: Entering directory '/home/ali/test/_build/.sandbox/7beb9933e56ba265be05835348f0b1d4/_private/default/.pkg/ocamlfind/source/src/findlib'
install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib"
install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/bin"
test 1 -eq 0 || install -d "/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml"
make[1]: Leaving directory '/home/ali/test/_build/.sandbox/7beb9933e56ba265be05835348f0b1d4/_private/default/.pkg/ocamlfind/source/src/findlib'
install: cannot change permissions of ‘/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml’: Read-only file system
make[1]: *** [Makefile:126: install] Error 1
make: *** [Makefile:24: install] Error 2
-> required by _build/_private/default/.pkg/ocamlfind/target/cookie

Looks to me that ocamlfind is trying to overwrite the ocaml install with a wrapper script. That obviously doesn't work since it doesn't have permissions.

emillon commented 9 months ago

I think it comes from this line:

https://github.com/ocaml/ocamlfind/blob/findlib-1.9.6/src/findlib/Makefile#L126

And indeed, the opam file only passes -no-topfind for ocaml compilers so it's trying to install it in the compiler directory.

Alizter commented 9 months ago

OK passing -no-topfind to the configure step by manually editing dune.lock lets the build progress further. We now have:

Error: Target _build/_private/default/.pkg/ocamlfind/target of kind
"directory" already exists in the build directory
-> required by _build/_private/default/.pkg/ocamlfind/target/cookie
Hint: delete this file manually or check the permissions of the parent
directory of this file

The makefile appears to leave the sandbox installing where the cookie should be before the sandbox step can copy it back it seems.

emillon commented 9 months ago

do you have a log of the succesful commands just before?

Alizter commented 9 months ago

# /home/ali/dune/_build/install/default/bin/dune build --display short --always-show-command-line
# OCAMLPARAM: unset
# Shared cache: enabled
# Shared cache location: /home/ali/.cache/dune/db
# Workspace root: /home/ali/test
$ (cd _build/.sandbox/3e52a4bda7971be6ae15af0d6d7ac996/_private/default/.pkg/ocaml/source && /nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/bin/ocaml ../../ocaml-config/target/share/ocaml-config/gen_ocaml_config.ml 4.14.1 ocaml)
# Dune context:
#  { name = "default"
#  ; kind = lock { default = true }
#  ; profile = Dev
#  ; merlin = true
#  ; fdo_target_exe = None
#  ; build_dir = In_build_dir "default"
#  ; installed_env =
#      map
#        { "INSIDE_DUNE" : "/home/ali/test/_build/default"
#        ; "MANPATH" :
#            "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man:/home/ali/test/_build/_private/default/.pkg/ocaml/target/man:/home/ali/test/_build/_private/default/.pkg/ocaml-config/target/man"
#        ; "OCAMLFIND_IGNORE_DUPS_IN" :
#            "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib:/home/ali/test/_build/_private/default/.pkg/ocaml/target/lib:/home/ali/test/_build/_private/default/.pkg/ocaml-config/target/lib"
#        ; "OCAMLTOP_INCLUDE_PATH" :
#            "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/toplevel:/home/ali/test/_build/_private/default/.pkg/ocaml/target/lib/toplevel:/home/ali/test/_build/_private/default/.pkg/ocaml-config/target/lib/toplevel"
#        ; "OCAML_COLOR" : "always"
#        ; "OCAML_TOPLEVEL_PATH" : "%{toplevel}%"
#        ; "OPAMCOLOR" : "always"
#        }
#  ; instrument_with = []
#  }
$ /run/current-system/sw/bin/curl -V > /tmp/nix-shell.wDvqr4/nix-shell.ueMKEN/dune_f29692_output 2> /dev/null
$ /run/current-system/sw/bin/curl -L -s --user-agent dune --write-out '"%{http_code}"' -o /tmp/nix-shell.wDvqr4/nix-shell.ueMKEN/dune_f99cb5_findlib-1.9.6.tar.gz/download --compressed -- http://download.camlcity.org/download/findlib-1.9.6.tar.gz > /tmp/nix-shell.wDvqr4/nix-shell.ueMKEN/dune_e86f08_output 2> /tmp/nix-shell.wDvqr4/nix-shell.ueMKEN/dune_f99cb5_findlib-1.9.6.tar.gz/curl.stderr
$ /nix/store/ygjmhqdm8labz14xxscz003srl1p2c94-gnutar-1.35/bin/tar xfz /tmp/nix-shell.wDvqr4/nix-shell.ueMKEN/dune_f99cb5_findlib-1.9.6.tar.gz/download -C /tmp/nix-shell.wDvqr4/nix-shell.ueMKEN/opam-3768183-5a61c5 > /tmp/nix-shell.wDvqr4/nix-shell.ueMKEN/dune-source-fetch_c2cf1e_stdout 2> /tmp/nix-shell.wDvqr4/nix-shell.ueMKEN/dune-source-fetch_bb44a6_stderr
$ (cd _build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source && /nix/store/l6wb1grmnawfvy6xqp5x29jsq9p5dym8-patch-2.7.6/bin/patch -p1 -i ./0001-Harden-test-for-OCaml-5.patch) > /dev/null
$ (cd _build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source && ./configure -bindir /home/ali/test/_build/_private/default/.pkg/ocamlfind/target/bin -sitelib /home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib -mandir /home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man -config /home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf -no-custom -no-topfind -no-camlp4)
> Welcome to findlib version 1.9.6
> Configuring core...
> Checking for #remove_directory...
> Testing threading model...
> systhread_supported: true
> Testing DLLs...
> Testing whether ppxopt can be supported...
> Checking for ocamlc -opaque...
> Checking for ocamlopt -g...
> Configuring libraries...
> unix: found
> dynlink: found
> bigarray: found
> compiler-libs: found
> dbm: not present (normal since 4.00)
> graphics: not present (normal since 4.09)
> num: not present (normal since 4.06)
> ocamlbuild: not present (normal since 4.03)
> ocamldoc: found (in +ocamldoc)
> raw_spacetime: not present (normal since 4.12)
> threads: found (in +threads)
> str: found
> labltk: not present (normal since 4.02)
> native dynlink: found
> camlp4: disabled
> bytes: found, installing fake library
> Configuration for stdlib written to site-lib-src/stdlib/META
> Configuration for unix written to site-lib-src/unix/META
> Configuration for dynlink written to site-lib-src/dynlink/META
> Configuration for bigarray written to site-lib-src/bigarray/META
> Configuration for compiler-libs written to site-lib-src/compiler-libs/META
> Configuration for ocamldoc written to site-lib-src/ocamldoc/META
> Configuration for threads written to site-lib-src/threads/META
> Configuration for str written to site-lib-src/str/META
> Configuration for bytes written to site-lib-src/bytes/META
> Detecting compiler arguments: (extractor built) ok
>
> About the OCAML core installation:
>     Standard library:      /nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml
>     Binaries:              /nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/bin
>     Manual pages:          /usr/local/man
>     Multi-threading type:  posix
> The directory of site-specific packages will be
>     site-lib:              /home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib
> The configuration file is written to:
>     findlib config file:   /home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf
> Software will be installed:
>     Libraries:             in <site-lib>/findlib
>     Binaries:              /home/ali/test/_build/_private/default/.pkg/ocamlfind/target/bin
>     Manual pages:          /home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man
>     topfind script:        omitted
> Topfind ppxopt support:    yes
> Toolbox:                   no
> Link custom runtime:       no
> Need bytes compatibility:  no
>
> Configuration has been written to Makefile.config
>
> You can now do 'make all', and optionally 'make opt', to build ocamlfind.
$ (cd _build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source && /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make all)
> for p in findlib; do ( cd src/$p; /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make all ) || exit; done
> make[1]: Entering directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source/src/findlib'
> ocamllex fl_meta.mll
> 22 states, 392 transitions, table size 1700 bytes
> USE_CYGPATH="0"; \
> export USE_CYGPATH; \
> cat findlib_config.mlp | \
>          ../../tools/patch '@CONFIGFILE@' '/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf' | \
>          ../../tools/patch '@STDLIB@' '/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml' | \
>   sed -e 's;@AUTOLINK@;true;g' \
>       -e 's;@SYSTEM@;linux;g' \
>        >findlib_config.ml
> if [ "true" = "true" ]; then                 \
>   cp topfind.ml.in topfind.ml;                             \
> else                                                             \
>   sed -e '/PPXOPT_BEGIN/,/PPXOPT_END/ d' topfind.ml.in     \
>       > topfind.ml ;                                   \
> fi
> ocamldep *.ml *.mli >depend
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c findlib_config.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_split.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_metatoken.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_meta.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -c fl_metascanner.mli
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_metascanner.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -c fl_topo.mli
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_topo.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -c fl_package_base.mli
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_package_base.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -c findlib.mli
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c findlib.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_args.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_lint.ml
> ocamlc -I +compiler-libs -a -o findlib.cma findlib_config.cmo fl_split.cmo fl_metatoken.cmo fl_meta.cmo fl_metascanner.cmo fl_topo.cmo fl_package_base.cmo findlib.cmo fl_args.cmo fl_lint.cmo
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c ocaml_args.ml
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c frontend.ml
> ocamlc -I +compiler-libs  -o ocamlfind -g findlib.cma unix.cma \
>            -I +unix -I +dynlink ocaml_args.cmo frontend.cmo
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -c topfind.mli
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c topfind.ml
> ocamlc -I +compiler-libs -a -o findlib_top.cma topfind.cmo
> USE_CYGPATH="0"; \
> export USE_CYGPATH; \
> cat topfind_rd1.p | \
>          ../../tools/patch '@SITELIB@' '/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib' \
>           >topfind
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -c fl_dynload.mli
> ocamlc -I +compiler-libs -opaque   -I +unix -I +dynlink -g -c fl_dynload.ml
> ocamlc -I +compiler-libs -a -o findlib_dynload.cma fl_dynload.cmo
> make[1]: Leaving directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source/src/findlib'
> /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make all-config
> make[1]: Entering directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source'
> USE_CYGPATH="0"; \
> export USE_CYGPATH; \
> cat findlib.conf.in | \
>      tools/patch '@SITELIB@' '/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib' | \
>        tools/patch '@FINDLIB_PATH@' '/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib' -p >findlib.conf
> if ./tools/cmd_from_same_dir ocamlc; then \
>   echo 'ocamlc="ocamlc.opt"' >>findlib.conf; \
> fi
> if ./tools/cmd_from_same_dir ocamlopt; then \
>   echo 'ocamlopt="ocamlopt.opt"' >>findlib.conf; \
> fi
> if ./tools/cmd_from_same_dir ocamldep; then \
>   echo 'ocamldep="ocamldep.opt"' >>findlib.conf; \
> fi
> if ./tools/cmd_from_same_dir ocamldoc; then \
>   echo 'ocamldoc="ocamldoc.opt"' >>findlib.conf; \
> fi
> make[1]: Leaving directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source'
> File "fl_package_base.ml", line 304, characters 22-40:
> 304 |      let pkg_ancestors = query_requirements predlist pkg in
>                                ^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "fl_package_base.ml", line 350, characters 18-36:
> 350 |   let ancestors = query_requirements predlist package_name in
>                         ^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "fl_package_base.ml", line 375, characters 23-41:
> 375 |       let pkg_ancestors = query_requirements predlist pkg in
>                                 ^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "fl_package_base.ml", line 510, characters 16-35:
> 510 |          ( let c = package_definitions search_path pkg.package_name in
>                          ^^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label search_path was omitted in the application of this function.
> File "findlib.ml", line 390, characters 2-26:
> 390 |   Fl_package_base.requires predlist pkg
>         ^^^^^^^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "findlib.ml", line 396, characters 2-33:
> 396 |   Fl_package_base.requires_deeply predlist pkglist
>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "frontend.ml", line 859, characters 10-39:
> 859 |           Fl_package_base.package_users predicates1 packages1
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "topfind.ml", line 128, characters 40-63:
> 128 |                      match Hashtbl.find Toploop.directive_table "ppx" with
>                                               ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 258, characters 4-27:
> 258 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 268, characters 4-27:
> 268 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 280, characters 4-27:
> 280 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 293, characters 4-27:
> 293 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 307, characters 4-27:
> 307 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 319, characters 4-27:
> 319 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
$ (cd _build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source && /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make opt)
> for p in findlib; do ( cd src/$p; /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make opt ) || exit; done
> make[1]: Entering directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source/src/findlib'
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c findlib_config.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_split.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_metatoken.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_meta.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_metascanner.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_topo.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_package_base.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c findlib.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_args.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_lint.ml
> ocamlopt -I +compiler-libs -g -a -o findlib.cmxa findlib_config.cmx fl_split.cmx fl_metatoken.cmx fl_meta.cmx fl_metascanner.cmx fl_topo.cmx fl_package_base.cmx findlib.cmx fl_args.cmx fl_lint.cmx
> if [ 1 -gt 0 ]; then \
>     ocamlopt -I +compiler-libs -g -shared -o findlib.cmxs findlib_config.cmx fl_split.cmx fl_metatoken.cmx fl_meta.cmx fl_metascanner.cmx fl_topo.cmx fl_package_base.cmx findlib.cmx fl_args.cmx fl_lint.cmx; \
> fi
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c ocaml_args.ml
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c frontend.ml
> ocamlopt -I +compiler-libs -g -o ocamlfind_opt findlib.cmxa unix.cmxa \
>      -I +unix -I +dynlink ocaml_args.cmx frontend.cmx
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c topfind.ml
> ocamlopt -I +compiler-libs -g -a -o findlib_top.cmxa topfind.cmx
> if [ 1 -gt 0 ]; then \
>     ocamlopt -I +compiler-libs -g -shared -o findlib_top.cmxs topfind.cmx; \
> fi
> ocamlopt -I +compiler-libs -g -opaque  -I +unix -I +dynlink -c fl_dynload.ml
> ocamlopt -I +compiler-libs -g -a -o findlib_dynload.cmxa fl_dynload.cmx
> if [ 1 -gt 0 ]; then \
>     ocamlopt -I +compiler-libs -g -shared -o findlib_dynload.cmxs fl_dynload.cmx; \
> fi
> make[1]: Leaving directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source/src/findlib'
> File "fl_package_base.ml", line 304, characters 22-40:
> 304 |      let pkg_ancestors = query_requirements predlist pkg in
>                                ^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "fl_package_base.ml", line 350, characters 18-36:
> 350 |   let ancestors = query_requirements predlist package_name in
>                         ^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "fl_package_base.ml", line 375, characters 23-41:
> 375 |       let pkg_ancestors = query_requirements predlist pkg in
>                                 ^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "fl_package_base.ml", line 510, characters 16-35:
> 510 |          ( let c = package_definitions search_path pkg.package_name in
>                          ^^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label search_path was omitted in the application of this function.
> File "findlib.ml", line 390, characters 2-26:
> 390 |   Fl_package_base.requires predlist pkg
>         ^^^^^^^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "findlib.ml", line 396, characters 2-33:
> 396 |   Fl_package_base.requires_deeply predlist pkglist
>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "frontend.ml", line 859, characters 10-39:
> 859 |           Fl_package_base.package_users predicates1 packages1
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Warning 6 [labels-omitted]: label preds was omitted in the application of this function.
> File "topfind.ml", line 128, characters 40-63:
> 128 |                      match Hashtbl.find Toploop.directive_table "ppx" with
>                                               ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 258, characters 4-27:
> 258 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 268, characters 4-27:
> 268 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 280, characters 4-27:
> 280 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 293, characters 4-27:
> 293 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 307, characters 4-27:
> 307 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
> File "topfind.ml", line 319, characters 4-27:
> 319 |     Toploop.directive_table
>           ^^^^^^^^^^^^^^^^^^^^^^^
> Alert deprecated: Toploop.directive_table
$ (cd _build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source && /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install)
> if [ "0" -eq 1 ]; then \
>     for x in camlp4 dbm graphics labltk num ocamlbuild; do \
>       if [ -f "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/$x/META" ]; then \
>         if ! grep -Fq '[distributed with Ocaml]' "//home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/$x/META"; then \
>           rm -f site-lib-src/$x/META; \
>         fi; \
>       fi; \
>     done; \
>     test -f "site-lib-src/num/META" || rm -f "site-lib-src/num-top/META"; \
>   fi
> echo 'SITELIB_META =' > Makefile.packages.in
> for x in `ls site-lib-src`; do test ! -f "site-lib-src/$x/META" || echo $x >> Makefile.packages.in; done
> tr '\n' ' ' < Makefile.packages.in > Makefile.packages
> rm Makefile.packages.in
> install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/bin"
> install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man"
> /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install-config
> make[1]: Entering directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source'
> install -d "`dirname \"/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf\"`"
> test -f "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf" || install -c findlib.conf "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib.conf"
> make[1]: Leaving directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source'
> for p in findlib; do ( cd src/$p; /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install ); done
> make[1]: Entering directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source/src/findlib'
> install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib"
> install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/bin"
> test 0 -eq 0 || install -d "/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml"
> test 0 -eq 0 || install -c topfind "/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml/"
> files=` ../../tools/collect_files ../../Makefile.config \
> findlib.cmi findlib.mli findlib.cma findlib.cmxa findlib.a findlib.cmxs \
> findlib_config.cmi findlib_config.ml topfind.cmi topfind.mli \
> fl_args.cmi fl_lint.cmi fl_meta.cmi fl_split.cmi fl_topo.cmi ocaml_args.cmi \
> fl_package_base.mli fl_package_base.cmi fl_metascanner.mli fl_metascanner.cmi \
> fl_metatoken.cmi findlib_top.cma findlib_top.cmxa findlib_top.a findlib_top.cmxs \
> findlib_dynload.cma findlib_dynload.cmxa findlib_dynload.a findlib_dynload.cmxs fl_dynload.mli fl_dynload.cmi \
> META` && \
> install -c $files "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib/"
> f="ocamlfind"; { test -f ocamlfind_opt && f="ocamlfind_opt"; }; \
> install -c $f "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/bin/ocamlfind"
> # the following "if" block is only needed for 4.00beta2
> if [ 1 -eq 0 -a -f "/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml/compiler-libs/topdirs.cmi" ]; then \
>     cd "/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml/compiler-libs/"; \
>     install -c topdirs.cmi toploop.cmi "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib/"; \
> fi
> make[1]: Leaving directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source/src/findlib'
> /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install-meta
> make[1]: Entering directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source'
> for x in bigarray bytes compiler-libs dynlink ocamldoc stdlib str threads unix ; do install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/$x"; install -c site-lib-src/$x/META "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/$x/META.tmp" && mv "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/$x/META.tmp" "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/$x/META"; done
> install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib"; install -c Makefile.packages "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/lib/findlib/Makefile.packages"
> make[1]: Leaving directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source'
> test ! -f 'site-lib-src/num-top/META' || { cd src/findlib; /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install-num-top; }
> test ! -f 'site-lib-src/camlp4/META' ||   install -c tools/safe_camlp4 "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/bin"
> /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install-doc
> make[1]: Entering directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source'
> install -d "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man/man1" "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man/man3" "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man/man5"
> install -c doc/ref-man/ocamlfind.1 "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man/man1"
> install -c doc/ref-man/META.5 doc/ref-man/site-lib.5 doc/ref-man/findlib.conf.5 "/home/ali/test/_build/_private/default/.pkg/ocamlfind/target/man/man5"
> make[1]: Leaving directory '/home/ali/test/_build/.sandbox/ba984a1ce5b68d9b599e7beec0434292/_private/default/.pkg/ocamlfind/source'

emillon commented 9 months ago

ok, so it finished make install. The install -m 0755... in the opam file is filtered out, so it completed the install: commands in the opam file. I'm not familiar enough with the pkg rules to help. But I remark that there's a ocamlfind.install file in the source tree as well, I'm not sure what opam is supposed to do in this case.

Alizter commented 9 months ago

The backtrace

Error: Target _build/_private/default/.pkg/ocamlfind/target of kind
"directory" already exists in the build directory
Raised at Stdune__User_error.raise in file
  "otherlibs/stdune/src/user_error.ml", line 9, characters 43-90
Called from Dune_engine__Sandbox.move_targets_to_build_dir.(fun) in file
  "src/dune_engine/sandbox.ml", line 358, characters 11-291
Called from Stdune__Set.Make.to_list_map.(fun) in file
  "otherlibs/stdune/src/set.ml", line 16, characters 37-40
Called from Stdlib__Set.Make.fold in file "set.ml", line 383, characters
  34-55
Called from Stdune__Set.Make.to_list_map in file
  "otherlibs/stdune/src/set.ml", line 16, characters 4-48

indicates that this is indeed as I suspected. The sandbox "moving" step fails because something is already installed. This likely happened because we are passing paths without the sandbox to the variables used in the install. I will try to make a reproduction case.

Alizter commented 9 months ago

I've made a reproduction case in https://github.com/ocaml/dune/pull/9349 and it is as I suspected.

Alizter commented 9 months ago

Actually never mind. I had a stale Dune that was built with #8862 which caused this issue. It seems that passing -no-topfind is enough to get ocamlfind to work. Thanks @emillon!

Alizter commented 9 months ago

We are still missing some configuration that ocamlfind is doing as evidenced in

9351
9350

rgrinberg commented 9 months ago

Ok, thanks to all involved for triaging. I think I know how to proceed.

rgrinberg commented 9 months ago

This PR should fix the issue https://github.com/ocaml/dune/pull/9410

Alizter commented 9 months ago

@rgrinberg This fixes the original comment, but after that I started passing --experimental-translate-opam-filter I get another error as I discussed further up.

With that PR I still get:

install: cannot change permissions of ‘/nix/store/bink43irwl4lp7fli0lh6j82zsrkycp5-ocaml-4.14.1/lib/ocaml’: Read-only file system

This can be fixed by passing -no-topfind to configure, but then ocamlfind doesn't install properly.

FTR the overlay in duniverse seems to be working fine: https://github.com/dune-universe/opam-overlays/blob/master/packages/ocamlfind/ocamlfind.1.9.5%2Bdune/opam and I've successfully built packages that rely on ocamlfind.

rgrinberg commented 9 months ago

What is the generated lock file for ocamlfind?

Alizter commented 9 months ago

(version 1.9.6)

(install
 (progn
  (run %{make} install)
  (when
   %{pkg:ocaml:preinstalled}
   (run install -m 0755 ocaml-stub %{bin}/ocaml))))

(build
 (progn
  (patch 0001-Harden-test-for-OCaml-5.patch)
  (run
   ./configure
   -bindir
   %{bin}
   -sitelib
   %{lib}
   -mandir
   %{man}
   -config
   %{lib}/findlib.conf
   -no-custom
   (when
    (and_absorb_undefined_var
     (not %{pkg:ocaml:preinstalled})
     (>= %{pkg:ocaml:version} 4.02.0))
    -no-camlp4)
   (when %{pkg:ocaml:preinstalled} -no-topfind))
  (run %{make} all)
  (when
   %{pkg:ocaml:native}
   (run %{make} opt))))

(deps ocaml)

(source
 (fetch
  (url http://download.camlcity.org/download/findlib-1.9.6.tar.gz)
  (checksum md5=96c6ee50a32cca9ca277321262dbec57)))

Since I am using the DUNE_PKG_OVERRIDE_OCAML var, perhaps we should be evaluating %{pkg:ocaml:preinstalled} in this case?

rgrinberg commented 9 months ago

Is this (when %{pkg:ocaml:preinstalled} -no-topfind) being evaluated correctly?

Alizter commented 9 months ago

For reference here is the lock file from using the duniverse overlay:

(version 1.9.5+dune)

(install
 (progn
  (when
   %{pkg:ocaml:preinstalled}
   (run install -m 0755 ocaml-stub %{bin}/ocaml))
  (when
   (not %{pkg:ocaml:preinstalled})
   (run cp %{lib}/toplevel/topfind %{lib}/ocaml/topfind))))

(build
 (run env FINDLIB_PREFIX=%{lib} dune build -p %{pkg-self:name} -j %{jobs}))

(deps ocaml dune findlib)

(source
 (fetch
  (url
   https://github.com/dune-universe/lib-findlib/releases/download/1.9.5%2Bdune/findlib-1.9.5.dune.tbz)
  (checksum
   sha256=5242e5a0cfb2af52d6b596767513d80d2a3588608b366c759f9df0841736a228)))

(exported_env
 (= OCAMLPATH "\%{lib}%"))

Alizter commented 9 months ago

Here is the command being run for regular ocamlfind:

$ (cd _build/.sandbox/b5c11eb28c48b9b6a560117fa6333f56/_private/default/.pkg/ocamlfind/source && ./configure -bindir ../target/bin -sitelib ../target/lib -mandir ../target/man -config ../target/lib/findlib.conf -n

As you can see there is no -no-topfind.

rgrinberg commented 9 months ago

What's the contents of your lock.dune file?

rgrinberg commented 9 months ago

Never mind. I think I understand the issue.

Alizter commented 9 months ago

FTR here is dune.lock/lock.dune:

(lang package 0.1)

(dependency_hash f9f9ebdcbd45b7248ae8845838db4c29)

(ocaml ocaml-base-compiler)

(repositories
 (complete true)
 (used
  ((source https://github.com/ocaml-dune/opam-overlays.git)
   (repo_id
    (git_hash 34cada1efb287fed7c4dcc69e1f2a3aa2297f36a)))
  ((source https://github.com/ocaml/opam-repository.git)
   (repo_id
    (git_hash db007ce67660e05f4096cd2d649620c6d263f526)))
  ((source https://github.com/dune-universe/opam-overlays.git)
   (repo_id
    (git_hash 91a371754a2c9f4febbb6c7bb039649ad49a3c13)))))

(expanded_solver_variable_bindings
 (variable_values
  (with-doc false)
  (os-family nixos)
  (os-distribution nixos)
  (os linux)
  (opam-version 2.2.0~alpha-vendored)
  (arch x86_64))
 (unset_variables
  with-test
  sys-ocaml-version
  post
  enable-ocaml-beta-repository
  dev
  build))

rgrinberg commented 9 months ago

Unassigning myself for now as I've done what is needed on the dune side. The next steps are investigating what it will take to make ocamlfind relocatable. @emillon has graciously agreed to investigate. The most sought after end-goal is to make ocamlfind relocatable and upstream whatever changes that will require.

Alizter commented 9 months ago

When building a package such as zarith:

        make default/.pkg/zarith/target [_private] (exit 2)
(cd _build/.sandbox/eecb810a4a72dd478b62f7721b5ea94e/_private/default/.pkg/zarith/source && /nix/store/k51cq3ap20c69wffjsl8qjs21wy5wlrq-gnumake-4.4.1/bin/make install)
ocamlfind install -destdir "" zarith META zarith.cma libzarith.a z.cmi q.cmi big_int_Z.cmi zarith_top.cma z.mli zarith.cmxa zarith_version.cmx z.cmx q.cmx big_int_Z.cmx zarith.cmxs zarith.h q.mli big_int_Z.mli zarith.a z.cmti q.cmti big_int_Z.cmti -optional dllzarith.so
ocamlfind: Config file not found - neither ../target/lib/findlib.conf nor the directory ../target/lib/findlib.conf.d
make: *** [project.mak:119: install] Error 2
-> required by _build/_private/default/.pkg/zarith/target

This is because we passed relative paths for ocamlfind so ocamlfind install looks directly at those which don't make sense in the zarith sandbox. Any way we can add findlib.conf as a dependency of build action for zarith?

emillon commented 8 months ago

I've had some good results by changing the build instructions of ocamlfind 1.9.6 to:

(build
 (progn
  (patch 0001-Harden-test-for-OCaml-5.patch)
  (run mkdir -p %{prefix}/lib)
  (write-file %{prefix}/lib/findlib.conf "")
  (run
   ./configure
   -bindir %{bin}
   -sitelib %{lib}
   -mandir %{man}
   -no-custom
   -no-camlp4
   -no-topfind)
  (run %{make} all)
  (when %{pkg:ocaml:native} (run %{make} opt))
  ))

When -config-file is not passed, the (vanilla) configure script will default to using $(ocamlc -where)/../lib/findlib.conf if it exists, or $(ocamlc -where)/../etc/findlib.conf otherwise. By creating the first path, we make it use that location.

The second source of nonreproducibility is the Makefile.config file. I don't think it's used at runtime, but it's possible to remove it by appending the following to the (install) instructions:

(run rm %{prefix}/lib/findlib/Makefile.config)

ocaml / dune

pkg: ocamlfind does not build #9013

9351

9350