Issues to compile dune from scratch

rjbou commented 1 year ago

When opening an issue, following issue instructions i began the process to add a test & open PR. I followed the hacking instruction from an empty switch, but wasn't able to easily compile.

Expected Behavior

Update guidelines/makefile to ease install from scratch, on an empty environment.

Actual Behavior

hacking.rst

Following hacking guidelines, i went for the simple (on an empty switch).

$ make dev
$ ./dune.exe build @check
$ ./dune.exe runtest

make dev builds dune.exe, but fails because of missing dependencies

$ make dev
ocamlc -output-complete-exe -w -24 -g -o .duneboot.exe -I boot unix.cma boot/libs.ml boot/duneboot.ml
./.duneboot.exe
./_boot/dune.exe build @install
File "otherlibs/dune-rpc-lwt/src/dune", line 4, characters 38-46:
4 |  (libraries result csexp dune_rpc lwt lwt.unix unix)
                                          ^^^^^^^^
Error: Library "lwt.unix" not found.
-> required by library "dune-rpc-lwt" in
   _build/default/otherlibs/dune-rpc-lwt/src
-> required by _build/default/META.dune-rpc-lwt
-> required by _build/install/default/lib/dune-rpc-lwt/META
-> required by _build/default/dune-rpc-lwt.install
-> required by alias install
make: *** [Makefile:53: dev] Error 1

However, it builds dune.exe, but checking that everything compile or launching test (the 2 other commands) fail as there is other missing dependencies (ppx_inline_test.config, ppx_inline_test.config, core_bench.inline_benchmarks, lwt, stdio, etc.).

Reproduction

$ opam switch create test-sw --empty # or an already empty switch
$ git clone git@github.com:ocaml/dune.git
$ cd dune
$ git reset --hard e8ad4cd5db8b9b2
$ make dev
$ ./dune.exe build @check
$ ./dune.exe runtest

Using `make dev-switch`

After discussion with @emillon, i was advised to use make dev-switch. Here it fails too, but for another reason: opam can't install dependencies at switch creation as they depend on dune.

test "test-sw" = "/tmp/dune" || \
    opam switch create -y . 4.14.0 --deps-only --with-test

<><> Installing new switch packages <><><><><><><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml-base-compiler" {= "4.14.0"} | "ocaml-system" {= "4.14.0"}]
The following actions will be performed:
=== install 38 packages
  ∗ base                  v0.15.1 [required by ppx_expect]
[...]
#=== ERROR while compiling cppo.1.6.9 =========================================#
# context     2.2.0~alpha~dev | linux/x86_64 | ocaml-base-compiler.4.14.0 | https://opam.ocaml.org
# path        /tmp/dune/_opam/.opam-switch/build/cppo.1.6.9
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p cppo -j 6
# exit-code   1
# env-file    ~/.opam/log/cppo-78510-577e06.env
# output-file ~/.opam/log/cppo-78510-577e06.out
### output ###
# bwrap: execvp dune: No such file or directory

<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build cppo                1.6.9
│ λ build csexp               1.5.1
│ λ build jane-street-headers v0.15.0
│ λ build ocaml-compiler-libs v0.12.4
│ λ build pp                  1.1.2
│ λ build ppx_derivers        1.2.1
│ λ build re                  1.10.4
│ λ build result              1.5
│ λ build sexplib0            v0.15.1
│ λ build stdlib-shims        0.3.0
[...]

Reproduction

$ opam switch create test-sw --empty # or an already empty switch
$ git clone git@github.com:ocaml/dune.git
$ cd dune
$ git reset --hard e8ad4cd5db8b9b2
$ make dev-switch

Using `make dev-switch` on an empty local switch

Finally, I decided to create a local empty switch to not let the makefile create one, and go directly to dependencies install. It worked well! I was able to build dune, check everything, and run testsuite. The only remaining issue is a missing binary on a test: node

$ ./dune.exe build @check
File "test/blackbox-tests/test-cases/dune", line 18, characters 8-19:
18 |   (deps %{bin:node})
             ^^^^^^^^^^^
Error: Program node not found in the tree or in PATH

Reproduction

$ opam switch create test-sw --empty # or an already empty switch
$ git clone git@github.com:ocaml/dune.git
$ cd dune
$ git reset --hard e8ad4cd5db8b9b2
$ opam switch create . --empty
$ make dev-switch
$ make dev
$ ./dune.exe build @check
$ ./dune.exe runtest

However, it wasn't a blocker for new test addition.

Specifications

Version of dune (output of dune --version): main branch, e8ad4cd5db8b9b2
Version of ocaml (output of ocamlc --version): irrelevant
Operating system (distribution and version): debian

Additional information

Full log can be found here.

Thanks!

Alizter commented 1 year ago

Maybe we need to update the doc. I don't remember what the make dev target was supposed to do. What it should really be is:

make _boot/dune.exe
./dune.exe build @check

You won't be able to fully runtest because some tests need dependencies on other libraries. A simple way is to do:

opam install ./dune.opam
make dev-deps

This should install the correct deps in the existing switch rather than creating a new one using the dev version of Dune.

snowleopard commented 1 year ago

I hit the same issue today. Should we update the doc indeed? It does say to run make dev first.

snowleopard commented 1 year ago

And we should probably also say to run make dev-deps... or make dev-deps-sans-melange since installing Melange failed for me after taking forever. Surprisingly, running the latter command complained about some Melange failures too.

It would be nice to have a small set of dependencies/tests sufficient for working on the build engine. Dune now supports a ton of stuff and installing everything and running all the tests locally seems like an overkill.

snowleopard commented 1 year ago

...and then the build still failed with pc-setup-dkml and gh-dkml-action-yml missing. Ugh.

snowleopard commented 1 year ago

Not sure why this issue was closed as "not planned". Clearly the Hacking doc and a bunch of stuff is currently broken/stale.

ocaml / dune