search

mirage / mirage-crypto

Cryptographic primitives for OCaml, in OCaml (also used in MirageOS)
ISC License
77 stars 43 forks source link

Ci use mirage4 #166

Closed hannesm closed 2 years ago

hannesm commented 2 years ago

//cc @TheLortex applying your suggestion from https://github.com/mirage/mirage/issues/1346 -- but somehow the CI fails since now a mirage build attempts to cross-compile everything, and not only the unikernel.

Also, in the generated Makefile (at top level) the build target omits the -f mirage/config.ml -- is this intentional?

TheLortex commented 2 years ago

Currently you have to use -f mirage/config.ml everywhere, which I agree is not very practical.

Also, in the generated Makefile (at top level) the build target omits the -f mirage/config.ml -- is this intentional?

That's a bug indeed.

hannesm commented 2 years ago

Thanks @TheLortex -- though do you have advise how to fix the CI here?

The failure I see at https://cirrus-ci.com/task/6202563144974336?logs=test_mirage#L55 is duplicated below -- it looks like the mirage build -f mirage/config.ml attempts to build all the sources and fails since there are packages (eio, async, ...) that are not relevant for the unikernel, but part of the source tree...

+ gmake depend
using overlay repository mirage: [opam-overlays, mirage-overlays] 
[WARNING] Running as root is not recommended
[opam-overlays] Initialised
[NOTE] Repository opam-overlays has been added to the selections of switch 4.13.1 only.
       Run `opam repository add opam-overlays --all-switches|--set-default' to use it in all existing switches, or in newly created switches, respectively.
[WARNING] Running as root is not recommended
[mirage-overlays] Initialised
[NOTE] Repository mirage-overlays has been added to the selections of switch 4.13.1 only.
       Run `opam repository add mirage-overlays --all-switches|--set-default' to use it in all existing switches, or in newly created switches, respectively.
 ↳ generate lockfile for monorepo dependencies
==> Using 1 locally scanned package as the target.
==> Found 74 opam dependencies for the target package.
==> Querying opam database for their metadata and Dune compatibility.
==> Calculating exact pins for each of them.
==> Wrote lockfile with 38 entries to mirage/mirage/crypto-test-unix.opam.locked. You can now run opam monorepo pull to fetch their sources.
removing overlay repository [opam-overlays, mirage-overlays]
[WARNING] Running as root is not recommended
Repositories removed from the selections of switch 4.13.1. Use '--all' to forget about them altogether.
[WARNING] Running as root is not recommended
Repositories removed from the selections of switch 4.13.1. Use '--all' to forget about them altogether.
 ↳ opam install switch dependencies
[WARNING] Running as root is not recommended
Nothing to do.
 ↳ install external dependencies for monorepo
==> Using lockfile mirage/mirage/crypto-test-unix.opam.locked
 ↳ fetch monorepo rependencies in the duniverse folder
==> Using lockfile mirage/mirage/crypto-test-unix.opam.locked
Successfully pulled 38/38 repositories
+ mirage build -f mirage/config.ml
File "rng/eio/dune", line 4, characters 12-15:
4 |  (libraries eio cstruct logs mirage-crypto-rng duration mtime.clock.os))
                ^^^
Error: Library "eio" not found.
-> required by library "mirage-crypto-rng-eio" in _build/default/rng/eio
-> required by _build/default/META.mirage-crypto-rng-eio
-> required by alias all
-> required by alias default
File "mirage/dune.build", line 19, characters 16-31:
19 |  (enabled_if (= %{context_name} "default"))
                     ^^^^^^^^^^^^^^^
Error: %{context_name} is only available since version 2.7 of the dune
language. Please update your dune-project file to have (lang dune 2.7).
File "tests/dune", line 76, characters 43-51:
76 |  (libraries mirage-crypto-rng-eio duration eio_main)
                                                ^^^^^^^^
Error: Library "eio_main" not found.
-> required by _build/default/tests/test_eio_entropy_collection.bc
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "astring" in _build/default/mirage/duniverse/astring/src
- "astring" in /.opam/4.13.1/lib/astring
  -> required by library "alcotest.stdlib_ext" in
     /.opam/4.13.1/lib/alcotest/stdlib_ext
  -> required by library "alcotest.engine" in
     /.opam/4.13.1/lib/alcotest/engine
  -> required by library "alcotest" in /.opam/4.13.1/lib/alcotest
-> required by executable test_ec in tests/dune:53
-> required by _build/default/tests/test_ec.bc
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "base" in _build/default/mirage/duniverse/base/src
- "base" in /.opam/4.13.1/lib/base
  -> required by library "core" in /.opam/4.13.1/lib/core
  -> required by library "async_kernel.config" in
     /.opam/4.13.1/lib/async_kernel/config
  -> required by library "async_kernel" in /.opam/4.13.1/lib/async_kernel
  -> required by library "async.async_command" in
     /.opam/4.13.1/lib/async/async_command
  -> required by library "async" in /.opam/4.13.1/lib/async
  -> required by library "mirage-crypto-rng-async" in
     _build/default/rng/async
-> required by executable test_entropy_collection_async in tests/dune:41
-> required by _build/default/tests/test_entropy_collection_async.bc
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "stdlib-shims" in _build/default/mirage/duniverse/stdlib-shims/src
- "stdlib-shims" in /.opam/4.13.1/lib/stdlib-shims
  -> required by library "ounit2.advanced" in
     /.opam/4.13.1/lib/ounit2/advanced
  -> required by library "ounit2" in /.opam/4.13.1/lib/ounit2
  -> required by library "test_common" in _build/default/tests
-> required by executable test_pk_runner in tests/dune:27
-> required by _build/default/tests/test_pk_runner.bc
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "base" in _build/default/mirage/duniverse/base/src
- "base" in /.opam/4.13.1/lib/base
  -> required by library "core" in /.opam/4.13.1/lib/core
  -> required by library "async_kernel.config" in
     /.opam/4.13.1/lib/async_kernel/config
  -> required by library "async_kernel" in /.opam/4.13.1/lib/async_kernel
  -> required by library "async.async_command" in
     /.opam/4.13.1/lib/async/async_command
  -> required by library "async" in /.opam/4.13.1/lib/async
-> required by
   _build/default/rng/async/.mirage_crypto_rng_async.objs/byte/mirage_crypto_rng_async.cmo
-> required by _build/default/rng/async/mirage_crypto_rng_async.cma
-> required by alias rng/async/all
-> required by alias default
Error: Conflict between the following libraries:
- "cstruct" in _build/default/mirage/duniverse/ocaml-cstruct/lib
- "cstruct" in /.opam/4.13.1/lib/cstruct
  -> required by library "hex" in /.opam/4.13.1/lib/hex
-> required by
   _build/default/tests/wycheproof/.wycheproof.objs/byte/wycheproof.cmo
-> required by _build/default/tests/wycheproof/wycheproof.cma
-> required by alias tests/wycheproof/all
-> required by alias default
Error: Conflict between the following libraries:
- "stdlib-shims" in _build/default/mirage/duniverse/stdlib-shims/src
- "stdlib-shims" in /.opam/4.13.1/lib/stdlib-shims
  -> required by library "ounit2.advanced" in
     /.opam/4.13.1/lib/ounit2/advanced
  -> required by library "ounit2" in /.opam/4.13.1/lib/ounit2
-> required by _build/default/tests/.test_common.objs/byte/test_common.cmo
-> required by _build/default/tests/test_common.cma
-> required by alias tests/all
-> required by alias default
Error: Conflict between the following libraries:
- "cstruct" in _build/default/mirage/duniverse/ocaml-cstruct/lib
- "cstruct" in /.opam/4.13.1/lib/cstruct
  -> required by library "randomconv" in /.opam/4.13.1/lib/randomconv
-> required by
   _build/default/tests/.test_common_random.objs/byte/test_common_random.cmo
-> required by _build/default/tests/test_common_random.cma
-> required by alias tests/all
-> required by alias default
(cd _build/default/mirage/duniverse/ocaml-gmp/src && /usr/local/bin/opam config subst build.sh)
[WARNING] Running as root is not recommended
(cd _build/default/src-freestanding && /usr/local/bin/gmake)
touch libmirage_crypto_freestanding_stubs.a
(cd _build/default/ec-freestanding && /usr/local/bin/gmake)
touch libmirage_crypto_ec_freestanding_stubs.a
(cd _build/default/mirage/duniverse/Zarith && /usr/local/bin/bash -e -u -o pipefail -c 'env CC="cc -O2 -fno-strict-aliasing -fwrapv -pthread -fPIC" LDFLAGS="-L/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/" CFLAGS="-I/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/" CPPFLAGS="-I/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/" ./configure --ocamllibdir /.opam/4.13.1/lib/ocaml')
binary ocaml: found in /.opam/4.13.1/bin
binary ocamlc: found in /.opam/4.13.1/bin
binary ocamldep: found in /.opam/4.13.1/bin
binary ocamlmklib: found in /.opam/4.13.1/bin
binary ocamldoc: found in /.opam/4.13.1/bin
binary cc: found in /usr/bin
binary ocamlopt: found in /.opam/4.13.1/bin
checking compilation with cc -O2 -fno-strict-aliasing -fwrapv -pthread -fPIC -I/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/: working
include caml/mlvalues.h: found
library dynlink.cmxa: found
binary ocamlfind: found in /.opam/4.13.1/bin
OCaml's word size is 64
binary uname: found in /usr/bin
include gmp.h: found
library gmp: found
OCaml supports -bin-annot to produce documentation
detected configuration:
  native-code:          yes
  dynamic linking:      yes
  defines:              -DHAS_GMP 
  libraries:            -L/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/ -lgmp
  C options:            -I/tmp/cirrus-ci-build/_build/default/mirage/duniverse/Zarith/../../../../install/default/lib/gmp/
  installation path:    /.opam/4.13.1/lib
  installation method   findlib
configuration successful!
now type "make" to build
then type "make install" or "sudo make install" to install
+ exit 1
hannesm commented 2 years ago

@TheLortex do you have an idea how to solve this issue, and get Mirage4 to be used in the CI with the example unikernel?

I guess it boils down to "how to compile the example unikernel in here with the HEAD branch of mirage-crypto-*?"

TheLortex commented 2 years ago

Hi, the mirage build command should only ask dune to build the unikernel. Currently it's invoking dune build --root . which is asking dune to build everything in the workspace, so that's not correct. The invocation of dune to only build the unikernel would be dune build --root . mirage/dist/

hannesm commented 2 years ago

Thanks again @TheLortex, I managed to adapt the .test-mirage.sh to use the dune build --root . mirage/dist/ directly instead of a mirage build, and the CI is passing now \o/