master does not build (with real operating systems)

[hannesm]

ocaml pkg/pkg.ml build --tests false --with-lwt true --xen false --freestanding true

ocamlfind ocamlopt -shared -package bytes -package cstruct src/nocrypto.cmx -o src/nocrypto.cmxs
ocamlfind ocamlopt -shared -package bytes -package cstruct unix/nocrypto_entropy_unix.cmx -o unix/nocrypto_unix.cmxs
ocamlfind ocamlopt -shared -package bytes -package cstruct lwt/nocrypto_entropy_lwt.cmx -o lwt/nocrypto_lwt.cmxs
ocamlfind ocamlopt -shared -linkall -package bytes -package cstruct mirage/nocrypto_mirage.cmxa -o mirage/nocrypto_mirage.cmxs
ocamlfind ocamlc -g -ccopt '-DACCELERATE -msse2 -maes' -ccopt '--std=c99 -Wall -Wextra -O3' -ccopt '-I/home/hannes/.opam/mirage-solo5-ukvm/lib/pkgconfig/../../include/ocaml-freestanding/include -nostdlibinc -ffreestanding -mno-red-zone -mno-3dnow -isystem /home/hannes/.opam/mirage-solo5-ukvm/lib/pkgconfig/../../include/solo5-kernel-virtio/include/host -I/home/hannes/.opam/mirage-solo5-ukvm/lib/pkgconfig/../../include/solo5-kernel-virtio/include' -ccopt '--std=c99 -Wall -Wextra -O3' -package bytes -package cstruct -c X/mirage-freestanding/src/native/hash/stubs.c
+ ocamlfind ocamlc -g -ccopt '-DACCELERATE -msse2 -maes' -ccopt '--std=c99 -Wall -Wextra -O3' -ccopt '-I/home/hannes/.opam/mirage-solo5-ukvm/lib/pkgconfig/../../include/ocaml-freestanding/include -nostdlibinc -ffreestanding -mno-red-zone -mno-3dnow -isystem /home/hannes/.opam/mirage-solo5-ukvm/lib/pkgconfig/../../include/solo5-kernel-virtio/include/host -I/home/hannes/.opam/mirage-solo5-ukvm/lib/pkgconfig/../../include/solo5-kernel-virtio/include' -ccopt '--std=c99 -Wall -Wextra -O3' -package bytes -package cstruct -c X/mirage-freestanding/src/native/hash/stubs.c
X/mirage-freestanding/src/native/hash/stubs.c:3:10: fatal error: 'md5.h' file not found
#include "md5.h"
         ^
1 error generated.
Command exited with code 2.

using cpuid and ocb-stubblr as dev~mirage on a FreeBSD host with clang 3.8.0. any hints?

[pqwy]

tree _build/X/mirage-freestanding

cat _build/X/mirage-freestanding/src/native/hash/stubs.c.depends

[hannesm]

# cat  _build/X/mirage-freestanding/src/native/hash/stubs.c.depends
stubs.o: X/mirage-freestanding/src/native/hash/stubs.c \
  X/mirage-freestanding/src/native/hash/../nocrypto.h caml/mlvalues.h \
  caml/bigarray.h hash/bitfn.h sha1.h

# find _build/X/mirage-freestanding/
_build/X/mirage-freestanding/
_build/X/mirage-freestanding/src
_build/X/mirage-freestanding/src/libnocrypto_stubs+mirage-freestanding.clib
_build/X/mirage-freestanding/src/native
_build/X/mirage-freestanding/src/native/misc.c
_build/X/mirage-freestanding/src/native/hash
_build/X/mirage-freestanding/src/native/hash/sha256.c.depends
_build/X/mirage-freestanding/src/native/hash/sha512.c.depends
_build/X/mirage-freestanding/src/native/hash/md5.c
_build/X/mirage-freestanding/src/native/hash/sha1.h
_build/X/mirage-freestanding/src/native/hash/sha512.c
_build/X/mirage-freestanding/src/native/hash/stubs.c.depends
_build/X/mirage-freestanding/src/native/hash/stubs.c
_build/X/mirage-freestanding/src/native/hash/sha1.c.depends
_build/X/mirage-freestanding/src/native/hash/md5.c.depends
_build/X/mirage-freestanding/src/native/hash/bitfn.h
_build/X/mirage-freestanding/src/native/hash/sha256.c
_build/X/mirage-freestanding/src/native/hash/sha1.c
_build/X/mirage-freestanding/src/native/des
_build/X/mirage-freestanding/src/native/des/generic.c
_build/X/mirage-freestanding/src/native/des/generic.c.depends
_build/X/mirage-freestanding/src/native/des/generic.h
_build/X/mirage-freestanding/src/native/nocrypto.h
_build/X/mirage-freestanding/src/native/misc.o
_build/X/mirage-freestanding/src/native/aes
_build/X/mirage-freestanding/src/native/aes/aesni.c.depends
_build/X/mirage-freestanding/src/native/aes/generic.c.depends
_build/X/mirage-freestanding/src/native/aes/aesni.c
_build/X/mirage-freestanding/src/native/aes/generic.c
_build/X/mirage-freestanding/src/native/misc.c.depends

[pqwy]

This is fairly insane. How about

cat _build/src/native/hash/stubs.c.depends cc -MM -MG _build/src/native/hash/stubs.c cc -MM -MG _build/X/mirage-freestanding/src/native/hash/stubs.c ?

What about -M instead of -MM?

On my end, both gcc and clang do something like

stubs.o: _build/src/native/hash/stubs.c \
 _build/src/native/hash/../nocrypto.h \
 _build/src/native/hash/../hash/bitfn.h _build/src/native/hash/md5.h \
 _build/src/native/hash/sha1.h _build/src/native/hash/sha256.h \
 _build/src/native/hash/sha512.h

resp.

stubs.o: _build/X/mirage-freestanding/src/native/hash/stubs.c \
 _build/X/mirage-freestanding/src/native/hash/../nocrypto.h \
 _build/X/mirage-freestanding/src/native/hash/../hash/bitfn.h \
 _build/X/mirage-freestanding/src/native/hash/md5.h \
 _build/X/mirage-freestanding/src/native/hash/sha1.h \
 _build/X/mirage-freestanding/src/native/hash/sha256.h \
 _build/X/mirage-freestanding/src/native/hash/sha512.h

Your cc apparently has a minority opinion on what exactly constitutes "system header directories." It probably picks up one of the myriad package-specific md5.h files in the system include dirs, and this kills header dependency discovery. Can you confirm that all of the headers missing in your .c.depends, as compared to mine, do exist somewhere in system includes?

The insane bit is that it works for _build/src but fails for _build/X/.../src.

[hannesm]

On 29/10/2016 19:24, David Kaloper wrote:

cat _build/src/native/hash/stubs.c.depends

stubs.o: src/native/hash/stubs.c src/native/hash/../nocrypto.h \
  caml/mlvalues.h caml/bigarray.h src/native/hash/../hash/bitfn.h \
  src/native/hash/md5.h src/native/hash/sha1.h src/native/hash/sha256.h \
  src/native/hash/sha512.h

cc -MM -MG _build/src/native/hash/stubs.c

cstubs.o: _build/src/native/hash/stubs.c \
  _build/src/native/hash/../nocrypto.h caml/mlvalues.h caml/bigarray.h \
  _build/src/native/hash/../hash/bitfn.h _build/src/native/hash/md5.h \
  _build/src/native/hash/sha1.h _build/src/native/hash/sha256.h \
  _build/src/native/hash/sha512.h

cc -M -MG _build/src/native/hash/stubs.c

stubs.o: _build/src/native/hash/stubs.c \
  _build/src/native/hash/../nocrypto.h /usr/include/stdint.h \
  /usr/include/sys/cdefs.h /usr/include/sys/_types.h \
  /usr/include/machine/_types.h /usr/include/x86/_types.h \
  /usr/include/machine/_limits.h /usr/include/x86/_limits.h \
  /usr/include/machine/_stdint.h /usr/include/x86/_stdint.h \
  /usr/include/sys/_stdint.h caml/mlvalues.h caml/bigarray.h \
  _build/src/native/hash/../hash/bitfn.h /usr/include/sys/endian.h \
  /usr/include/machine/endian.h /usr/include/x86/endian.h \
  _build/src/native/hash/md5.h _build/src/native/hash/sha1.h \
  _build/src/native/hash/sha256.h _build/src/native/hash/sha512.h

cc -MM -MG _build/X/mirage-freestanding/src/native/hash/stubs.c ?

stubs.o: _build/X/mirage-freestanding/src/native/hash/stubs.c \
  _build/X/mirage-freestanding/src/native/hash/../nocrypto.h \
  caml/mlvalues.h caml/bigarray.h \
  _build/X/mirage-freestanding/src/native/hash/../hash/bitfn.h \
  _build/X/mirage-freestanding/src/native/hash/sha1.h

cc -M -MG _build/X/mirage-freestanding/src/native/hash/stubs.c ?

stubs.o: _build/X/mirage-freestanding/src/native/hash/stubs.c \
  _build/X/mirage-freestanding/src/native/hash/../nocrypto.h \
  /usr/include/stdint.h /usr/include/sys/cdefs.h \
  /usr/include/sys/_types.h /usr/include/machine/_types.h \
  /usr/include/x86/_types.h /usr/include/machine/_limits.h \
  /usr/include/x86/_limits.h /usr/include/machine/_stdint.h \
  /usr/include/x86/_stdint.h /usr/include/sys/_stdint.h caml/mlvalues.h \
  caml/bigarray.h \
  _build/X/mirage-freestanding/src/native/hash/../hash/bitfn.h \
  /usr/include/sys/endian.h /usr/include/machine/endian.h \
  /usr/include/x86/endian.h /usr/include/md5.h /usr/include/sys/md5.h \
  _build/X/mirage-freestanding/src/native/hash/sha1.h \
  /usr/include/sha256.h /usr/include/sys/types.h \
  /usr/include/sys/_pthreadtypes.h /usr/include/sys/select.h \
  /usr/include/sys/_sigset.h /usr/include/sys/_timeval.h \
  /usr/include/sys/timespec.h /usr/include/sys/_timespec.h \
  /usr/include/sha512.h

What I struggle with:

ls _build/src/native/hash/

_build/src/native/hash:
bitfn.h           md5.o             sha1.o            sha256.o
sha512.o
md5.c             sha1.c            sha256.c          sha512.c
stubs.c
md5.c.depends     sha1.c.depends    sha256.c.depends  sha512.c.depends
stubs.c.depends
md5.h             sha1.h            sha256.h          sha512.h
stubs.o

vs

ls _build/X/mirage-freestanding/src/native/hash/

_build/X/mirage-freestanding/src/native/hash:
bitfn.h           sha1.c            sha256.c          sha512.c.depends
md5.c             sha1.c.depends    sha256.c.depends  stubs.c
md5.c.depends     sha1.h            sha512.c          stubs.c.depends

[pqwy]

Ah, I get it.

cc is run over the .c files. Locally found headers take precedence over the system ones. Finally, the system ones are culled.

For src this is obvious, it finds includes and dumps them.

X/**/* are created on-demand. At the point when the headers are scanned for, there are none in there.

Here, it skips various instances of md5.h deeper in /usr/include, assumes this is generated, and spits them out.

While there, is resolves it to /usr/include and drops it from the list. This causes those to be missing from .depends and recursive rule never triggers their copying. Finally, the compilation breaks.

Lemme rethink header discovery in the presence of shadow-clones.....

[hannesm]

in the freestanding case, you might want to consider using the cflags provided via pkg_config by mirage-solo5... (on my installation, they are -nostdlibinc -ffreestanding -mno-red-zone -mno-3dnow -isystem ${includedir}/host -I${includedir} which avoids to convolute the include path with system headers). not sure whether this is a viable solution

[pqwy]

Better?

[hannesm]

different (opam info ocb-stubblr | grep pinned --> pinned: git (fa7f5afd):

ocaml-nocrypto] ocaml pkg/pkg.ml build --tests false --with-lwt true --freestanding true
ocamlfind ocamlopt -package unix -package ocamlbuild -linkpkg -package ocb-stubblr myocamlbuild.ml /home/hannes/.opam/mirage-solo5-ukvm/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
cc -MM -MG -MF src/native/misc.c.depends src/native/misc.c
cc -MM -MG -MF src/native/hash/stubs.c.depends src/native/hash/stubs.c
cc -MM -MG -MF src/native/hash/md5.c.depends src/native/hash/md5.c
cc -MM -MG -MF src/native/hash/sha1.c.depends src/native/hash/sha1.c
cc -MM -MG -MF src/native/hash/sha256.c.depends src/native/hash/sha256.c
cc -MM -MG -MF src/native/hash/sha512.c.depends src/native/hash/sha512.c
cc -MM -MG -MF src/native/aes/generic.c.depends src/native/aes/generic.c
cc -MM -MG -MF src/native/aes/aesni.c.depends src/native/aes/aesni.c
cc -MM -MG -MF src/native/des/generic.c.depends src/native/des/generic.c
ocamlfind ocamlc -g -ccopt '-DACCELERATE -msse2 -maes' -ccopt '--std=c99 -Wall -Wextra -O3' -ccopt '--std=c99 -Wall -Wextra -O3' -package bytes -package cstruct -c src/native/misc.c
mv misc.o src/native/misc.o
ocamlfind ocamlc -g -ccopt '-DACCELERATE -msse2 -maes' -ccopt '--std=c99 -Wall -Wextra -O3' -ccopt '--std=c99 -Wall -Wextra -O3' -package bytes -package cstruct -c src/native/hash/stubs.c
+ ocamlfind ocamlc -g -ccopt '-DACCELERATE -msse2 -maes' -ccopt '--std=c99 -Wall -Wextra -O3' -ccopt '--std=c99 -Wall -Wextra -O3' -package bytes -package cstruct -c src/native/hash/stubs.c
In file included from src/native/hash/stubs.c:3:
In file included from /usr/include/md5.h:44:
/usr/include/sys/md5.h:37:3: error: unknown type name 'u_int32_t'
  u_int32_t state[4];   /* state (ABCD) */
  ^
/usr/include/sys/md5.h:38:3: error: unknown type name 'u_int32_t'
  u_int32_t count[2];   /* number of bits, modulo 2^64 (lsb first) */
  ^
...

[pqwy]

Still?

[hannesm]

compiles fine now, thanks!