Closed adamsteen closed 2 years ago
That weird because I believed that this PR (included into 6.6) fixed the error: https://github.com/mirage/ocaml-freestanding/pull/99 can you check with ~master
? May be we did something wrong with the release.
@dinosaure whats the best way to pin ~master
, sorry i am more a c developer?
So if you have opam
(you can follow this documentation: https://opam.ocaml.org/doc/Install.html), a simple opam pin add git+https://github.com/mirage/ocaml-freestanding.git
will try to download and install the last version.
EDIT: I will try to reproduce on my side.
$ opam pin add git+https://github.com/mirage/ocaml-freestanding.git
[ocaml-freestanding] synchronised from git+https://github.com/mirage/ocaml-freestanding.git
[ocaml-freestanding.0.7.0] synchronised from git+https://github.com/mirage/ocaml-freestanding.git
ocaml-freestanding is now pinned to git+https://github.com/mirage/ocaml-freestanding.git (version 0.7.0)
The following actions will be performed:
- install ocaml-freestanding 0.7.0*
- recompile io-page 2.4.0 [uses ocaml-freestanding]
- recompile mirage-unix 4.0.1 [uses io-page]
===== 1 to install | 2 to recompile =====
Do you want to continue? [Y/n] y
<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[io-page.2.4.0] found in cache
[mirage-unix.4.0.1] found in cache
Processing 3/3: [ocaml-freestanding.0.7.0: git]
[ocaml-freestanding.0.7.0] synchronised from git+https://github.com/mirage/ocaml-freestanding.git
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ERROR] The compilation of ocaml-freestanding failed at "/usr/local/bin/gmake".
#=== ERROR while compiling ocaml-freestanding.0.7.0 ===========================#
# context 2.0.8 | openbsd/x86_64 | ocaml-base-compiler.4.13.1 | pinned(git+https://github.com/mirage/ocaml-freestanding.git#f1c86d2c)
# path ~/.opam/default/.opam-switch/build/ocaml-freestanding.0.7.0
# command /usr/local/bin/gmake
# exit-code 2
# env-file ~/.opam/log/ocaml-freestanding-84561-4197d2.env
# output-file ~/.opam/log/ocaml-freestanding-84561-4197d2.out
### output ###
# [...]
# checking for log2... yes
# checking for cbrt... yes
# checking for acosh... yes
# checking for asinh... yes
# checking for atanh... yes
# checking for erf... yes
# checking for erfc... yes
# checking for trunc... yes
# checking for round... yes
# checking for copysign... yes
# configure: error: C99 float ops unavailable, enable replacements with --enable-imprecise-c99-float-ops
# gmake: *** [Makefile:43: ocaml/Makefile.config] Error 1
<><> Error report <><><>+- The following actions failed
| - build ocaml-freestanding 0.7.0
+-
- No changes have been performed
[NOTE] Pinning command successful, but your installed packages may be out of sync.
ps off to bed now, will check back over the weekend
Ok, the error persist on OpenBSD so. I will try to figure out why. Thanks.
Ok it seems that fma
requires few symbols that they are not defined:
__guard_local
__stack_smash_handler
I will try to figure out why cc
does not find them when it seems that Solo5 defines them for us.
The s_fma.c
compiled with Debian 10 requires, at least, __stack_chk_fail
and __stack_chk_guard
which is probably expected due to -fstack-protector-strong
. It seems that on OpenBSD, the compilation does not produce the same file. Indeed, it expects, as I said, __guard_local
and __stack_smash_handler
. In that case, only solo5_hvt.o
defines them but, at this stage, we don't want to try to link with the Solo5 object file because:
1) it's a specific one for hvt
2) we should not use, at this stage, which "target" we want
The question is simple, why OpenBSD has a different behavior than Debian. One use clang.11.1.0
and the other uses gcc.10.3.0
but I don't think that such error appears on FreeBSD which uses clang
too. /cc @hannesm, can you give to us the result of nm s_fma.c.o
on your FreeBSD to compare with the OpenBSD one?
NOTE: I suspect that the next release of ocaml-freestanding
and the new layout/toolchain of Solo5 will fix the problem. To be clear, if Solo5/solo5#506 and mirage/ocaml-freestanding#92, I will not try to go further and let you to use the next release of MirageOS 4 đź‘Ť .
$ nm `find . -name s_fma.c.o`
0000000000000000 r .LCPI0_0
0000000000000000 r .LCPI0_1
0000000000000008 r .LCPI0_2
0000000000000010 r .LCPI0_3
U __isfinite
U __isnormal
U __stack_chk_fail
U __stack_chk_guard
U copysign
U feraiseexcept
0000000000000000 T fma
U frexp
U ilogb
U ldexp
U nextafter
In solo5 (bindings/GNUmakefile) I see:
ifeq ($(CONFIG_HOST), OpenBSD)
SSP_GUARD=__guard_local
SSP_FAIL=__stack_smash_handler
else
SSP_GUARD=__stack_chk_guard
SSP_FAIL=__stack_chk_fail
endif
Are these fine with OpenBSD's current system? Have they been renamed recently? Is there need for such a condition in freestanding? What I wonder is that solo5 compiles and test run fine on OpenBSD. Maybe time to setup an OpenBSD VM.
Yes those look correct, I haven’t heard of them changing! (see __stack_smash_handler search of the OpenBSD code on GitHub)
we do need these flags for OpenBSD on solo5
MAKECONF_CFLAGS="-mno-retpoline -fno-ret-protector -nostdlibinc"
MAKECONF_LDFLAGS="-nopie"
after reading hannesm post a few times, I expect we would need a condition like that.
edited: remove my response after not reading the whole thread, added too
I looked a bit deeper into this issue, and encountered the following:
The configure and build process we are doing in ocaml-freestanding is basically cross-compilation, but OCaml:
configure
script checks for -lm
and whether cos
is available (and adds -lm
to all configure testing scripts)-lm
is added before any LIBS/LDFLAGSecho '#define HAS_GETTIMEOFDAY' >> ocaml/runtime/caml/s.h echo '#define HAS_SECURE_GETENV' >> ocaml/runtime/caml/s.h echo '#define HAS_TIMES' >> ocaml/runtime/caml/s.h
in Makefile) to s.h -- OCaml's configure outputNow, fma
does function calls and thus needs the stack protection symbols. But nobody provides them.
Attempts to fix this: (a) let's use solo5 ldflags, similar to how cflags are used (and since OCaml's configure script is fine if executables can't be run, this is fine):
diff --git a/configure.sh b/configure.sh
index 4d154ed..c346628 100755
--- a/configure.sh
+++ b/configure.sh
@@ -24,6 +24,7 @@ fi
ocamlfind query ocaml-src >/dev/null || exit 1
FREESTANDING_CFLAGS="$(pkg-config --cflags ${PKG_CONFIG_DEPS})"
+FREESTANDING_LDFLAGS="$(pkg-config --ldlags ${PKG_CONFIG_DEPS})"
BUILD_ARCH="$(uname -m)"
BUILD_OS="$(uname -s)"
OCAML_BUILD_ARCH=
@@ -51,6 +52,7 @@ fi
cat <<EOM >Makeconf
FREESTANDING_CFLAGS=${FREESTANDING_CFLAGS}
+FREESTANDING_LDFLAGS=${FREESTANDING_LDFLAGS}
BUILD_ARCH=${BUILD_ARCH}
BUILD_OS=${BUILD_OS}
OCAML_BUILD_ARCH=${OCAML_BUILD_ARCH}
Result is the same on OpenBSD.
(b) let's remove -lm
from configure (manually where it says "LIBS = -lm $LIBS")
ubuntu (config.log):
configure:11148: checking for fma
configure:11148: cc -ffreestanding -fstack-protector-strong -nostdinc -mstack-protector-guard=global -isystem /home/administrator/.opam/4.12.0/lib/pkgconfig/../../include/solo5-bindings-hvt/c
rt -I/home/administrator/.opam/4.12.0/lib/pkgconfig/../../include/solo5-bindings-hvt/solo5 -I/home/administrator/ocaml-freestanding/nolibc/include -include _freestanding/overrides.h -I/home/a
dministrator/ocaml-freestanding/openlibm/include -I/home/administrator/ocaml-freestanding/openlibm/src -nostdlib -o conftest -ffreestanding -fstack-protector-strong -nostdinc -mstack-protect
or-guard=global -isystem /home/administrator/.opam/4.12.0/lib/pkgconfig/../../include/solo5-bindings-hvt/crt -I/home/administrator/.opam/4.12.0/lib/pkgconfig/../../include/solo5-bindings-hvt/
solo5 -I/home/administrator/ocaml-freestanding/nolibc/include -include _freestanding/overrides.h -I/home/administrator/ocaml-freestanding/openlibm/include -I/home/administrator/ocaml-freestan
ding/openlibm/src -L/home/administrator/ocaml-freestanding/openlibm/ conftest.c -lopenlibm >&5
/usr/bin/ld: warning: cannot find entry symbol _start; defaulting to 0000000000001020
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libopenlibm.a(s_fma.c.o): in function `fma':
s_fma.c:(.text+0x1c): undefined reference to `__stack_chk_guard'
/usr/bin/ld: s_fma.c:(.text+0x5a9): undefined reference to `__stack_chk_fail'
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libopenlibm.a(fenv.c.o): in function `fesetexceptflag':
fenv.c:(.text+0xb): undefined reference to `__stack_chk_guard'
/usr/bin/ld: fenv.c:(.text+0x5e): undefined reference to `__stack_chk_fail'
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libopenlibm.a(fenv.c.o): in function `feraiseexcept':
fenv.c:(.text+0x7e): undefined reference to `__stack_chk_guard'
/usr/bin/ld: fenv.c:(.text+0xaf): undefined reference to `__stack_chk_fail'
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libopenlibm.a(fenv.c.o): in function `feholdexcept':
fenv.c:(.text+0xdb): undefined reference to `__stack_chk_guard'
/usr/bin/ld: fenv.c:(.text+0x11c): undefined reference to `__stack_chk_fail'
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libopenlibm.a(fenv.c.o): in function `feupdateenv':
fenv.c:(.text+0x12f): undefined reference to `__stack_chk_guard'
/usr/bin/ld: fenv.c:(.text+0x16b): undefined reference to `__stack_chk_fail'
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libopenlibm.a(fenv.c.o): in function `feenableexcept':
fenv.c:(.text+0x17b): undefined reference to `__stack_chk_guard'
/usr/bin/ld: fenv.c:(.text+0x1d7): undefined reference to `__stack_chk_fail'
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libopenlibm.a(fenv.c.o): in function `fedisableexcept':
fenv.c:(.text+0x1eb): undefined reference to `__stack_chk_guard'
/usr/bin/ld: fenv.c:(.text+0x243): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status
configure:11148: $? = 1
Same on FreeBSD and OpenBSD. so the actual error is now exposed (and on all these systems now openlibm is used, not libm)
--> maybe our openlibm.a
should be named libm.a
? See https://github.com/hannesm/ocaml-freestanding/tree/libm for a branch (error on ubuntu below -- this is already when cos
is attempted to be linked with our -lm
being the openlibm
):
configure:10065: checking for cos in -lm
configure:10090: cc -ffreestanding -fstack-protector-strong -nostdinc -mstack-protector-guard=global -isystem /home/administrator/.opam/4.13.1/lib/pkgconfig/../../include/solo5-bindings-hvt/crt -I/home/administrator/.opam/4.13.1/lib/pkgconfig/../../include/solo5-bindings-hvt/solo5 -I/home/administrator/ocaml-freestanding/nolibc/include -include _freestanding/overrides.h -I/home/administrator/ocaml-freestanding/openlibm/include -I/home/administrator/ocaml-freestanding/openlibm/src -nostdlib -o conftest -ffreestanding -fstack-protector-strong -nostdinc -mstack-protector-guard=global -isystem /home/administrator/.opam/4.13.1/lib/pkgconfig/../../include/solo5-bindings-hvt/crt -I/home/administrator/.opam/4.13.1/lib/pkgconfig/../../include/solo5-bindings-hvt/solo5 -I/home/administrator/ocaml-freestanding/nolibc/include -include _freestanding/overrides.h -I/home/administrator/ocaml-freestanding/openlibm/include -I/home/administrator/ocaml-freestanding/openlibm/src -L/home/administrator/ocaml-freestanding/openlibm/ conftest.c -lm >&5
/usr/bin/ld: warning: cannot find entry symbol _start; defaulting to 0000000000001020
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libm.a(s_cos.c.o): in function `cos':
s_cos.c:(.text+0xc): undefined reference to `__stack_chk_guard'
/usr/bin/ld: s_cos.c:(.text+0x127): undefined reference to `__stack_chk_fail'
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libm.a(e_rem_pio2.c.o): in function `__ieee754_rem_pio2':
e_rem_pio2.c:(.text+0x23): undefined reference to `__stack_chk_guard'
/usr/bin/ld: e_rem_pio2.c:(.text+0x474): undefined reference to `__stack_chk_fail'
/usr/bin/ld: /home/administrator/ocaml-freestanding/openlibm//libm.a(k_rem_pio2.c.o): in function `__kernel_rem_pio2':
k_rem_pio2.c:(.text+0x26): undefined reference to `__stack_chk_guard'
/usr/bin/ld: k_rem_pio2.c:(.text+0x10ea): undefined reference to `__stack_chk_guard'
/usr/bin/ld: k_rem_pio2.c:(.text+0x134c): undefined reference to `__stack_chk_fail'
collect2: error: ld returned 1 exit status
configure:10090: $? = 1
But how to fix this situation now? Should we avoid the (not-really-cross-compilation-safe) OCaml configure script? Should we attempt to snitch in an object that defines the stack protector symbols? Should we patch OCaml's configure to behave slightly differently?
Somehow, we need to convince configure to use crt.o for its cc invocations -- the crt.o from solo5 bindings/crt.c... But I'm not sure how to do this. At least the above investigation shows how to reproduce the (very valid) error on Linux systems.
Unfortunately, the problem persists with Solo5/solo5#506 and #92. I will try to find a good solution from these PRs - we are close to make a release of them. Thanks for your feedback @hannesm, we should indeed be aware about -lm
and our libopenlibm.a
- we should have a way to link with the right crt.o
with the new toolchain.
The older solo5-bindings-*.0.6.9
toolchain expects an explicit choice of the target (solo5-bindings-hvt
, -virto
, etc.). So, even if it's possible to bring one crt.o
for the configure
, it's not the accurate one used then for the mirage configure -t ...
. The new toolchain give to us a ARCH-solo5-none-static-ld
which has a new option -z
to permit us to link object files with a fake Solo5 unikernel (the stub one). By this way, and if we update the configure
on the right way, we will be able to bring for the ocaml/configure
a fake crt.o
and fix the initial error.
In #102, the ldflags are passed through. This results in OCaml's configure (conftest) attempting to link an executable, and missing the solo5_app_main
symbol. I guess it could work with solo5 0.7 -- for the 0.6 series, maybe we need a stub.o
that provides this symbol?
I updated #102 which works fine for me on FreeBSD (I only tested ocaml-freestanding, not to compile a unikernel) -- does this also work on OpenBSD and Linux? Does it allow to assemble unikernels that work? The "trick" is to define a weak solo5_app_main symbol in sysdeps_solo5.c, and OCaml's configure is able to generate executables :)
On my side, I found a solution for OpenBSD 7 and the new Solo5 toolchain (and #92), this is the diff (mostly inspired from your patch):
diff --git a/Makefile b/Makefile
index a305dca..4220a8a 100644
--- a/Makefile
+++ b/Makefile
@@ -7,7 +7,8 @@ all: openlibm/libopenlibm.a nolibc/libnolibc.a ocaml ocaml-freestanding.pc frees
TOP=$(abspath .)
# CFLAGS used to build nolibc / openlibm / ocaml runtime
-LOCAL_CFLAGS=$(MAKECONF_CFLAGS) -I$(TOP)/nolibc/include -include _freestanding/overrides.h
+LOCAL_CFLAGS=$(MAKECONF_CFLAGS) -I$(TOP)/nolibc/include -include _freestanding/overrides.h \
+ -L$(TOP)/openlibm/ -lopenlibm
# CFLAGS used by the OCaml compiler to build C stubs
GLOBAL_CFLAGS=$(MAKECONF_CFLAGS) -I$(MAKECONF_PREFIX)/freestanding-sysroot/include/nolibc/ -include _freestanding/overrides.h
# LIBS used by the OCaml compiler to link executables
The error is a bit different that ocaml-freestanding.0.6.6
where ld.ldd
tries to bring the host libm.a
which uses some undefined symbols from the Solo5 toolchain. However, -lm
is added by the ocaml/configure.sh
in anyway (and I don't know how to remove it properly (@dra27, do you have any clues about that?)). In that patch we use a behavior of ld.ldd
which will take functions from the first archive which defines them - and, in your case, libopenlibm.a
.
I will try your PR @hannesm to produce an unikernel and if it's working, we will cut a release. Then, I will integrate my fragile patch into #92 and we will be ready to cut Solo5.0.7.0 and ocaml-freestanding.0.7.0 🎉 !
EDIT: that mostly means that we have two errors:
1) it's about the missing crt.o
which is given now by #102 via the FREESTANDING_LDFLAGS
. However. this specific error is solved by the new Solo5 toolchain which takes the stub unikernel by default (with needed symbols)
2) only since OCaml 4.13, ocaml/configure.sh
puts in anyway its own -lm
. The linker then takes the host's libm.a
and its symbols instead of our libopenlibm.a
. The host's libm.a
can use some undefined symbols from our toolchain and it's why the ocaml/configure.sh
fails specially when it tests availability of math functions.
only since OCaml 4.13,
ocaml/configure
puts in anyway its own-lm
According to git, this is the case at least since OCaml 4.08 (when the switch to autoconf happened). So, my suggestion here is to carry in freestanding a patch for OCaml's configure to remove -lm
testing. Then there's no need for any renaming.
@adamsteen with a deep look on Solo5/ocaml-freestanding with @hannesm, we agreed that the current toolchain proposed by Solo5 and used by ocaml-freestanding
is not good enough to cross-compile OCaml. I would like to say that what we try to do in this project is not really expected by the OCaml core team and we still need to hack on the configure
given by the OCaml distribution to be able to plug the underlying layer Solo5 as we want - the problem is not really about Solo5 but the lack of cross-compilation support of OCaml.
The problem triggered by OpenBSD 7 is about the libm.a
required by OCaml (in anyway) where, on our side, we want to use our libopenlibm.a
to compile then the caml runtime and use it then to compile an OCaml program.
The initial problem is the configure
which tries to compile some C codes to check the availability of some functions. However, libopenlibm.a
and libnolibc.a
expects a certain context of compilation to be able then to link them. Specially it expects crt.o
given by Solo5 which provides missing symbols about the stack. For instance: __stack_chk_fail
and __stack_chk_guard
.
We were disturb for a long time by the GNU's linker which seems "smarter"/"unpredictable" than ld.ldd
. Indeed, it seems that the GNU's linker is able to take libm.a
even if libopenlibm.a
fails because the first one works... This is what @hannesm discovered when he deleted -lm
from the OCaml's configure
to explicitly say to it that we must want to use our libopenlibm.a
. In that case, on Linux, we failed as we do on OpenBSD 7.
If we go further on this way and be sure to use libopenlibm.a
, we must provide the required "context" needed by libopenlibm.a
and proposed by Solo5. However, Solo5.0.6.9 does not provide a Solo5 distribution but multiple "bindings" and each of them is a specialization for a specific target (muen
, hvt
, xen
, etc.).
So, if we go further, we should add some option at the link time for any programs needed for the OCaml's configure
to link with Solo5. And, according to Solo5.0.6.9, that mostly means: to link with one of the target proposed by Solo5. The initial and default one is solo5-bindings-hvt
.
Now, we got an other problem. solo5-bindings-hvt
exports specific things and others targets exports some others different specific things needed for their deployment. For instance, virtio
/muen
/xen
exports __solo5_mft1_note
but hvt
does not exports it. If we took by default solo5-bindings-hvt
to compile OCaml - and say that every OCaml program should follows the same compilation schema - we already did a choice about the target produced at the end by mirage
(even if we did mirage configure -t virtio
for instance).
So, finally, for the only purpose to be able to compile some C programs to help the configure
to discover what is available or not, we need a fake underlying layer which can be replaced then by solo5-bindings-{hvt,virtio,muen,...}
. At this stage, the problem becomes a huge problem.
If I spent my time to explain all of this, it's to say that the next release of Solo5.0.7.0 will fix the problem via a new toolchain which is able to provide a fake underlying layer for us and specially for the OCaml's configure
script. In others words, try to fix this specific issue seems close to redo what Solo5.0.7.0 already does.
Finally, given this description, the way to fix it and the current situation: we are not able to spend more times on ocaml-freestanding.0.6.6
. We found a solution (see #102) but it's an hacky one and we are not sure that the result works for everybody (even if it's work for OpenBSD, we are not sure about Linux support - and we already did this mistake because we believed that ocaml-freestanding
worked for Linux and others platforms but you prove that it's not the case 🙂 ). The best for all is to prepare surely with the OpenBSD 7 support ocaml-freestanding.0.7.0
.
So what do you think @adamsteen about this solution?
Indeed, #102 is a hacky solution for this issue in MirageOS 3 land -- unclear whether it is worth to go down this path.
In MirageOS 4 land (with solo5 https://github.com/Solo5/solo5/pull/506) and the instructions on https://next.mirage.io/docs/mirage-4 (plus an "opam pin add ocaml-freestanding.0.7.0 git+https://github.com/hannesm/ocaml-freestanding.git#cross-compiler-take3" which adds #104), this should now work on OpenBSD 7.0 :)
Since MirageOS 4 is close to a release, and we would like to deprecate MirageOS 3, I think it is best to focus on MirageOS 4 and get that out of the door (including OpenBSD support) rather sooner than later.
@dinosaure and @hannesm i don’t think it worth fixing this for mirage 3, I don’t use it for production and I expect I am the only one using it on OpenBSD.
I will test ocaml-freestanding 0.7.0 (as above) as soon as I have a few moments!
just so I am clear, this is a real bug not just a OpenBSD being different?
just so I am clear, this is a real bug not just a OpenBSD being different?
No, it is an actual issue that is (for some reason) only exposed on OpenBSD: OCaml's configure script inserts a -lm
for feature detection (which links the host system math library - MirageOS uses openlibm (vendored in this repository)). The current (released) way on how configure is used here is pretty incomplete. The fix for this issue is:
sed -i -e 's/ac_cv_lib_m_cos=yes/ac_cv_lib_m_cos=no/' ocaml/configure
Which sets the -lm
detection always to no
.
With MirageOS 4, the OCaml configure will use a cross toolchain that is able to link executables (maybe even execute them) - solo5 )0.7.0 - not yet released) provides a stub target for this. With the released ocaml-freestanding, as the cross-chain is incomplete, the fix is more involved (also trying to build complete executables with solo5 ldflags, and providing missing symbols).
I will test ocaml-freestanding 0.7.0 (as above) as soon as I have a few moments!
I tried to compile as above, but was getting unable to resolve dependency errors, not built with dune, probably user error, will try again in the coming days and report back!
I am unable to test this atm, as when I try make depends with the mirage next instructions, gmake depend
Fails with many things not built with dune
according to #102 comment at the bottom, the fixes of #105 (and 0.7.0+ series) fix this issue. I'll close this issue, if it persists in newer (MirageOS 4+) environments, please report and re-open. Thanks.
if there is anything else you need or i can help with please let me know