Update Libc

[ehmry]

The current port of the FreeBSD libc lacks support for AArch64, therefore it must be replaced.

I've been working on porting FreeBSD 12.0, and I've gotten as far as running the basic run/libc test. The problem with more complicated scenarios like Noux+GNU coreutils is a thread-local-locale quagmire that seems to depend on either native TLS or EmuTLS from GCC.

For what its worth, Musl just puts a locale pointer on the Pthread struct: https://git.musl-libc.org/cgit/musl/tree/src/internal/locale_impl.h#n38

Also, FreeBSD has an indirection table now for some of its calls, which means defining up three symbols for some calls (close(), _close(), __sys_close()).

As an intermediate step I ported OpenLibm to replace the FreeBSD libm, which I think will be easier to maintain. Licensing is about the same, and its compatible with both FreeBSD 8.2 and 12.0.

[nfeske]

Thanks for the interesting update. Cool that the first libc-using programs are already resurrected. ;-)

[ehmry]

FreeBSD now supports the xlocale thread-local-locales and implements this with thread_local or __thread keywords, depending on whether a C or C++ compiler is in use.

[ehmry]

... unless you are using RISC-V https://svnweb.freebsd.org/base/release/12.0.0/sys/sys/cdefs.h?view=markup#l777.

[ehmry]

I have two patches that pass the depot autopilot test, and can be considered for staging, 044366a87c564232a933e1f5202772df54750bd8 and 14049fca28c94e6464e78f7a5aa58d534783c204.

[ehmry]

I have rebase my libc commits to staging, now they are e4dc4e547b640e2eb76aa0e80b3765dfbb027086, f2666054b678acf763ef2a813a20db129ac25b4f, and cad22cb2d635ef8477fe8c636a53b0b0d37fdee0.

I think they are ready for staging, depot_autopilot, noux, qt, and virtualbox tests are passing.

[cproc]

When running 'make LIB=libm' without having the openlibm port prepared, the following error is reported:

checking library dependencies...
/home/.../repos/libports/lib/mk/libm.inc:40: src/lib/openlibm/src/Make.files: No such file or directory
/home/.../repos/libports/lib/mk/libm.inc:40: src/lib/openlibm/amd64/Make.files: No such file or directory
/home/.../repos/libports/lib/mk/libm.inc:40: src/lib/openlibm/bsdsrc/Make.files: No such file or directory
/home/.../repos/libports/lib/mk/libm.inc:40: src/lib/openlibm/ld80/Make.files: No such file or directory
make[1]: *** No rule to make target 'src/lib/openlibm/ld80/Make.files'.  Stop.
make[1]: *** No rule to make target 'libm.lib', needed by 'all'.  Stop.
Makefile:282: recipe for target 'gen_deps_and_build_targets' failed
make: *** [gen_deps_and_build_targets] Error 2

Instead it should say:

Error: Ports not prepared or outdated:
  openlibm

triggered by 'select_from_ports'. When commenting out line 40, the missing port detection works, so maybe line 40 is evaluated too early or something like that.

Also, the openlibm port hash needs to be updated from 'dummy'.

[nfeske]

Shouldn't openlibm be prepared as part of the libc.port?

[cproc]

'make LIB=libutil' fails with the following error:

    MERGE    libutil.lib.a
genode-x86-ar: logout.o: No such file or directory

It looks like 'logout.c' was previously part of the libc contrib code. I couldn't find out where 'libutil' is used, though ('libutil.mk' mentions 'libinetutils', but grepping for that didn't show anything), so maybe 'libutil.mk' can be removed?

[cproc]

'make LIB=e2fsprogs' fails with the following error:

    COMPILE  lib/blkid/getsize.o
In file included from /home/.../contrib/e2fsprogs-lib-7302da4fea6929fd6c2e4b4c70f81b20b9dd7d38/src/lib/e2fsprogs/lib/blkid/getsize.c:34:
/home/.../contrib/libc-d513432517dda0a02558634aff6cb428ee37a988/include/libc/sys/disklabel.h:43:10: fatal error: sys/disk/bsd.h: No such file or directory
 #include <sys/disk/bsd.h>
          ^~~~~~~~~~~~~~~~

[cproc]

'make app/lighttpd' fails with the following error:

    LINK     lighttpd
fdevent.o: In function `fdevent_accept_listenfd':
/home/.../contrib/lighttpd-0665eee22a27970bb977c93abdea7e89d3011a1d/src/app/lighttpd/src/fdevent.c:580: undefined reference to `accept4'

[cproc]

'make app/openvpn' fails with the following error:

    COMPILE  route.o
In file included from /home/.../contrib/openvpn-c6b6a601bdd5ef3e92c9e367a5e3d932478f2652/src/app/openvpn/src/openvpn/route.c:2606:0:
/home/.../contrib/libc-d513432517dda0a02558634aff6cb428ee37a988/include/libc/net/route.h:38:25: fatal error: sys/counter.h: No such file or directory
 #include <sys/counter.h>

[ehmry]

E2fsprogs needs an update for compatiblity.

[cproc]

'make LIB=stdcxx' for x86_32 fails with error messages mentioning 'size_t' mismatches:

/home/.../contrib/stdcxx-912a4049e19eb70678485ad579b684b1e118da1d/include/stdcxx/bits/basic_string.h:5512:21: note:   mismatched types ‘std::size_t {aka long unsigned int}’ and ‘size_t {aka unsigned int}

It looks like 'types.patch' needs to be adapted to the new libc version.

[cproc]

I'm working on the 'types.patch' update.

[cproc]

Commit 1fdbc64 updates the size types patch.

[cproc]

'make test/ieee754' on x86_32 fails with the following error:

    LINK     test-ieee754
tst-ieee754.o: In function `fesetround':
/home/.../contrib/libc-92c7163f1ba6457e90067551f77e8ba9363cc3fc/include/spec/x86/libc/fenv.h:232: undefined reference to `__has_sse'
/home/.../contrib/libc-92c7163f1ba6457e90067551f77e8ba9363cc3fc/include/spec/x86/libc/fenv.h:232: undefined reference to `__test_sse'

[cproc]

'make LIB=libc' for pbxa9 fails with the following error:

    COMPILE  strcasecmp.o
In file included from /home/.../contrib/libc-92c7163f1ba6457e90067551f77e8ba9363cc3fc/src/lib/libc/lib/libc/locale/xlocale_private.h:42:0,
                 from /home/.../contrib/libc-92c7163f1ba6457e90067551f77e8ba9363cc3fc/src/lib/libc/lib/libc/string/strcasecmp.c:45:
/home/.../contrib/libc-92c7163f1ba6457e90067551f77e8ba9363cc3fc/include/spec/arm/libc/machine/atomic.h:53:31: fatal error: machine/atomic-v6.h: No such file or directory
 #include <machine/atomic-v6.h>

[ssumpf]

@ehmry: Could you add f300b19 and 765449d to your freebsd-update branch, they enable ARM64 support for libc and libm? This way the changes will be in one place.

[ehmry]

I had some issues with the FreeBSD resolver that were avoided by implementing poll at the Libc plugins (32c0cf55b06d4fe287f5fb3a16c1b0e55e9ff05d). I suspect this is related to the fetchurl connect issue, but this not a proper fix for that.

[ehmry]

Squashed to f8a17501e7cf1d79bdb4c41be1cbfea8b7d41c57 and cacb189be3e9c2b7e4cb9f023518b1141ac71cda.

[ehmry]

All of the labor intensive work on the libc itself should be done, what remains is testing and fixups or updates to components that use libc.

My biggest uncertainty is strong and weak symbol interposing, so if close, __sys_close, or __libc_close used, and whether or not these sorts of symbols are overridden properly by Noux. I think I've caught them all, but I suspect there may be one or two problem symbols lurking in signal handing.

As mentioned before, VirtualBox, Qt, and stdc++ are working, e2fs-utils are not.

[ehmry]

Fix for OpenVPN at 08c5aa10d81d1763b5e5c9d357c3b74eaa189c56

[chelmuth]

I got

contrib/lighttpd-0665eee22a27970bb977c93abdea7e89d3011a1d/src/app/lighttpd/src/fdevent.c:580: undefined reference to `accept4'

just now (and was building pbxa9 which may not matter).

[ehmry]

08c5aa10d81d1763b5e5c9d357c3b74eaa189c56 adds accept4 to the libc symbols, which fixes it for me.

[ehmry]

@chelmuth patch for e2fsprogs at fa2e5b96a2e75ee63ce03706b963d9d40e052e00.

[chelmuth]

I merged the series to staging now. :crossed_fingers:

[ehmry]

I have a patch at b1d0d6cbc9ddc04739dad29c23057508dfbdb871 to fix usleep and a patch at
ba59027584c548c3c308b3bff050737a54821cea to fix Seoul.

I will now try to replace the GNU libiconv with the FreeBSD libiconv, unless anyone has a reason not to.

[cnuke]

On reason would be reliance on GNU extension in libiconv that are not provided in FreeBSD's version or the fact that we have older contrib sources that do not know how to use the FreeBSD's version. But either way, go for it and see what breaks.

[ssumpf]

@chelmuth: 920c76c adjusts the ieee754 run script to the updated libc/libm.

[chelmuth]

We have libiconv mostly for Qt and VirtualBox, with the latter being more risky with your envisioned replacement. What speaks against staying with the GNU version?

[ehmry]

Iconv fix at 3f11d6dd3f83e3587ee6524d2c3d7a66da0f6158.

[ssumpf]

@chelmuth: https://github.com/ssumpf/genode-world/commit/693066bc6da8e0bcd1f807c37411be3f464dfa94 fixes JDK (arm/x86) for the current libc/libm in world

[cnuke]

@chelmuth 997e499 and ed1b27d will make the wifi_drv work again.

[ehmry]

I have a fix for the libc_setjmp api package at 57379e2b17a0a99cb049cf62f3640794b983e406. This package is however not in use since 953c62c04c66e2c8b5b7a87174cb17d359bd4d67.

[chelmuth]

If this is the case, we should schedule the library for removal after the release.

[ehmry]

I have three more patches for the libc, 66c2307b41a18ec215f31975c75b8d1ff39bd75d, 281e4f556c9a3ffb4a7b92edc61c39944df3353e, 64dc7b02ccca3d33643afb973c124e2e41c679d0. This fixes the missing _execve symbol and Noux signal symbol overrides.

[cnuke]

0e86a1f fixes the problem that was reported by @trimpim on the mailing list today.

The updated libc sets __FreeBSD__ unconditionally, which leads to contrib source trying to do the right thing.

EDIT: as did the old one apparently. That leaves my somewhat puzzled why it worked before.

[ehmry]

@chelmuth arc4random fixup at 3aa3cc0b5117f057f1f39523b9aa2b71a89cb05e.

[nfeske]

@ehmry Can you please create a fixup for including the libm port into the libc.port? This way the switch to the openlibm (and a future switch back to FreeBSD's libm) will be transparent for users.

[ehmry]

@chelmuth signal patch at cf4fddb4da9c18140821450f6a082c91e17ee28f.

[ehmry]

Libc/libm port merge patch at ab6994f9f03547409c3c3c6d9487bc30fbf77c71.

[ehmry]

Patch to silence madvise at 6794e796d514a7f8fc287415128bc24535173ee8.

[nfeske]

Thanks @ehmry. Merged https://github.com/genodelabs/genode/commit/6794e796d514a7f8fc287415128bc24535173ee8 to staging.

[ehmry]

I have a patch to fix some issues with the toolchain and arc4random/getrandom/getentropy at ~2ad30e332bc406f4aee7f66d04af02a96e2a5365~ 0a881b795f4196e3cc80507126a84903c9e10577.

[cnuke]

@chelmuth please merge b690a3f to staging.

[cnuke]

If we go with the vanilla arc4random implementation we need the minherit symbol (see eecfcd6). It stands to reason if returing 0 instead of -1 is the proper choice.

[ehmry]

A random/entropy backend with a simple fallback at 3a4d35ee2d9dd79b477ff692c1f27ee25372079a.

[ehmry]

As found by @cproc, I have a patch at 37a281ad6f9204bcef175ae8954e09a2d104d781 to restore poll wakeup behavior on the libc_pipe plugin, this should fix issues with smartcards and the ssh_terminal.

[nfeske]

Thanks @ehmry. I merged https://github.com/genodelabs/genode/commit/37a281ad6f9204bcef175ae8954e09a2d104d781 to staging.

[ehmry]

Patch to fix Vbox at 9733817766dcc133e1be44c507aecdfc194027ab (@alex-ab).