Tracking the Progress of Compiling Glibc into WebAssembly (WASM)

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(gconv_dl.o): undefined symbol: __libc_dlopen_mode wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(gconv_dl.o): undefined symbol: __libc_dlsym wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(gconv_dl.o): undefined symbol: __libc_dlsym wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(gconv_dl.o): undefined symbol: __libc_dlsym wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(gconv_dl.o): undefined symbol: __libc_dlclose wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_mutex_unlock.o): undefined symbol: __lll_unlock_elision wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __futex_abstimed_wait_cancelable64

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(gconv_dl.o): undefined symbol: __libc_dlopen_mode This error is fixed by disabling the // args.caller_dlopen = RETURN_ADDRESS (0);

[rennergade]

Yeah it seems like we need to figure out either how to make the dynamic loader work earlier than I expected or probably better figure out why its trying to use it.

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(dl-debug.o): undefined symbol: _r_debug_extended This error is fixed by converting dl-debug-symbols.S to dl-debug-symbols.c

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(rtld_static_init.o): undefined symbol: __dlopen
wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(rtld_static_init.o): undefined symbol: __dlsym
wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(rtld_static_init.o): undefined symbol: __dlvsym
wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(rtld_static_init.o): undefined symbol: __dlmopen Above error are fixed by #define RETURN_ADDRESS(nr) (NULL) in /glibc/include/libc-symbols.h

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(dl-reloc.o): undefined symbol: _dl_lookup_symbol_x This error is fixed by changing int gscope_flag; to _Atomic int gscope_flag; in /glibc/sysdeps/i386/nptl/tls.h

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(ldbl2mpn.o): undefined symbol: __clz_tab This error is fixed by adding implementation of unsigned char __clz_tab[] in /glibc/sysdeps/i386/mp_clz_tab.c

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(stat64.o): undefined symbol: __fstatat64_time64 This error is fixed by disabling fstatat64_time64_stat in /glibc/sysdeps/unix/sysv/linux/fstatat64.c for now

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_mutex_unlock.o): undefined symbol: __lll_unlock_elision wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __futex_abstimed_wait_cancelable64 wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_rwlock_rdlock.o): undefined symbol: __futex_abstimed_wait64 wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_rwlock_rdlock.o): undefined symbol: __futex_abstimed_wait64 wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __futex_abstimed_wait_cancelable64 wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_rwlock_rdlock.o): undefined symbol: __futex_abstimed_wait64 wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_rwlock_rdlock.o): undefined symbol: __futex_abstimed_wait64 wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_mutex_lock.o): undefined symbol: __lll_lock_elision wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_mutex_lock.o): undefined symbol: __futex_lock_pi64 wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_rwlock_rdlock.o): undefined symbol: __futex_abstimed_wait64 wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(fileops.o): undefined symbol: __open Above are the only errors we have; all other issues have been fixed.

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __futex_abstimed_wait_cancelable64 This error is fixed by disabling INTERNAL_SYSCALL_CANCEL for futex_time64 in /glibc/nptl/futex-internal.c

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_rwlock_rdlock.o): undefined symbol: __futex_abstimed_wait64 This error is fixed by disabling INTERNAL_SYSCALL_CANCEL for futex (__futex_abstimed_wait_common32) in /glibc/nptl/futex-internal.c

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(fileops.o): undefined symbol: __open This error is fixed by changing the implementation of __libc_open to return MAKE_SYSCALL(10, "syscall|open", (uint64_t) file, (uint64_t) oflag, (uint64_t) mode, NOTUSED, NOTUSED, NOTUSED); in /glibc/sysdeps/unix/sysv/linux/open.c

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_mutex_unlock.o): undefined symbol: __lll_unlock_elision This error is fixed by disabling _xend() ''hardware unlock'' in /glibc/sysdeps/unix/sysv/linux/x86/elision-unlock.c

[yzhang71]

wasm-ld: error: ../../glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_mutex_lock.o): undefined symbol: __lll_lock_elision This error is fixed by disabling xbegin() /glibc/sysdeps/unix/sysv/linux/x86/elision-lock.c

[yzhang71]

Now, both the pthread test case and the hello-world test case can be compiled successfully.

[yzhang71]

When running pthread test case, we encountered the error of:

../../wasmtime/target/debug/wasmtime run --wasi threads=y --wasi preview2=n thread.wasm 
Error: WebAssembly translation error

Caused by:
    Invalid input WebAssembly code at offset 260453: type mismatch: values remaining on stack at end of block

[yzhang71]

When running printf test case, we encountered the error of:

Error: failed to run main module `hello.wasm`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: <unknown>!memcpy
           1: 0x21840 - <unknown>!memcpy
           2: 0x21840 - <unknown>!memcpy
           ...
           6526: 0x21863 - <unknown>!memcpy
           6527:  0x65f - <unknown>!__printf_buffer_write
           6528: 0x955b - <unknown>!__printf_buffer
           6529: 0xc5b8 - <unknown>!__vfprintf_internal
           6530: 0xc72e - <unknown>!__printf
           6531:  0x4ee - <unknown>!__original_main
           6532:  0x4a7 - <unknown>!_start
           6533: 0x3d038 - <unknown>!_start.command_export
       note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
    2: wasm trap: call stack exhausted

[yzhang71]

The previous printf error has been fixed by re-implementing BYTE_COPY_FWD and BYTE_COPY_BWD in c (/glibc/sysdeps/i386/memcopy.h)

#undef BYTE_COPY_FWD
#define BYTE_COPY_FWD(dst_bp, src_bp, nbytes)                     \
  do                                          \
    {                                         \
      size_t __nbytes = (nbytes);                         \
      if (__nbytes & 1)                               \
        {                                     \
      ((byte *) dst_bp)[0] =  ((byte *) src_bp)[0];               \
      src_bp += 1;                                \
      dst_bp += 1;                                \
      __nbytes -= 1;                              \
        }                                     \
      while (__nbytes > 0)                            \
    {                                     \
      byte __x = ((byte *) src_bp)[0];                    \
      byte __y = ((byte *) src_bp)[1];                    \
      src_bp += 2;                                \
      __nbytes -= 2;                              \
      ((byte *) dst_bp)[0] = __x;                         \
      ((byte *) dst_bp)[1] = __y;                         \
      dst_bp += 2;                                \
    }                                     \
    } while (0)

/* Copy exactly NBYTES_TO_COPY bytes from SRC_END_PTR to DST_END_PTR,
   beginning at the bytes right before the pointers and continuing towards
   smaller addresses.  Don't assume anything about alignment of the
   pointers.  */
#undef BYTE_COPY_BWD
#define BYTE_COPY_BWD(dst_ep, src_ep, nbytes)                     \
  do                                          \
    {                                         \
      size_t __nbytes = (nbytes);                         \
      if (__nbytes & 1)                               \
        {                                     \
      src_ep -= 1;                                \
      dst_ep -= 1;                                \
      ((byte *) dst_ep)[0] =  ((byte *) src_ep)[0];               \
      __nbytes -= 1;                              \
        }                                     \
      while (__nbytes > 0)                            \
    {                                     \
      byte __x, __y;                              \
      src_ep -= 2;                                \
      __y = ((byte *) src_ep)[1];                         \
      __x = ((byte *) src_ep)[0];                         \
      dst_ep -= 2;                                \
      __nbytes -= 2;                              \
      ((byte *) dst_ep)[1] = __y;                         \
      ((byte *) dst_ep)[0] = __x;                         \
    }                                     \
    } while (0)

[yzhang71]

Now we have the error of:

Error: failed to run main module `hello.wasm`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: 0x354e6 - _IO_doallocbuf
                           at /home/dennis/Documents/lind-wasm/glibc/libio/genops.c:347:9
           1: 0x3a453 - _IO_new_file_overflow
                           at /home/dennis/Documents/lind-wasm/glibc/libio/fileops.c:745:4
           2: 0x3b071 - _IO_new_file_xsputn
                           at /home/dennis/Documents/lind-wasm/glibc/libio/fileops.c:1244:11
           3: 0x612a - __printf_buffer_flush_to_file
                           at /home/dennis/Documents/lind-wasm/glibc/stdio-common/printf_buffer_to_file.c:59:20
           4: 0x62a1 - __printf_buffer_to_file_done
                           at /home/dennis/Documents/lind-wasm/glibc/stdio-common/printf_buffer_to_file.c:120:3
           5: 0xc5c5 - __vfprintf_internal
                           at /home/dennis/Documents/lind-wasm/glibc/stdio-common/vfprintf-internal.c:1550:14
           6: 0xc730 - __printf
                           at /home/dennis/Documents/lind-wasm/glibc/stdio-common/printf.c:33:10
           7:  0x4f0 - main
                           at /home/dennis/Documents/lind-wasm/lind-wasm-tests/hello-world/hello.c:4:4
           8:  0x4a9 - <unknown>!_start
           9: 0x3d36d - <unknown>!_start.command_export
    2: wasm trap: uninitialized element

[yzhang71]

The above error is due to WebAssembly not correctly handling the vtable, causing the function pointer to point to a wild destination and resulting in a segmentation fault. However, printf finally works, but we still need to take care of the vtable. Now, we use #define _IO_XSPUTN(FP, DATA, N) _IO_file_xsputn(FP, DATA, N) to directly call _IO_file_xsputn, skipping the vtable in /glibc/libio/libioP.h:177

@yizhuoliang Hi Coulson, Nick finds a commit relate to libioP.h (https://github.com/Lind-Project/glibc/commit/13b865cd370df130eee35777152673c76acb4f17) and will this affect the vtable in wasm?

[JustinCappos]

Weird. Do other function pointers work normally?

[yzhang71]

Yes, function pointer works normally, below is the testcase I just tested:

#include <stdio.h>
#include <unistd.h>
#include <string.h>

typedef void (*write_message_func_ptr)(const char *);

void do_write(const char *message) {
    size_t length = strlen(message);
    ssize_t bytes_written = write(STDOUT_FILENO, message, length);
}

void call_write_function(write_message_func_ptr write_func, const char *message) {
    write_func(message);
}

int main() {
    const char *message = "Hello, World!\n";

    // Define a function pointer and assign it to the do_write function
    write_message_func_ptr write_func = do_write;

    // Call the function using the function pointer
    call_write_function(write_func, message);
    return 0;
}

[yizhuoliang]

@yzhang71 That commit changes one line, removing the inline assembly (.globl) that marks a symbol accessible globally during linking. I removed that because the assembly can't be compiled to WASM target.

[rennergade]

So from my understanding of what globl does its just basically doing an extern.

This macro libio_static_fn_required seems to only be called twice in stdio.c:

libio_static_fn_required (_IO_file_open);
libio_static_fn_required (_IO_file_doallocate);

I think you may just be able to extern those to fix this?

[yzhang71]

I suspect this might be the reason:

This commit puts all libio vtables in a dedicated, read-only ELF section, so that 
they are consecutive in memory. Before any indirect jump, the vtable pointer is 
checked against the section boundaries, and the process is terminated if the 
vtable pointer does not fall into the special ELF section.

[yzhang71]

Now printf (hello-world example) is fully functional. The problem with the vtables was due to the developers explicitly placing all libio vtables into the relro section in ELF. The solution is simple: we removed attribute_relro from /glibc/libio/vtables.c:92.

[yzhang71]

We are not back to the threading error of:

Error: WebAssembly translation error

Caused by:
    Invalid input WebAssembly code at offset 262763: type mismatch: values remaining on stack at end of block

Even though we are able to compile the test case, strangely we cannot even use gdb to debug the main function.

[yzhang71]

The issue has been identified. It was caused by int err = allocate_stack(iattr, &pd, &stackaddr, &stacksize);. The allocate_stack function returns a usable stack for a new thread either by allocating a new stack or reusing a cached stack of sufficient size. The ATTR parameter must be non-NULL and point to a valid pthread_attr. The PDP parameter must also be non-NULL.

We are currently working on fixing this issue.

[yzhang71]

Ok, the problem with allocatestack.c not being compiled into an object file is due to the following line:

/* Code to allocate and deallocate a stack.  */
#include "allocatestack.c"

This line includes the code from allocatestack.c directly into pthread_create.c. This is an unconventional method, but it seems like people use it anyway. It makes debugging harder, but I can debug it by adding print statements."

[yzhang71]

In the function allocate_stack in /glibc/nptl/allocatestack.c, the two assertions fail, so we have disabled them for now to continue compiling.

assert (powerof2 (pagesize_m1 + 1));
assert (TCB_ALIGNMENT >= STACK_ALIGN);

[yzhang71]

Next, we are getting error from:

/* Compute the size of the static TLS area based on data from the
   dynamic loader.  */
static inline size_t
__nptl_tls_static_size_for_stack (void)
{
  return roundup (GLRO (dl_tls_static_size), GLRO (dl_tls_static_align));
}

Thread 1 "wasmtime" received signal SIGFPE, Arithmetic exception.