bytecodealliance / wasm-micro-runtime

WebAssembly Micro Runtime (WAMR)
Apache License 2.0
4.98k stars 628 forks source link

zephyr: unable to compile with `WAMR_BUILD_LIBC_WASI` flag. #3311

Open lucasAbadFr opened 7 months ago

lucasAbadFr commented 7 months ago

Hello, I'm currently experimenting with sockets in WAMR on Zephyr, and I'm encountering some challenges.

  1. I've adapted a section of the Zephyr socket API.
  2. I'm attempting to run an HTTP client on WAMR (largely inspired by this sample).
  3. I'm working with the stm32 nucleo-h563zi board, but I'm facing compilation issues.

Compilation issues

  1. Plain compilation

    I compilated the http_client to a wasm module and like with the simple sample, I included it in a header file. But when I ran it on the board I had:

      *** Booting Zephyr OS build v3.6.0-961-gddb147d0a45d ***
      [00:00:00:000 - 20001190]: warning: failed to link import function (env, getaddrinfo)
      [00:00:00:000 - 20001190]: warning: failed to link import function (env, socket)
      [00:00:00:000 - 20001190]: warning: failed to link import function (env, connect)
      [00:00:00:000 - 20001190]: warning: failed to link import function (env, send)
      [00:00:00:000 - 20001190]: warning: failed to link import function (env, recv)
      [00:00:00:000 - 20001190]: warning: failed to link import function (env, close)
      [00:00:00:000 - 20001190]: warning: failed to link import function (env, __errno_location)
      Preparing HTTP GET request for http://192.0.2.10:8000/
      Exception: failed to call unlinked import function (env, getaddrinfo)
      elapsed: 72

    📄 Notes: I later adapted the sample to not use getaddrinfo because it add a useless dependency to netdb.h that I did not want.

  2. WAMR_BUILD_LIBC_WASI flag

    After looking at the socket-api sample, I found that I need to use the WAMR_BUILD_LIBC_WASI flag, so I added this flag to the CMakeLists.txt file but it didn't work.

    The output of the compilation is the same described in this issue My errors concist mainly of:

    1. libc_errno.h not found.

          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/random.c:16:10: fatal error: libc_errno.h: No such file or directory
          16 | #include "libc_errno.h"
                |          ^~~~~~~~~~~~~~
          compilation terminated.
    2. nfds_t unknown type.

          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.h:57:65: error: unknown type name 'nfds_t'
          57 | blocking_op_poll(wasm_exec_env_t exec_env, struct pollfd *pfds, nfds_t nfds,
    3. pthread_condattr_t unknown type & CLOCK_MONOTONIC undeclared.

          # Skip previous logs
          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:137:5: error: unknown type name 'pthread_condattr_t'
          137 |     pthread_condattr_t attr;
                |     ^~~~~~~~~~~~~~~~~~
          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:139:9: warning: implicit declaration of function 'pthread_condattr_init' [-Wimplicit-function-declaration]
          139 |     if (pthread_condattr_init(&attr) != 0)
                |         ^~~~~~~~~~~~~~~~~~~~~
          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:142:9: warning: implicit declaration of function 'pthread_condattr_setclock' [-Wimplicit-function-declaration]
          142 |     if (pthread_condattr_setclock(&attr, CLOCK_MONOTONIC) != 0)
                |         ^~~~~~~~~~~~~~~~~~~~~~~~~
          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:142:42: error: 'CLOCK_MONOTONIC' undeclared (first use in this function)
          142 |     if (pthread_condattr_setclock(&attr, CLOCK_MONOTONIC) != 0)
                |                                          ^~~~~~~~~~~~~~~
          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:142:42: note: each undeclared identifier is reported only once for each function it appears in
          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:145:9: warning: implicit declaration of function 'pthread_cond_init'; did you mean 'os_thread_env_init'? [-Wimplicit-function-declaration]
          145 |     if (pthread_cond_init(&cond->object, &attr) != 0)
                |         ^~~~~~~~~~~~~~~~~
                |         os_thread_env_init
          /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:150:5: warning: implicit declaration of function 'pthread_condattr_destroy' [-Wimplicit-function-declaration]
          150 |     pthread_condattr_destroy(&attr);
                |     ^~~~~~~~~~~~~~~~~~~~~~~~
          # Skip following logs
  3. Try to resolve

    I browsed the issues and found this Pull Request, but applying the fix lead to more errors and just one error was fixed.

    In core/shared/platform/zephyr/shared_platform.cmake I added the following lines:

      + if (WAMR_BUILD_LIBC_WASI EQUAL 1)
      +   list(APPEND source_all ${PLATFORM_SHARED_DIR}/../common/posix/posix_file.c)
      +   include (${CMAKE_CURRENT_LIST_DIR}/../common/libc-util/platform_common_libc_util.cmake)
      +   set(source_all ${source_all} ${PLATFORM_COMMON_LIBC_UTIL_SOURCE})
      + endif ()

    The error libc_errno.h was fixed but the other errors still persist. I also get all the errors from the /core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c file, mostly linked to the File System API ( e.g errors on struct pollfd).

  4. Test on host

    I reused the socket-api sample to compile and run the http_client module and it worked on my host machine.

      ./iwasm --addr-pool=192.0.2.10 http_get.wasm
      Preparing HTTP GET request for http://127.0.0.1:8000/
      sock = 5
      Response:
    
      HTTP/1.0 200 OK
      Server: SimpleHTTP/0.6 Python/3.10.12
      Date: Thu, 11 Apr 2024 13:41:11 GMT
      Content-type: text/html
      Content-Length: 14054
      Last-Modified: Thu, 30 Mar 2023 09:11:09 GMT
    
      # Skip HTML content

Runtime configuration

  1. Am I missing something ?

    My runtime configuration is the following:

      -- Build Configurations:
            Build as target THUMBV8
            CMAKE_BUILD_TYPE
            WAMR Interpreter enabled
            WAMR AOT enabled
            WAMR Fast JIT disabled
            WAMR LLVM ORC JIT disabled
            Libc builtin enabled
            Libc WASI enabled
            Fast interpreter disabled
            Multiple modules disabled
            Bulk memory feature enabled
            Wakeup of blocking operations enabled
            Reference types disabled
            GC performance profiling disabled
            Global heap pool enabled
            Custom global heap size: 131072
            Module instance context enabled
            Quick AOT/JIT entries enabled
  2. How to use the address pool ?

    I would like to have some help with the libc_wasi_init(wasm_module, argc, argv, &wasi_parse_ctx) function because if I understood it's mandatory to run a network application on WAMR.

    I looked at the product-mini/plateforms/posix/main.c & product-mini/plateforms/common/libc_wasi.c files but didn't fully understand the implementation because it is made to parse arguments from the command line.

    Meanwhile, I'm trying to run the http_client module on the board, so I don't have the possibility to pass arguments to the module.

    I came up with this configuration before wasm_runtime_full_init(&init_args) call in main.c.

      #if WASM_ENABLE_LIBC_WASI != 0
      #define HUMAN_READABLE_ADDRESS "192.0.2.10\\24"
      libc_wasi_parse_context_t wasi_parse_ctx;
      memset(&wasi_parse_ctx, 0, sizeof(wasi_parse_ctx));
    
      libc_wasi_parse_result_t result = libc_wasi_parse(HUMAN_READABLE_ADDRESS, &wasi_parse_ctx);
      switch (result) {
            case LIBC_WASI_PARSE_RESULT_OK:
                  continue;
            case LIBC_WASI_PARSE_RESULT_NEED_HELP:
                  return;
            case LIBC_WASI_PARSE_RESULT_BAD_PARAM:
                  return;
      }
      libc_wasi_init(wasm_module, argc, argv, &wasi_parse_ctx);
      #endif

    But I'm not sure if it's the right way to use the libc_wasi_parse function.

  3. Should I change build policy ?

    I tried to adapt the CMakeLists.txt file from the simple sample. What I changed:

    1. I added the WAMR_BUILD_LIBC_WASI flag.
    2. I added some env conf (path to wamr-sdk, wasi-sdk, python, ...)
    3. I added the wamr-sdk as an external projecxt and compile the libapp_framework.a.
    4. I added a custom target to compile the http_client module.
    5. I added a custom command that use a python script to generate the http_client.h module.

      • .wasm size ~= 1.3kB
      • .h size ~= 7.6kB

Any help will be appreciated. Thanks.

lucasAbadFr commented 7 months ago

Status

  1. Abstracting libc primitives: After looking at how the POSIX primitives where abstracted in core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/ to support Windows. I think that we could do the same to support Zephyr, with the final idea to have an implementation entirely relying on the os_ APIs.

    • functions:
      • poll --> os_poll
      • ioctl --> os_ioctl
    • structs/types:
      • timespec --> os_timespec
      • pollfd --> os_poll_file_handle
      • nfds_t --> os_nfds_t
  2. Extending platform_api_extension.h: The following API were added.

    __wasi_errno_t
    os_ioctl(os_file_handle *handle, int request, void *argp);
    
    // Not sure for return type
    __wasi_errno_t
    os_poll(os_poll_file_handle *pfds, os_nfds_t nfs, int timeout);
  3. Changing the sandboxed system primitives to use the new types and functions.

    • random.c:
      #elif defined(BH_PLATFORM_ZEPHYR)
      static void
      open_urandom(void)
      {
          // Not implemented
      }
      __wasi_errno_t
      random_buf(void *buf, size_t len)
      {
          return __WASI_ENOSYS;
      }
    • blocking.h:
      __wasi_errno_t
      - os_poll(pollfd *pfds, os_nfds_t nfds, int timeout)
      + os_poll(os_poll_file_handle *pfds, os_nfds_t nfds, int timeout)
    • locking.h: change every struct timespec to struct os_timespec
    • posix.c: change every struct pollfd to struct os_poll_file_handle
  4. Defining new types in platform_internal.h: Based on the os_file_handle type implementation we define the following types (and flags).

    typedef struct zsock_pollfd os_poll_file_handle;
    typedef unsigned int os_nfds_t;
    
    // To use posix flags
    #define POLLIN ZSOCK_POLLIN
    #define POLLPRI ZSOCK_POLLPRI
    #define POLLOUT ZSOCK_POLLOUT
    #define POLLERR ZSOCK_POLLERR
    #define POLLHUP ZSOCK_POLLHUP
    #define POLLNVAL ZSOCK_POLLNVAL
    
    #define FIONREAD ZFD_IOCTL_FIONREAD
    
    typedef struct {
            time_t tv_sec;
            long tv_nsec;
    } os_timespec;
    
    #define CLOCK_REALTIME 1
    #define CLOCK_MONOTONIC 4
    
    typedef struct {
        struct k_mutex mtx;  // Mutex for exclusive access
        struct k_sem sem;    // Semaphore for shared access
        int read_count;      // Number of readers
    } korp_rwlock;
  5. Implementing the new functions in platform_socket.c:

    __wasi_errno_t
    os_ioctl(os_file_handle *handle, int request, void *argp)
    {
        __wasi_errno_t wasi_errno = __WASI_ESUCCESS;
    
        if(zsock_ioctl_wrapper(handle, request, argp) < 0){
            wasi_errno = zephyr_to_wasi_errno(errno);
        }
        return wasi_errno;
    }
    
    __wasi_errno_t
    os_poll(os_poll_file_handle *fds, os_nfds_t nfs, int timeout)
    {
        __wasi_errno_t wasi_errno = __WASI_ESUCCESS;
        int rc = 0;
    
        rc = zsock_poll(fds, nfs, timeout) 
        if(rc < 0){
            wasi_errno = zephyr_to_wasi_errno(errno);
        }
        switch(rc){
            case 0:
                wasi_errno = __WASI_ETIMEOUT;
                break;
            case -1:
                wasi_errno = zephyr_to_wasi_errno(errno);
                break;
            default:
                break;
        }
        return wasi_errno;
    }
  6. compilation errors The compilation still fail with the same warnings as before. The unknown flags errors have disappeared. But the warning about the struct timespec and struct pollfd still persist.

    • Error on locking.h:
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:200:12: error: variable 'ts' has initializer but incomplete type
      200 |     struct os_timespec ts = {
          |            ^~~~~~~~~~~
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:201:10: error: 'struct os_timespec' has no member named 'tv_sec'
      201 |         .tv_sec = (time_t)(timeout / 1000000000),
          |          ^~~~~~
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:201:19: warning: excess elements in struct initializer
      201 |         .tv_sec = (time_t)(timeout / 1000000000),
          |                   ^
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:201:19: note: (near initialization for 'ts')
    • Error on posix.c:
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:2153:61: error: invalid application of 'sizeof' to incomplete type 'struct os_poll_file_handle'
      2153 |         wasm_runtime_malloc((uint32)(nsubscriptions * sizeof(*pfds)));
          |                                                             ^
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:2177:25: error: invalid use of undefined type 'struct os_poll_file_handle'
      2177 |                     pfds[i] = (struct os_poll_file_handle){
          |                         ^
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:2178:26: error: 'struct os_poll_file_handle' has no member named 'fd'
      2178 |                         .fd = fos[i]->file_handle,
          |                          ^~
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:2178:31: warning: excess elements in struct initializer
      2178 |                         .fd = fos[i]->file_handle,
          |

    This look like the compiler is not able to find the definition of the new types.

    • I tried to include the platform_internal.h in the locking.h and posix.c files to see if it solves the issue. But it didn't work.
    • I also tried to directly define the types in a new header and directly include it in the locking.h and posix.c files. But the issue still persist.
    • I even defined the whole file System API in a zephyr_file.c file (just returning __WASI_ENOSYS for now), and changed shared_platform.cmake. But the issue still persist.

    I'm still stuck on this issue. I will try to find a solution.

    If anyone is familiar with the implemenation of the sanboxed system primitives, or the how the abstract POSIX APIs are passed to sanboxed system primitives.

    Any help on this would be appreciated.

wenyongh commented 7 months ago

@lucasAbadFr could you upload the patch or create a PR and change it to draft or WIP, so that others can have a try according to your work?

lucasAbadFr commented 7 months ago

@wenyongh thanks for your reply.

A new zephyr sample called simpe-http was added to demonstrate socket usage on zephyr. Refer to the sample readme for build purpose.

I've just opened a draft and noticed that I didn't adhere to the code guideline or other guidelines.

These changes will cause most tests to fail.

Please note that the work wasn't intended to be shared in this state.

lucasAbadFr commented 6 months ago

Hi @wenyongh,

I managed to make a simple wasi module work (by breaking a lot of things).

I would be interested to know if it work on someone else board.

I updated the readme.md under the new sample to give a brief overview of what was done.

Also i'm open to any tips to compile a WASI module with libc (from wasi-sdk) linked.

wenyongh commented 6 months ago

@lucasAbadFr I am not sure who is using zephyr but I believe there are some developers requiring libc-wasi on zephyr. I found there are many modifications in wasm_runtime_common.c and core/iwasm/libraries/libc-wasi and wonder whether it is necessary? It would be better to add APIs and structures in zephyr platform and leave some APIs empty (or return false), and reduce the modification on core/iwasm and make the CIs run successfully.

lucasAbadFr commented 6 months ago

Thanks for the feedback.

The CI fail due to the new struct and API defined in platform_api_extension.h, to make it pass I could just typedef the new struct in platform_internal.h, and declare the API in their respective folder.

As for the changes in core/iwasm/libraries/libc-wasi/sandboxed-system-primitives, they are mostly here to abstract POSIX functions and use os_ abstraction. I think this is necessary to have at least a full abstraction.

The main changes are in core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c where I had an offset due to struct size being different on both platform.

I'm curently investigating how to solve this issue and working to maintain compatibility with POSIX platform, which is the most used/maintained one.

wenyongh commented 6 months ago

OK, I read the code and added some comments, it should be OK to abstract some os_xxx structures and functions. I think maybe you can enable the basic functionality and make CI run fine first, and then we can let others help review this PR and give suggestions, like how to abstract the APIs.