search

wasix-org / cargo-wasix

Cargo wrapper for working with Webassembly wasi(x).
Apache License 2.0
52 stars 10 forks source link

Wasix-Axum WASM application thread panics at runtime after load stress test #48

Open MikeTeddyOmondi opened 11 months ago

MikeTeddyOmondi commented 11 months ago

Upon load testing the application the application thread panics as below:

picasso@picasso:~/Desktop/src/projects/wasm/wasix-rust-examples/wasix-axum$ wasmer run target/wasm32-wasmer-wasi/debug/wasix-axum.wasm --net --enable-threads --env PORT=8080
Listening on http://127.0.0.1:8080
thread 'main' panicked at /home/picasso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.8/src/thread_parker/wasm_atomic.rs:51:13:
assertion failed: r == 0 || r == 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: RuntimeError: unreachable
    at __rust_start_panic (<module>[9542]:0x9c5e94)
    at rust_panic (<module>[1904]:0x6daaa8)
    at std::panicking::rust_panic_with_hook::hfdfcaa6e69ebc719 (<module>[391]:0x3f5bdb)
    at std::panicking::begin_panic_handler::{{closure}}::h3f7789fd24ed3f08 (<module>[3741]:0x826189)
    at std::sys_common::backtrace::__rust_end_short_backtrace::hbe7fbe04baf16200 (<module>[9479]:0x9c4caf)
    at rust_begin_unwind (<module>[5666]:0x8ef2a4)
    at core::panicking::panic_fmt::h07835762205d9da6 (<module>[7816]:0x97fdc9)
    at core::panicking::panic::h18678cb124bcc731 (<module>[6962]:0x94ea04)
    at <parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park::h239fcb653d5e8208 (<module>[1990]:0x6eff0d)
    at parking_lot_core::parking_lot::park::{{closure}}::h747f8f4c6ffb9884 (<module>[117]:0x25927a)
    at parking_lot_core::parking_lot::park::h0c82da8664c01e84 (<module>[1006]:0x5b82c7)
    at parking_lot::condvar::Condvar::wait_until_internal::h14feacfd5e810820 (<module>[815]:0x54f5bd)
    at parking_lot::condvar::Condvar::wait::h46b42241deee20ee (<module>[4998]:0x8b404b)
    at tokio::loom::std::parking_lot::Condvar::wait::h5886f82229a2d832 (<module>[4795]:0x8a0e07)
    at tokio::park::thread::Inner::park::h13d6871b9cf36ff9 (<module>[329]:0x3b1764)
    at <tokio::park::thread::CachedParkThread as tokio::park::Park>::park::{{closure}}::h074b229273ffab5b (<module>[7480]:0x971b6c)
    at tokio::park::thread::CachedParkThread::with_current::{{closure}}::he9760f52961154e1 (<module>[7116]:0x95a3f0)
    at std::thread::local::LocalKey<T>::try_with::hd1d2223d0d96b4d4 (<module>[1764]:0x6b8615)
    at tokio::park::thread::CachedParkThread::with_current::hd43a4cfda3f350a4 (<module>[5388]:0x8d7a29)
    at <tokio::park::thread::CachedParkThread as tokio::park::Park>::park::h69abcd569cf15580 (<module>[3223]:0x7e1ca7)
    at tokio::park::thread::CachedParkThread::block_on::h66d5e8981ec1b1db (<module>[697]:0x508d16)
    at tokio::runtime::enter::Enter::block_on::h9c87833c42353e15 (<module>[5418]:0x8da492)
    at tokio::runtime::thread_pool::ThreadPool::block_on::h165f604ff938f099 (<module>[3650]:0x819f00)
    at tokio::runtime::Runtime::block_on::h23f1bdae2f89888e (<module>[2530]:0x76305f)
    at wasix_axum::main::h17aa3b46cb99d6f2 (<module>[2144]:0x713075)
    at core::ops::function::FnOnce::call_once::ha9b2c8eb2ddd4496 (<module>[7844]:0x981660)
    at std::sys_common::backtrace::__rust_begin_short_backtrace::h311b36d1bc0c7f41 (<module>[7926]:0x987e9a)
    at std::rt::lang_start::{{closure}}::h55a3403b35cc2a28 (<module>[5827]:0x8ff999)
    at std::rt::lang_start_internal::h9f91ddae2fba41c4 (<module>[5810]:0x8fe1ba)
    at std::rt::lang_start::h3abfa2c193587021 (<module>[4130]:0x85942e)
    at __main_void (<module>[8977]:0x9b985a)
    at _start (<module>[8636]:0x9b41d6)
╰─▶ 1: RuntimeError: unreachable
           at __rust_start_panic (<module>[9542]:0x9c5e94)
           at rust_panic (<module>[1904]:0x6daaa8)
           at std::panicking::rust_panic_with_hook::hfdfcaa6e69ebc719 (<module>[391]:0x3f5bdb)
           at std::panicking::begin_panic_handler::{{closure}}::h3f7789fd24ed3f08 (<module>[3741]:0x826189)
           at std::sys_common::backtrace::__rust_end_short_backtrace::hbe7fbe04baf16200 (<module>[9479]:0x9c4caf)
           at rust_begin_unwind (<module>[5666]:0x8ef2a4)
           at core::panicking::panic_fmt::h07835762205d9da6 (<module>[7816]:0x97fdc9)
           at core::panicking::panic::h18678cb124bcc731 (<module>[6962]:0x94ea04)
           at <parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park::h239fcb653d5e8208 (<module>[1990]:0x6eff0d)
           at parking_lot_core::parking_lot::park::{{closure}}::h747f8f4c6ffb9884 (<module>[117]:0x25927a)
           at parking_lot_core::parking_lot::park::h0c82da8664c01e84 (<module>[1006]:0x5b82c7)
           at parking_lot::condvar::Condvar::wait_until_internal::h14feacfd5e810820 (<module>[815]:0x54f5bd)
           at parking_lot::condvar::Condvar::wait::h46b42241deee20ee (<module>[4998]:0x8b404b)
           at tokio::loom::std::parking_lot::Condvar::wait::h5886f82229a2d832 (<module>[4795]:0x8a0e07)
           at tokio::park::thread::Inner::park::h13d6871b9cf36ff9 (<module>[329]:0x3b1764)
           at <tokio::park::thread::CachedParkThread as tokio::park::Park>::park::{{closure}}::h074b229273ffab5b (<module>[7480]:0x971b6c)
           at tokio::park::thread::CachedParkThread::with_current::{{closure}}::he9760f52961154e1 (<module>[7116]:0x95a3f0)
           at std::thread::local::LocalKey<T>::try_with::hd1d2223d0d96b4d4 (<module>[1764]:0x6b8615)
           at tokio::park::thread::CachedParkThread::with_current::hd43a4cfda3f350a4 (<module>[5388]:0x8d7a29)
           at <tokio::park::thread::CachedParkThread as tokio::park::Park>::park::h69abcd569cf15580 (<module>[3223]:0x7e1ca7)
           at tokio::park::thread::CachedParkThread::block_on::h66d5e8981ec1b1db (<module>[697]:0x508d16)
           at tokio::runtime::enter::Enter::block_on::h9c87833c42353e15 (<module>[5418]:0x8da492)
           at tokio::runtime::thread_pool::ThreadPool::block_on::h165f604ff938f099 (<module>[3650]:0x819f00)
           at tokio::runtime::Runtime::block_on::h23f1bdae2f89888e (<module>[2530]:0x76305f)
           at wasix_axum::main::h17aa3b46cb99d6f2 (<module>[2144]:0x713075)
           at core::ops::function::FnOnce::call_once::ha9b2c8eb2ddd4496 (<module>[7844]:0x981660)
           at std::sys_common::backtrace::__rust_begin_short_backtrace::h311b36d1bc0c7f41 (<module>[7926]:0x987e9a)
           at std::rt::lang_start::{{closure}}::h55a3403b35cc2a28 (<module>[5827]:0x8ff999)
           at std::rt::lang_start_internal::h9f91ddae2fba41c4 (<module>[5810]:0x8fe1ba)
           at std::rt::lang_start::h3abfa2c193587021 (<module>[4130]:0x85942e)
           at __main_void (<module>[8977]:0x9b985a)
           at _start (<module>[8636]:0x9b41d6)

The first 1,000 requests load test past well but after changing the requests to 10,000 it failed after almost 2,800 requests as shown below: 1,000 requests

$ bombardier -c 2 -n 1000 http://localhost:8080
Bombarding http://localhost:8080 with 1000 request(s) using 2 connection(s)
 1000 / 1000 [=====================================================================================================================================================================] 100.00% 43/s 22s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec        44.36      30.88     117.87
  Latency       45.05ms     1.44ms    51.13ms
  HTTP codes:
    1xx - 0, 2xx - 1000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     8.83KB/s

10,000 requests

 $ bombardier -c 4 -n 10000 http://localhost:8080
Bombarding http://localhost:8080 with 10000 request(s) using 4 connection(s)
 10000 / 10000 [==================================================================================================================================================================] 100.00% 316/s 31s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       320.87    2950.83   45532.62
  Latency       12.61ms     9.93ms    52.27ms
  HTTP codes:
    1xx - 0, 2xx - 2801, 3xx - 0, 4xx - 0, 5xx - 0
    others - 7199
  Errors:
    dial tcp 127.0.0.1:8080: connect: connection refused - 7199
  Throughput:    17.69KB/s

Development environment:

theduke commented 10 months ago

@MikeTeddyOmondi can you: