search

scylladb / seastar

High performance server-side application framework
http://seastar.io
Apache License 2.0
8.37k stars 1.55k forks source link

unit test failed on arm machine #2394

Open deafwolf opened 2 months ago

deafwolf commented 2 months ago 
./configure.py --mode sanitize --c-compiler /usr/bin/clang --compiler /usr/bin//clang++ --cook boost
ninja -C build/sanitize unit_tests
./test.py --mode sanitize
 7/77 Test  #7: Seastar.unit.chunk_parsers ....................***Failed    0.19 sec
[0/1] cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/chunk_parsers_test -- -c 2
Running 2 test cases...
WARNING: debug mode. Not for benchmarking or production
INFO  2024-08-21 03:50:52,251 seastar - Reactor backend: io_uring
random-seed=2202370702
/home/stdb/seastar/tests/unit/chunk_parsers_test.cc(113): fatal error: in "test_trailer_headers_parsing": critical check parser.failed() != tset.parsable has failed [true == true]

*** 1 failure is detected in the test module "Master Test Suite"
FAILED: tests/unit/CMakeFiles/test_unit_chunk_parsers_run /home/stdb/seastar/build/sanitize/tests/unit/CMakeFiles/test_unit_chunk_parsers_run
cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/chunk_parsers_test -- -c 2
ninja: build stopped: subcommand failed.

31/77 Test #31: Seastar.unit.httpd ............................***Failed   33.85 sec
[0/1] cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/httpd_test -- -c 2
Running 47 test cases...
WARNING: debug mode. Not for benchmarking or production
INFO  2024-08-21 03:51:27,160 seastar - Reactor backend: io_uring
random-seed=615576222
==33903==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x48a42f 0x75239f 0x6d7da7 0x6d5edb 0x6d60df 0x7ff 0xb3fc7 0x4e1c9f 0x59e07b 0x5cf23b 0x5cef7b 0x5c486b 0x5c44af 0x5c42f3 0x5c4103 0x6ec883 0x6f09ef 0x6f2473 0x6f1557 0x651d2f 0x651157 0x63ca0b 0x6adeb3 0x4e0783 0x96797 0x101b4b
kernel callstack: 0xffffffffffffff80 0xffffd63bd54faf98 0xffffd63bd54fc694 0xffffd63bd5531448 0xffffd63bd55317e8 0xffffd63bd518e7e8 0xffffd63bd54d2b10 0xffffd63bd54d2c54 0xffffd63bd54d32b8 0xffffd63bd54d38f4 0xffffd63bd54d3aa0 0xffffd63bd638dd9c 0xffffd63bd638e1bc 0xffffd63bd518e0c0 0xffffd63bd6378704 0xffffd63bd6379484 0xffffd63bd5161d94
Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x48a42f 0x75239f 0x6d7da7 0x6d5edb 0x6d60df 0x7ff 0xb3fc7 0x4e1c9f 0x59e07b 0x5cf23b 0x5cef7b 0x5c486b 0x5c44af 0x5c42f3 0x5c4103 0x6ec883 0x6f09ef 0x6f2473 0x6f1557 0x651d2f 0x651157 0x63ca0b 0x6adeb3 0x4e0783 0x96797 0x101b4b
kernel callstack: 0xffffffffffffff80 0xffffd63bd5485294 0xffffd63bd5485de8 0xffffd63bd54d2dd4 0xffffd63bd54d32b8 0xffffd63bd54d38f4 0xffffd63bd54d3aa0 0xffffd63bd638dd9c 0xffffd63bd638e1bc 0xffffd63bd518e0c0 0xffffd63bd6378704 0xffffd63bd6379484 0xffffd63bd5161d94
Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x48a42f 0x75239f 0x6d7da7 0x6d5edb 0x6d60df 0x7ff 0xb53bb 0x44a27f 0x44a0b3 0x449d63 0x44ba13 0x522453 0x5c4887 0x5c44af 0x5c42f3 0x5c4103 0x6ec883 0x6f09ef 0x6f2473 0x6f1557 0x651d2f 0x651157 0x63ca0b 0x6adeb3 0x4e0783 0x96797 0x101b4b
kernel callstack:
Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: 0x48a42f 0x75239f 0x6d7da7 0x6d5edb 0x6d60df 0x7ff 0x44891b 0x446b87 0x521ba7 0x5cf08f 0x5cef53 0x5c486b 0x5c44af 0x5c42f3 0x5c4103 0x6ec883 0x6f09ef 0x6f2473 0x6f1557 0x651d2f 0x651157 0x63ca0b 0x6adeb3 0x4e0783 0x96797 0x101b4b
kernel callstack: 0xffffffffffffff80 0xffffd63bd638b210 0xffffd63bd54852a4 0xffffd63bd5485de8 0xffffd63bd54cab28 0xffffd63bd54cd394 0xffffd63bd54d39cc 0xffffd63bd54d3aa0 0xffffd63bd638dd9c 0xffffd63bd518e0c0 0xffffd63bd6378704 0xffffd63bd6379484 0xffffd63bd5161d94
Reactor stalled for 90 ms on shard 0, in scheduling group main. Backtrace: 0x48a42f 0x75239f 0x6d7da7 0x6d5edb 0x6d60df 0x7ff 0xb3fc7 0x4e1c9f 0x59e07b 0x5cf23b 0x5cef7b 0x5c486b 0x5c44af 0x5c42f3 0x5c4103 0x6ec883 0x6f09ef 0x6f2473 0x6f1557 0x651d2f 0x651157 0x63ca0b 0x6adeb3 0x4e0783 0x96797 0x101b4b
kernel callstack: 0xffffffffffffff80 0xffffd63bd63483cc 0xffffd63bd54fc694 0xffffd63bd5531448 0xffffd63bd55317e8 0xffffd63bd518e7e8 0xffffd63bd54d2b10 0xffffd63bd54d2c54 0xffffd63bd54d32b8 0xffffd63bd54d38f4 0xffffd63bd54d3aa0 0xffffd63bd638dd9c 0xffffd63bd638e1bc 0xffffd63bd518e0c0 0xffffd63bd6378704 0xffffd63bd6379484 0xffffd63bd5161d94

Returning exception from server
Returning exception from server
Simple request test
Request with body test
Request with content-length body
Request with 128-kbytes content-length body
Request with chunked body
Request with expect-continue
Request with incomplete content-length body
Request with too large content-length body
Simple request test
Request with body test
Request with content-length body
Request with 128-kbytes content-length body
Request with chunked body
Request with expect-continue
Request with incomplete content-length body
Request with too large content-length body
Simple request test
Request with body test
Request with content-length body
Request with 128-kbytes content-length body
Request with chunked body
Request with expect-continue
Request with incomplete content-length body
Request with too large content-length body
Simple request test
Request with body test
Request with content-length body
Request with 128-kbytes content-length body
Request with chunked body
Request with expect-continue
Request with incomplete content-length body
Request with too large content-length body
/home/stdb/seastar/tests/unit/httpd_test.cc(1191): fatal error: in "test_full_chunk_format": critical check std::string(resp.get(), resp.size()).find(std::move(str)) != std::string::npos has failed [18446744073709551615 == 18446744073709551615]
httpd_test: /home/stdb/seastar/include/seastar/core/gate.hh:91: seastar::gate::~gate(): Assertion `!_count && "gate destroyed with outstanding requests"' failed.
Aborting on shard 0, in scheduling group main.
Backtrace:
  0x48a42f
  0x751edb
  0x751dc7
  0x6d7ddb
  0x6ff99f
  0x72e98f
  linux-vdso.so.1+0x7ff
  /lib64/libc.so.6+0x9859f
  /lib64/libc.so.6+0x459bf
  /lib64/libc.so.6+0x30287
  /lib64/libc.so.6+0x3e3df
  /lib64/libc.so.6+0x3e453
  0x5f5b5f
  0x595cf7
  0x5847bb
  0x841cb7
Subprocess aborted
FAILED: tests/unit/CMakeFiles/test_unit_httpd_run /home/stdb/seastar/build/sanitize/tests/unit/CMakeFiles/test_unit_httpd_run
cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/httpd_test -- -c 2
ninja: build stopped: subcommand failed.

45/77 Test #45: Seastar.unit.request_parser ...................***Failed    0.19 sec
[0/1] cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/request_parser_test -- -c 2
Running 1 test case...
WARNING: debug mode. Not for benchmarking or production
INFO  2024-08-21 03:52:06,282 seastar - Reactor backend: io_uring
random-seed=2262752221
/home/stdb/seastar/tests/unit/request_parser_test.cc(67): fatal error: in "test_header_parsing": critical check parser.failed() != tset.parsable has failed [true == true]

*** 1 failure is detected in the test module "Master Test Suite"
FAILED: tests/unit/CMakeFiles/test_unit_request_parser_run /home/stdb/seastar/build/sanitize/tests/unit/CMakeFiles/test_unit_request_parser_run
cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/request_parser_test -- -c 2
ninja: build stopped: subcommand failed.
seastar version 1526684cc6a09bc9ecb3b6927506e7e8d8bc8ddc
Linux arm-build00 6.8.5-301.fc40.aarch64 #1 SMP PREEMPT_DYNAMIC Thu Apr 11 20:30:05 UTC 2024 aarch64 GNU/Linux

Architecture:             aarch64
  CPU op-mode(s):         32-bit, 64-bit
  Byte Order:             Little Endian
CPU(s):                   32
  On-line CPU(s) list:    0-31
Vendor ID:                ARM
  Model name:             Neoverse-N1
    Model:                1
    Thread(s) per core:   1
    Core(s) per socket:   32
    Socket(s):            1
    Stepping:             r3p1
    BogoMIPS:             243.75
    Flags:                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
Caches (sum of all):
  L1d:                    2 MiB (32 instances)
  L1i:                    2 MiB (32 instances)
  L2:                     32 MiB (32 instances)
  L3:                     32 MiB (1 instance)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-31
Vulnerabilities:
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; __user pointer sanitization
  Spectre v2:             Mitigation; CSV2, BHB
  Srbds:                  Not affected
  Tsx async abort:        Not affected
tchaikov commented 2 months ago

@deafwolf may i learn if this test failure is reproducible ? have you tried to use another backend for testing, like "linux-aio" instead of "io_uring" ?

deafwolf commented 2 months ago

@tchaikov yes，I can reproduce the failure with seastar version 1526684cc6a09bc9ecb3b6927506e7e8d8bc8ddc and f03f7df1639d00581012a5bc6bda62ad82a849ca

the configure command is ./configure.py --mode sanitize --c-compiler /usr/bin/clang --compiler /usr/bin//clang++ --cook boost --disable-io_uring

tchaikov commented 2 months ago

but the logging message shows:

INFO  2024-08-21 03:50:52,251 seastar - Reactor backend: io_uring

so you did test both backends?

deafwolf commented 2 months ago

here is the log of aio

['cmake', '/home/stdb/seastar/build/sanitize', '-DSeastar_TEST_TIMEOUT=300', '-DSeastar_EXECUTE_ONLY_FAST_TESTS=no', '-DSeastar_UNIT_TEST_SMP=2', '-DSeastar_JENKINS=']
-- Found Boost: /usr/include (found suitable version "1.83.0", minimum required is "1.73.0")
-- Found Boost: /usr/include (found suitable version "1.83.0", minimum required is "1.73.0") found components: filesystem program_options thread unit_test_framework chrono atomic
-- Configuring done (0.4s)
-- Generating done (0.2s)
-- Build files have been written to: /home/stdb/seastar/build/sanitize
['ctest', '/home/stdb/seastar/build/sanitize', '--output-on-failure']
Test project /home/stdb/seastar/build/sanitize
      Start  1: Seastar.unit.abort_source
 1/77 Test  #1: Seastar.unit.abort_source .....................   Passed    0.25 sec
      Start  2: Seastar.unit.alloc
 2/77 Test  #2: Seastar.unit.alloc ............................   Passed    0.69 sec
      Start  3: Seastar.unit.allocator
 3/77 Test  #3: Seastar.unit.allocator ........................   Passed    6.03 sec
      Start  4: Seastar.unit.alien
 4/77 Test  #4: Seastar.unit.alien ............................   Passed    0.24 sec
      Start  5: Seastar.unit.checked_ptr
 5/77 Test  #5: Seastar.unit.checked_ptr ......................   Passed    0.09 sec
      Start  6: Seastar.unit.chunked_fifo
 6/77 Test  #6: Seastar.unit.chunked_fifo .....................   Passed    0.17 sec
      Start  7: Seastar.unit.chunk_parsers
 7/77 Test  #7: Seastar.unit.chunk_parsers ....................***Failed    0.24 sec
[0/1] cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/chunk_parsers_test -- -c 2
Running 2 test cases...
WARNING: debug mode. Not for benchmarking or production
INFO  2024-09-02 09:58:46,007 seastar - Reactor backend: linux-aio
random-seed=1615764535
/home/stdb/seastar/tests/unit/chunk_parsers_test.cc(113): fatal error: in "test_trailer_headers_parsing": critical check parser.failed() != tset.parsable has failed [true == true]

*** 1 failure is detected in the test module "Master Test Suite"
FAILED: tests/unit/CMakeFiles/test_unit_chunk_parsers_run /home/stdb/seastar/build/sanitize/tests/unit/CMakeFiles/test_unit_chunk_parsers_run
cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/chunk_parsers_test -- -c 2
ninja: build stopped: subcommand failed.

      Start  8: Seastar.unit.circular_buffer
 8/77 Test  #8: Seastar.unit.circular_buffer ..................   Passed    1.02 sec
      Start  9: Seastar.unit.circular_buffer_fixed_capacity
 9/77 Test  #9: Seastar.unit.circular_buffer_fixed_capacity ...   Passed    1.11 sec
      Start 10: Seastar.unit.condition_variable
10/77 Test #10: Seastar.unit.condition_variable ...............   Passed    2.38 sec
      Start 11: Seastar.unit.connect
11/77 Test #11: Seastar.unit.connect ..........................   Passed    0.24 sec
      Start 12: Seastar.unit.content_source
12/77 Test #12: Seastar.unit.content_source ...................   Passed    0.24 sec
      Start 13: Seastar.unit.coroutines
13/77 Test #13: Seastar.unit.coroutines .......................   Passed    1.56 sec
      Start 14: Seastar.unit.defer
14/77 Test #14: Seastar.unit.defer ............................   Passed    0.09 sec
      Start 15: Seastar.unit.deleter
15/77 Test #15: Seastar.unit.deleter ..........................   Passed    0.09 sec
      Start 16: Seastar.unit.directory
16/77 Test #16: Seastar.unit.directory ........................   Passed    0.24 sec
      Start 17: Seastar.unit.distributed
17/77 Test #17: Seastar.unit.distributed ......................   Passed    2.77 sec
      Start 18: Seastar.unit.dns
18/77 Test #18: Seastar.unit.dns ..............................   Passed    3.38 sec
      Start 19: Seastar.unit.execution_stage
19/77 Test #19: Seastar.unit.execution_stage ..................   Passed    0.30 sec
      Start 20: Seastar.unit.expiring_fifo
20/77 Test #20: Seastar.unit.expiring_fifo ....................   Passed    0.23 sec
      Start 21: Seastar.unit.abortable_fifo
21/77 Test #21: Seastar.unit.abortable_fifo ...................   Passed    0.24 sec
      Start 22: Seastar.unit.io_queue
22/77 Test #22: Seastar.unit.io_queue .........................   Passed    9.86 sec
      Start 23: Seastar.unit.fair_queue
23/77 Test #23: Seastar.unit.fair_queue .......................   Passed    2.62 sec
      Start 24: Seastar.unit.file_io
24/77 Test #24: Seastar.unit.file_io ..........................   Passed    1.43 sec
      Start 25: Seastar.unit.file_utils
25/77 Test #25: Seastar.unit.file_utils .......................   Passed    0.41 sec
      Start 26: Seastar.unit.foreign_ptr
26/77 Test #26: Seastar.unit.foreign_ptr ......................   Passed    0.45 sec
      Start 27: Seastar.unit.fsnotifier
27/77 Test #27: Seastar.unit.fsnotifier .......................   Passed    0.28 sec
      Start 28: Seastar.unit.fstream
28/77 Test #28: Seastar.unit.fstream ..........................   Passed    0.38 sec
      Start 29: Seastar.unit.futures
29/77 Test #29: Seastar.unit.futures ..........................   Passed    3.40 sec
      Start 30: Seastar.unit.sharded
30/77 Test #30: Seastar.unit.sharded ..........................   Passed    0.24 sec
      Start 31: Seastar.unit.httpd
31/77 Test #31: Seastar.unit.httpd ............................***Failed   33.88 sec
[0/1] cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/httpd_test -- -c 2
Running 47 test cases...
WARNING: debug mode. Not for benchmarking or production
INFO  2024-09-02 09:59:19,216 seastar - Reactor backend: linux-aio
random-seed=3021684185
==13361==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x48a36f 0x75407f 0x6d7ce7 0x6d5e1b 0x6d601f 0x7ff 0xb3fcb 0x4e1bdf 0x59dfbb 0x5cf17b 0x5ceebb 0x5c47ab 0x5c43ef 0x5c4233 0x5c4043 0x6ec7c3 0x6f092f 0x6f23b3 0x6f1497 0x651c6f 0x651097 0x63c94b 0x6addf3 0x4e06c3 0x96797 0x101b4b
kernel callstack:
Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x48a36f 0x75407f 0x6d7ce7 0x6d5e1b 0x6d601f 0x7ff 0xb53bb 0x448b5f 0x446ac7 0x521ae7 0x5cefcf 0x5cee93 0x5c47ab 0x5c43ef 0x5c4233 0x5c4043 0x6ec7c3 0x6f092f 0x6f23b3 0x6f1497 0x651c6f 0x651097 0x63c94b 0x6addf3 0x4e06c3 0x96797 0x101b4b
kernel callstack: 0xffffffffffffff80 0xffffdde0b1ff86f4 0xffffdde0b1ff9484 0xffffdde0b0de1d94
Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x48a36f 0x75407f 0x6d7ce7 0x6d5e1b 0x6d601f 0x7ff 0xb53bb 0x44a1bf 0x449ff3 0x449ca3 0x44b953 0x522393 0x5c47c7 0x5c43ef 0x5c4233 0x5c4043 0x6ec7c3 0x6f092f 0x6f23b3 0x6f1497 0x651c6f 0x651097 0x63c94b 0x6addf3 0x4e06c3 0x96797 0x101b4b
kernel callstack:
Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: 0x48a36f 0x75407f 0x6d7ce7 0x6d5e1b 0x6d601f 0x7ff 0xb3fc7 0x4e1bdf 0x59dfbb 0x5cf17b 0x5ceebb 0x5c47ab 0x5c43ef 0x5c4233 0x5c4043 0x6ec7c3 0x6f092f 0x6f23b3 0x6f1497 0x651c6f 0x651097 0x63c94b 0x6addf3 0x4e06c3 0x96797 0x101b4b
kernel callstack: 0xffffffffffffff80 0xffffdde0b200b9c0 0xffffdde0b117aec0 0xffffdde0b117c694 0xffffdde0b11b1448 0xffffdde0b11b17e8 0xffffdde0b0e0e7e8 0xffffdde0b1152b10 0xffffdde0b1152c54 0xffffdde0b11532b8 0xffffdde0b11538f4 0xffffdde0b1153aa0 0xffffdde0b200dd9c 0xffffdde0b200e1bc 0xffffdde0b0e0e0c0 0xffffdde0b1ff8704 0xffffdde0b1ff9484 0xffffdde0b0de1d94
Reactor stalled for 90 ms on shard 0, in scheduling group main. Backtrace: 0x48a36f 0x75407f 0x6d7ce7 0x6d5e1b 0x6d601f 0x7ff 0xb3fc7 0x4e1bdf 0x59dfbb 0x5cf17b 0x5ceebb 0x5c47ab 0x5c43ef 0x5c4233 0x5c4043 0x6ec7c3 0x6f092f 0x6f23b3 0x6f1497 0x651c6f 0x651097 0x63c94b 0x6addf3 0x4e06c3 0x96797 0x101b4b
kernel callstack: 0xffffffffffffff80 0xffffdde0b1fc83cc 0xffffdde0b117c694 0xffffdde0b11b1448 0xffffdde0b11b17e8 0xffffdde0b0e0e7e8 0xffffdde0b1152b10 0xffffdde0b1152c54 0xffffdde0b11532b8 0xffffdde0b11538f4 0xffffdde0b1153aa0 0xffffdde0b200dd9c 0xffffdde0b200e1bc 0xffffdde0b0e0e0c0 0xffffdde0b1ff8704 0xffffdde0b1ff9484 0xffffdde0b0de1d94

Returning exception from server
Returning exception from server
Simple request test
Request with body test
Request with content-length body
Request with 128-kbytes content-length body
Request with chunked body
Request with expect-continue
Request with incomplete content-length body
Request with too large content-length body
Simple request test
Request with body test
Request with content-length body
Request with 128-kbytes content-length body
Request with chunked body
Request with expect-continue
Request with incomplete content-length body
Request with too large content-length body
Simple request test
Request with body test
Request with content-length body
Request with 128-kbytes content-length body
Request with chunked body
Request with expect-continue
Request with incomplete content-length body
Request with too large content-length body
Simple request test
Request with body test
Request with content-length body
Request with 128-kbytes content-length body
Request with chunked body
Request with expect-continue
Request with incomplete content-length body
Request with too large content-length body
/home/stdb/seastar/tests/unit/httpd_test.cc(1191): fatal error: in "test_full_chunk_format": critical check std::string(resp.get(), resp.size()).find(std::move(str)) != std::string::npos has failed [18446744073709551615 == 18446744073709551615]
httpd_test: /home/stdb/seastar/include/seastar/core/gate.hh:91: seastar::gate::~gate(): Assertion `!_count && "gate destroyed with outstanding requests"' failed.
Aborting on shard 0, in scheduling group main.
Backtrace:
  0x48a36f
  0x753bbb
  0x753aa7
  0x6d7d1b
  0x7003ff
  0x72f3ef
  linux-vdso.so.1+0x7ff
  /lib64/libc.so.6+0x9859f
  /lib64/libc.so.6+0x459bf
  /lib64/libc.so.6+0x30287
  /lib64/libc.so.6+0x3e3df
  /lib64/libc.so.6+0x3e453
  0x5f5a9f
  0x595c37
  0x5846fb
  0x844b33
Subprocess aborted
FAILED: tests/unit/CMakeFiles/test_unit_httpd_run /home/stdb/seastar/build/sanitize/tests/unit/CMakeFiles/test_unit_httpd_run
cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/httpd_test -- -c 2
ninja: build stopped: subcommand failed.

      Start 32: Seastar.unit.websocket
32/77 Test #32: Seastar.unit.websocket ........................   Passed    0.24 sec
      Start 33: Seastar.unit.ipv6
33/77 Test #33: Seastar.unit.ipv6 .............................   Passed    0.24 sec
      Start 34: Seastar.unit.network_interface
34/77 Test #34: Seastar.unit.network_interface ................   Passed    0.24 sec
      Start 35: Seastar.unit.json_formatter
35/77 Test #35: Seastar.unit.json_formatter ...................   Passed    0.24 sec
      Start 36: Seastar.unit.locking
36/77 Test #36: Seastar.unit.locking ..........................   Passed    0.28 sec
      Start 37: Seastar.unit.lowres_clock
37/77 Test #37: Seastar.unit.lowres_clock .....................   Passed    0.57 sec
      Start 38: Seastar.unit.metrics
38/77 Test #38: Seastar.unit.metrics ..........................   Passed    3.17 sec
      Start 39: Seastar.unit.net_config
39/77 Test #39: Seastar.unit.net_config .......................   Passed    0.09 sec
      Start 40: Seastar.unit.noncopyable_function
40/77 Test #40: Seastar.unit.noncopyable_function .............   Passed    0.09 sec
      Start 41: Seastar.unit.output_stream
41/77 Test #41: Seastar.unit.output_stream ....................   Passed    0.24 sec
      Start 42: Seastar.unit.packet
42/77 Test #42: Seastar.unit.packet ...........................   Passed    0.09 sec
      Start 43: Seastar.unit.program_options
43/77 Test #43: Seastar.unit.program_options ..................   Passed    0.09 sec
      Start 44: Seastar.unit.queue
44/77 Test #44: Seastar.unit.queue ............................   Passed    0.25 sec
      Start 45: Seastar.unit.request_parser
45/77 Test #45: Seastar.unit.request_parser ...................***Failed    0.24 sec
[0/1] cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/request_parser_test -- -c 2
Running 1 test case...
WARNING: debug mode. Not for benchmarking or production
INFO  2024-09-02 09:59:58,926 seastar - Reactor backend: linux-aio
random-seed=2734391369
/home/stdb/seastar/tests/unit/request_parser_test.cc(67): fatal error: in "test_header_parsing": critical check parser.failed() != tset.parsable has failed [true == true]

*** 1 failure is detected in the test module "Master Test Suite"
FAILED: tests/unit/CMakeFiles/test_unit_request_parser_run /home/stdb/seastar/build/sanitize/tests/unit/CMakeFiles/test_unit_request_parser_run
cd /home/stdb/seastar/build/sanitize/tests/unit && /usr/bin/cmake -E env ASAN_OPTIONS=disable_coredump=0:abort_on_error=1:detect_stack_use_after_return=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 BOOST_TEST_CATCH_SYSTEM_ERRORS=no /home/stdb/seastar/build/sanitize/tests/unit/request_parser_test -- -c 2
ninja: build stopped: subcommand failed.

      Start 46: Seastar.unit.rpc
46/77 Test #46: Seastar.unit.rpc ..............................   Passed   43.37 sec
      Start 47: Seastar.unit.semaphore
47/77 Test #47: Seastar.unit.semaphore ........................   Passed    1.22 sec
      Start 48: Seastar.unit.shared_ptr
48/77 Test #48: Seastar.unit.shared_ptr .......................   Passed    0.09 sec
      Start 49: Seastar.unit.signal
49/77 Test #49: Seastar.unit.signal ...........................   Passed    0.24 sec
      Start 50: Seastar.unit.simple_stream
50/77 Test #50: Seastar.unit.simple_stream ....................   Passed    0.09 sec
      Start 51: Seastar.unit.smp
51/77 Test #51: Seastar.unit.smp ..............................   Passed    0.24 sec
      Start 52: Seastar.unit.app-template
52/77 Test #52: Seastar.unit.app-template .....................   Passed    2.58 sec
      Start 53: Seastar.unit.socket
53/77 Test #53: Seastar.unit.socket ...........................   Passed    3.74 sec
      Start 54: Seastar.unit.sstring
54/77 Test #54: Seastar.unit.sstring ..........................   Passed    0.09 sec
      Start 55: Seastar.unit.stall_detector
55/77 Test #55: Seastar.unit.stall_detector ...................   Passed    0.24 sec
      Start 56: Seastar.unit.stream_reader
56/77 Test #56: Seastar.unit.stream_reader ....................   Passed    0.25 sec
      Start 57: Seastar.unit.thread
57/77 Test #57: Seastar.unit.thread ...........................   Passed    0.26 sec
      Start 58: Seastar.unit.scheduling_group
58/77 Test #58: Seastar.unit.scheduling_group .................   Passed    0.27 sec
      Start 59: Seastar.unit.thread_context_switch
59/77 Test #59: Seastar.unit.thread_context_switch ............   Passed    5.35 sec
      Start 60: Seastar.unit.timer
60/77 Test #60: Seastar.unit.timer ............................   Passed    4.64 sec
      Start 61: Seastar.unit.uname
61/77 Test #61: Seastar.unit.uname ............................   Passed    0.09 sec
      Start 62: Seastar.unit.source_location
62/77 Test #62: Seastar.unit.source_location ..................   Passed    0.09 sec
      Start 63: Seastar.unit.shared_token_bucket
63/77 Test #63: Seastar.unit.shared_token_bucket ..............   Passed    0.24 sec
      Start 64: Seastar.unit.tls
64/77 Test #64: Seastar.unit.tls ..............................   Passed   79.12 sec
      Start 65: Seastar.unit.tuple_utils
65/77 Test #65: Seastar.unit.tuple_utils ......................   Passed    0.09 sec
      Start 66: Seastar.unit.unix_domain
66/77 Test #66: Seastar.unit.unix_domain ......................   Passed    0.26 sec
      Start 67: Seastar.unit.unwind
67/77 Test #67: Seastar.unit.unwind ...........................   Passed    0.09 sec
      Start 68: Seastar.unit.weak_ptr
68/77 Test #68: Seastar.unit.weak_ptr .........................   Passed    0.09 sec
      Start 69: Seastar.unit.log_buf
69/77 Test #69: Seastar.unit.log_buf ..........................   Passed    0.25 sec
      Start 70: Seastar.unit.exception_logging
70/77 Test #70: Seastar.unit.exception_logging ................   Passed    0.10 sec
      Start 71: Seastar.unit.closeable
71/77 Test #71: Seastar.unit.closeable ........................   Passed    0.26 sec
      Start 72: Seastar.unit.pipe
72/77 Test #72: Seastar.unit.pipe .............................   Passed    0.25 sec
      Start 73: Seastar.unit.spawn
73/77 Test #73: Seastar.unit.spawn ............................   Passed    0.25 sec
      Start 74: Seastar.unit.json2code
74/77 Test #74: Seastar.unit.json2code ........................   Passed    0.33 sec
      Start 75: Seastar.unit.prometheus
75/77 Test #75: Seastar.unit.prometheus .......................   Passed    0.31 sec
      Start 76: Seastar.app.memcached.memcached
76/77 Test #76: Seastar.app.memcached.memcached ...............   Passed   92.76 sec
      Start 77: Seastar.app.memcached.ascii
77/77 Test #77: Seastar.app.memcached.ascii ...................   Passed   24.15 sec

96% tests passed, 3 tests failed out of 77

Total Test time (real) = 342.02 sec

The following tests FAILED:
      7 - Seastar.unit.chunk_parsers (Failed)
     31 - Seastar.unit.httpd (Failed)
     45 - Seastar.unit.request_parser (Failed)
Errors while running CTest
Traceback (most recent call last):
  File "/home/stdb/seastar/./test.py", line 71, in <module>
    run_tests(mode)
  File "/home/stdb/seastar/./test.py", line 68, in run_tests
    subprocess.check_call(CTEST_ARGS, shell=False, cwd=BUILD_PATH)
  File "/usr/lib64/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ctest', '/home/stdb/seastar/build/sanitize', '--output-on-failure']' returned non-zero exit status 8.