Timeout in historical_queries_test

Describe the bug Observed on https://github.com/microsoft/CCF/pull/5413, in https://dev.azure.com/MSRC-CCF/CCF/_build/results?buildId=72629&view=results.

To Reproduce

ctest --repeat-until-fail 100 -VV -R historical_queries_test

Does not work every time, chances increase with release build and no verbose logging.

Despite doctest reporting an abort, no core file seems to be produced. Making the following change on :1178 allows attaching a debugger at the time of failure:

      if (Clock::now() - start_time > too_long)
      {
        error_printer();
        std::this_thread::sleep_for(std::chrono::seconds(10000));
        return false;
      }

(lldb) thread list
Process 25818 stopped
* thread #1: tid = 25818, 0x00007f3a6a4a4cd7 libpthread.so.0`__pthread_clockjoin_ex(threadid=139888754157312, thread_return=0x0000000000000000, clockid=<unavailable>, abstime=<unavailable>, block=<unavailable>) at pthread_join_common.c:145:6, name = 'historical_quer', stop reason = signal SIGSTOP
  thread #2: tid = 25819, 0x00007f3a6a21c23f libc.so.6`__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f3a6a12eba8, rem=0x00007f3a6a12eba8) at clock_nanosleep.c:78:7, name = 'historical_quer', stop reason = signal SIGSTOP
  thread #3: tid = 25825, 0x00007f3a6a21c23f libc.so.6`__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f3a637fd868, rem=0x00007f3a637fd868) at clock_nanosleep.c:78:7, name = 'historical_quer', stop reason = signal SIGSTOP

(lldb) bt
* thread #2, name = 'historical_quer', stop reason = signal SIGSTOP
  * frame #0: 0x00007f3a6a21c23f libc.so.6`__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f3a6a12eba8, rem=0x00007f3a6a12eba8) at clock_nanosleep.c:78:7
    frame #1: 0x00007f3a6a221ec7 libc.so.6`__GI___nanosleep(requested_time=<unavailable>, remaining=<unavailable>) at nanosleep.c:27:13
    frame #2: 0x00007f3a6a8cc25b libc++.so.1`std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>> const&) + 91
    frame #3: 0x00005566f8c1df3f historical_queries_test`void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_5>>(void*) at thread:383:9
    frame #4: 0x00005566f8c1df2b historical_queries_test`void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_5>>(void*) at historical_queries.cpp:1152:7
    frame #5: 0x00005566f8c1dd92 historical_queries_test`void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_5>>(void*) [inlined] decltype(__f=0x00005566f9c96ff8)::$_5>()()) std::__1::__invoke[abi:v15007]<DOCTEST_ANON_FUNC_129()::$_5>(DOCTEST_ANON_FUNC_129()::$_5&&) at invoke.h:394:23
    frame #6: 0x00005566f8c1dd92 historical_queries_test`void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_5>>(void*) [inlined] void std::__1::__thread_execute[abi:v15007]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_5>(__t=size=2, (null)=<unavailable>)::$_5>&, std::__1::__tuple_indices<>) at thread:284:5
    frame #7: 0x00005566f8c1dd92 historical_queries_test`void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_5>>(__vp=0x00005566f9c96ff0) at thread:295:5
    frame #8: 0x00007f3a6a4a3609 libpthread.so.0`start_thread(arg=<unavailable>) at pthread_create.c:477:8
    frame #9: 0x00007f3a6a25e133 libc.so.6`__clone at clone.S:95

(lldb) bt
* thread #3, name = 'historical_quer', stop reason = signal SIGSTOP
  * frame #0: 0x00007f3a6a21c23f libc.so.6`__GI___clock_nanosleep(clock_id=0, flags=0, req=0x00007f3a637fd868, rem=0x00007f3a637fd868) at clock_nanosleep.c:78:7
    frame #1: 0x00007f3a6a221ec7 libc.so.6`__GI___nanosleep(requested_time=<unavailable>, remaining=<unavailable>) at nanosleep.c:27:13
    frame #2: 0x00007f3a6a8cc25b libc++.so.1`std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>> const&) + 91
    frame #3: 0x00005566f8c2195e historical_queries_test`decltype(std::declval<DOCTEST_ANON_FUNC_129()::$_8>()(std::declval<unsigned long>())) std::__1::__invoke[abi:v15007]<DOCTEST_ANON_FUNC_129()::$_8, unsigned long>(DOCTEST_ANON_FUNC_129()::$_8&&, unsigned long&&) at thread:383:9
    frame #4: 0x00005566f8c2193c historical_queries_test`decltype(std::declval<DOCTEST_ANON_FUNC_129()::$_8>()(std::declval<unsigned long>())) std::__1::__invoke[abi:v15007]<DOCTEST_ANON_FUNC_129()::$_8, unsigned long>(DOCTEST_ANON_FUNC_129()::$_8&&, unsigned long&&) [inlined] auto DOCTEST_ANON_FUNC_129(this=0x00007ffe17410530, fetch_result=<unavailable>, check_result=<unavailable>, error_printer=<unavailable>)::$_6::operator()<auto DOCTEST_ANON_FUNC_129()::$_12::operator()<DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'()>(ds::ContiguousSet<unsigned long> const&, unsigned long, DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'() const&) const::'lambda'(), auto DOCTEST_ANON_FUNC_129()::$_12::operator()<DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'()>(ds::ContiguousSet<unsigned long> const&, unsigned long, DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'() const&) const::'lambda0'(), DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'()>(DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'() const&, auto DOCTEST_ANON_FUNC_129()::$_12::operator()<DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'()>(ds::ContiguousSet<unsigned long> const&, unsigned long, DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'() const&) const::'lambda0'() const&, DOCTEST_ANON_FUNC_129()::$_8::operator()(unsigned long) const::'lambda'() const&) const at historical_queries.cpp:1179:9
    frame #5: 0x00005566f8c2190b historical_queries_test`decltype(std::declval<DOCTEST_ANON_FUNC_129()::$_8>()(std::declval<unsigned long>())) std::__1::__invoke[abi:v15007]<DOCTEST_ANON_FUNC_129()::$_8, unsigned long>(DOCTEST_ANON_FUNC_129()::$_8&&, unsigned long&&) at historical_queries.cpp:1318:5
    frame #6: 0x00005566f8c2190b historical_queries_test`decltype(std::declval<DOCTEST_ANON_FUNC_129()::$_8>()(std::declval<unsigned long>())) std::__1::__invoke[abi:v15007]<DOCTEST_ANON_FUNC_129()::$_8, unsigned long>(DOCTEST_ANON_FUNC_129()::$_8&&, unsigned long&&) at historical_queries.cpp:1410:13
    frame #7: 0x00005566f8c1fcbf historical_queries_test`decltype(__f=0x00005566f9c759d8, __args=0x00005566f9c75a10)::$_8>()(std::declval<unsigned long>())) std::__1::__invoke[abi:v15007]<DOCTEST_ANON_FUNC_129()::$_8, unsigned long>(DOCTEST_ANON_FUNC_129()::$_8&&, unsigned long&&) at invoke.h:394:23
    frame #8: 0x00005566f8c1fbe0 historical_queries_test`void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_8, unsigned long>>(void*) [inlined] void std::__1::__thread_execute[abi:v15007]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_8, unsigned long, 2ul>(__t=size=3, (null)=<unavailable>)::$_8, unsigned long>&, std::__1::__tuple_indices<2ul>) at thread:284:5
    frame #9: 0x00005566f8c1fbd3 historical_queries_test`void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DOCTEST_ANON_FUNC_129()::$_8, unsigned long>>(__vp=0x00005566f9c759d0) at thread:295:5
    frame #10: 0x00007f3a6a4a3609 libpthread.so.0`start_thread(arg=<unavailable>) at pthread_create.c:477:8
    frame #11: 0x00007f3a6a25e133 libc.so.6`__clone at clone.S:95

Failure obtained with more requests per thread (300):

17: Test command: /home/amchamay/CCF/build/historical_queries_test
17: Environment variables: 
17:  TSAN_OPTIONS=suppressions=/home/amchamay/CCF/tsan_env_suppressions
17:  ASAN_OPTIONS=alloc_dealloc_mismatch=0
17: Test timeout computed to be: 10000000
17: [doctest] doctest version is "2.4.11"
17: [doctest] run with "--help" for options
17: Thread <6>, i [204]: Ranges 3->5, 7->8, 10->11 [States] - still no answer!
17: I've previously used handle 6 to request:
17:   Range 10->21 [States]
17:   Point 8 [States]
17:   Point 10 [Stores]
17:   Ranges 13->14, 16->17 [States]
17:   Point 22 [Stores]
17:   Point 12 [Stores]
17:   Point 16 [Stores]
17:   Ranges 6->6 [States]
17:   Ranges 7->8, 10->11, 13->14 [States]
17:   Ranges 7->8, 10->11, 13->14, 16->18 [Stores]
17:   Ranges 8->8, 10->11, 13->14, 16->17 [Stores]
17:   Point 7 [States]
17:   Point 4 [Stores]
17:   Ranges 16->17, 19->20 [States]
17:   Range 17->21 [Stores]
17:   Point 7 [States]
17:   Ranges 19->21 [Stores]
17:   Point 14 [States]
17:   Point 3 [Stores]
17:   Ranges 16->17 [Stores]
17:   Point 9 [States]
17:   Point 13 [States]
17:   Range 3->4 [Stores]
17:   Range 6->14 [Stores]
17:   Range 3->14 [Stores]
17:   Ranges 9->9 [Stores]
17:   Range 9->23 [States]
17:   Range 16->21 [Stores]
17:   Range 9->20 [States]
17:   Ranges 5->5, 7->8, 10->11, 13->14 [Stores]
17:   Range 19->20 [States]
17:   Point 3 [Stores]
17:   Range 13->14 [States]
17:   Point 19 [Stores]
17:   Range 4->20 [Stores]
17:   Point 3 [Stores]
17:   Ranges 4->5, 7->8, 10->11, 13->14, 16->17, 19->21 [Stores]
17:   Point 4 [States]
17:   Ranges 8->8, 10->10 [Stores]
17:   Point 18 [Stores]
17:   Ranges 7->8, 10->11, 13->14, 16->18 [Stores]
17:   Point 20 [States]
17:   Point 16 [States]
17:   Point 16 [States]
17:   Range 16->20 [States]
17:   Ranges 8->8, 10->11, 13->14, 16->17, 19->20, 22->23 [Stores]
17:   Point 10 [States]
17:   Range 8->11 [Stores]
17:   Point 10 [States]
17:   Point 22 [States]
17:   Ranges 17->17, 19->20 [States]
17:   Range 18->23 [Stores]
17:   Range 13->18 [Stores]
17:   Ranges 4->5, 7->8, 10->11 [States]
17:   Point 15 [Stores]
17:   Point 4 [States]
17:   Ranges 4->5, 7->8, 10->10 [States]
17:   Ranges 9->11 [States]
17:   Range 6->16 [Stores]
17:   Ranges 7->7 [Stores]
17:   Point 3 [States]
17:   Range 9->17 [Stores]
17:   Range 8->8 [Stores]
17:   Point 23 [States]
17:   Range 5->6 [States]
17:   Ranges 6->8, 10->11, 13->14, 16->17, 19->20, 22->22 [Stores]
17:   Point 18 [States]
17:   Ranges 4->5, 7->8, 10->10 [States]
17:   Ranges 10->11, 13->14, 16->17, 19->20, 22->23 [Stores]
17:   Range 6->22 [Stores]
17:   Ranges 6->8, 10->11, 13->14, 16->18 [Stores]
17:   Ranges 5->5, 7->8, 10->11, 13->14, 16->17, 19->20, 22->23 [Stores]
17:   Ranges 4->5, 7->8, 10->11, 13->13 [States]
17:   Ranges 5->5, 7->8, 10->10 [States]
17:   Ranges 8->8, 10->11, 13->13 [Stores]
17:   Range 10->12 [Stores]
17:   Point 17 [Stores]
17:   Point 5 [States]
17:   Point 15 [States]
17:   Ranges 7->7 [States]
17:   Point 20 [States]
17:   Point 17 [Stores]
17:   Range 11->22 [States]
17:   Range 7->12 [States]
17:   Range 4->22 [States]
17:   Ranges 7->8, 10->11, 13->14, 16->18 [Stores]
17:   Range 4->19 [Stores]
17:   Range 3->23 [Stores]
17:   Ranges 15->18 [States]
17:   Point 10 [Stores]
17:   Point 13 [States]
17:   Range 18->20 [States]
17:   Ranges 7->8, 10->12 [States]
17:   Range 7->9 [States]
17:   Point 22 [States]
17:   Range 7->10 [Stores]
17:   Ranges 3->4 [Stores]
17:   Point 10 [Stores]
17:   Ranges 7->8, 10->11 [States]
17:   Range 10->14 [States]
17:   Point 16 [States]
17:   Ranges 7->8, 10->11, 13->14, 16->17, 19->20, 22->22 [States]
17:   Point 6 [Stores]
17:   Range 8->12 [Stores]
17:   Range 3->8 [Stores]
17:   Point 19 [Stores]
17:   Ranges 15->17, 19->20 [States]
17:   Point 22 [States]
17:   Point 4 [Stores]
17:   Range 9->17 [Stores]
17:   Range 12->22 [Stores]
17:   Point 21 [Stores]
17:   Point 18 [Stores]
17:   Ranges 3->6 [Stores]
17:   Range 6->18 [Stores]
17:   Range 7->11 [States]
17:   Ranges 13->14, 16->16 [States]
17:   Range 3->14 [States]
17:   Ranges 15->17, 19->20, 22->23 [States]
17:   Ranges 9->11, 13->14, 16->17, 19->20 [States]
17:   Range 10->18 [States]
17:   Point 8 [States]
17:   Ranges 4->5, 7->8, 10->11, 13->15 [Stores]
17:   Range 9->16 [States]
17:   Ranges 7->8, 10->12 [Stores]
17:   Point 14 [States]
17:   Point 13 [States]
17:   Range 15->20 [States]
17:   Point 16 [States]
17:   Point 11 [Stores]
17:   Point 5 [Stores]
17:   Ranges 3->5, 7->8, 10->11, 13->14, 16->17 [Stores]
17:   Ranges 10->11, 13->15 [States]
17:   Range 11->12 [States]
17:   Range 13->15 [States]
17:   Ranges 19->19 [Stores]
17:   Ranges 8->9 [Stores]
17:   Range 10->17 [Stores]
17:   Point 19 [States]
17:   Point 3 [States]
17:   Point 21 [States]
17:   Ranges 18->18 [Stores]
17:   Point 8 [Stores]
17:   Range 11->16 [States]
17:   Point 11 [States]
17:   Range 17->22 [Stores]
17:   Ranges 3->5, 7->8, 10->11, 13->13 [Stores]
17:   Point 15 [States]
17:   Range 11->12 [Stores]
17:   Range 7->14 [States]
17:   Range 9->16 [States]
17:   Ranges 5->5, 7->8, 10->11, 13->14, 16->18 [States]
17:   Ranges 18->21 [States]
17:   Range 15->16 [Stores]
17:   Range 5->23 [Stores]
17:   Range 5->8 [States]
17:   Ranges 10->11 [States]
17:   Range 3->7 [States]
17:   Ranges 8->8, 10->11, 13->14, 16->18 [Stores]
17:   Range 11->14 [Stores]
17:   Ranges 11->11, 13->14, 16->17 [States]
17:   Range 17->22 [States]
17:   Ranges 9->11, 13->14, 16->17, 19->20 [Stores]
17: Thread <1>, i [221]: Range 3->8 [States] - still no answer!  Ranges 12->14 [States]
17:   Point 16 [States]
17:   Point 12 [States]
17:   Ranges 5->5, 7->9 [States]
17: 
17:   Range 8->17 [Stores]
17:   I've previously used handle 1 to request:
17:   Range 16->23 [Stores]
17:   Ranges 3->5, 7->8, 10->11 [Stores]
17:   Ranges 16->17, 19->20 [Stores]
17: Range 3->9 [States]  Range 9->22 [States]
17:   Range 19->20 [Stores]
17:   Point 23 [States]
17:   Ranges 10->11, 13->14, 16->16 [States]
17:   Point 18 [Stores]
17:   Ranges 7->8, 10->11, 13->14, 16->16 [Stores]
17:   Range 3->3 [Stores]
17:   Range 8->20 [Stores]
17:   Ranges 10->11, 13->14 [States]
17:   Point 21 [States]
17:   Ranges 7->8, 10->11, 13->14, 16->17, 19->21 [States]
17:   Ranges 14->15 [States]
17:   Range 4->20 [Stores]
17:   Range 8->22 [States]
17:   Ranges 5->5, 7->8, 10->11 [Stores]
17:   Range 20->22 [States]
17:   Ranges 12->14, 16->17, 19->20, 22->23 [States]
17:   Point 4 [Stores]
17:   Range 4->5 [States]
17:   Range 10->23 [States]
17:   Point 22 [Stores]
17:   Ranges 10->11, 13->14 [States]
17:   Point 13 [Stores]
17:   Ranges 12->15 [Stores]
17:   Range 3->23 [States]
17:   Range 15->20 [States]
17: 
17:   Ranges 14->14, 16->16 [Stores]
17:   Range 5->23 [Stores]
17:   Range 3->17 [Stores]
17:   Point 17 [Stores]
17:   Point 10 [States]
17:   Range 14->21 [Stores]
17:   Range 22->22 [Stores]
17:   Range 9->13 [Stores]
17:   Ranges 7->8, 10->11, 13->14, 16->17, 19->20, 22->23 [States]
17:     Range 12->19 [Stores]
17:   Ranges 5->5, 7->8, 10->11 [Stores]
17:   Range 6->13 [Stores]
17:   Ranges 5->5, 7->8, 10->11, 13->14, 16->17, 19->20, 22->23 [States]
17:   Ranges 3->5, 7->8, 10->10 [Stores]
17:   Ranges 4->5, 7->8, 10->12 [States]
17:   Ranges 15->17, 19->20, 22->23 [States]
17:   Range 7->11 [States]
17: Ranges 4->5 [Stores]  Ranges 15->17, 19->19 [States]
17:   Range 21->23 [Stores]
17:   Ranges 16->17, 19->20 [Stores]
17:   Ranges 3->5, 7->8, 10->10 [Stores]
17: 
17:   Point 6 [Stores]  Range 4->4 [States]
17:   Ranges 15->17, 19->20, 22->23 [States]
17:   Range 6->15 [States]
17:   Point 16 [States]
17: 
17:     Ranges 3->6 [Stores]
17:   Range 4->10 [States]
17:   Range 9->11 [Stores]
17:   Point 22 [States]
17:   Ranges 8->8, 10->11, 13->14, 16->17, 19->19 [States]
17:   Range 5->10 [Stores]
17:   Range 12->19 [Stores]
17:   Ranges 7->8, 10->11, 13->15 [States]
17:   Ranges 8->8, 10->11, 13->15 [Stores]
17: Point 12 [Stores]  Range 14->20 [States]
17:   Range 3->9 [Stores]
17:   Point 22 [States]
17:   Ranges 4->5, 7->7 [States]
17: 
17:   Range 12->23 [States]
17:   Ranges 16->18 [States]
17:   Ranges 11->11, 13->14, 16->17, 19->20 [Stores]
17:   Range 3->15 [States]
17:   Range 13->15 [States]
17:   Range 3->13 [Stores]
17:   Ranges 13->14 [Stores]
17:   Range 7->17 [Stores]
17:     Range 13->14 [States]
17: Range 12->17 [States]  Ranges 6->8, 10->11, 13->13 [Stores]
17:   Point 15 [Stores]
17:   Range 4->20 [Stores]
17:   Point 17 [Stores]
17:   Point 12 [Stores]
17:   Point 23 [States]
17: 
17:     Ranges 15->17, 19->21 [States]
17:   Range 13->17 [States]
17: Range 8->13 [States]  Range 7->11 [Stores]
17:   Ranges 20->20, 22->23 [States]
17:   Point 19 [States]
17:   Point 7 [Stores]
17: 
17:     Range 12->20 [States]
17:   Ranges 4->5, 7->8, 10->11, 13->14, 16->17, 19->21 [Stores]
17:   Range 5->21 [States]
17:   Ranges 6->8 [States]
17: Range 5->8 [Stores]  Ranges 13->14 [States]
17: 
17:   Ranges 3->5, 7->8, 10->11 [States]
17:   Range 6->12 [Stores]
17:   Ranges 13->14, 16->17 [States]
17:   Ranges 6->8, 10->11, 13->14, 16->17, 19->20 [States]
17:   Point 17 [Stores]
17:   Point 23 [Stores]
17:   Range 10->22 [States]
17:   Range 12->23 [States]
17: ===============================================================================
17:   Ranges 3->5, 7->8, 10->11, 13->14, 16->17, 19->20, 22->23 [Stores]
17:   Ranges 3->5, 7->8, 10->11, 13->14, 16->17, 19->20 [States]
17:   Ranges 11->11, 13->14, 16->18 [States]
17:   Range 14->23 [Stores]
17:   Ranges 6->8, 10->10 [Stores]../src/node/test/historical_queries.cpp
17:   Ranges 10->11, 13->15 [Stores]
17:   Ranges 15->18 [States]
17:   Point 8 [States]
17: :  Point 14 [Stores]
17:   Range 7->12 [States]
17:   Range 15->21 [Stores]
17:   Range 18->19 [States]
17:   Point 8 [States]
17:   Ranges 3->5, 7->8, 10->11, 13->14, 16->17, 19->20, 22->22 [States]
17:   Point 9 [States]
17:   Point 22 [States]
17:   Point 22 [Stores]
17:   Point 7 [Stores]
17:   Range 16->21 [Stores]
17:   Ranges 7->8, 10->11, 13->15 [Stores]
17:   Ranges 4->5, 7->8, 10->11, 13->14, 16->17, 19->20 [Stores]
17:   Ranges 4->5, 7->8, 10->11, 13->14, 16->17 [States]
17: 1082  Range 12->23 [States]:
17: 
17:   Range 15->20 [Stores]
17: TEST CASE:    StateCache concurrent accessRanges 7->8, 10->11, 13->14, 16->17, 19->20 [States]
17: 
17:   Point 22 [States]
17:   Ranges 4->5, 7->7 [Stores]
17:   Range 9->20 [Stores]
17: 
17:   Range 11->13 [States]../src/node/test/historical_queries.cpp
17: :1324  Range 4->6 [States]
17:   Point 23 [States]
17:   Range 10->16 [Stores]
17:   Range 3->16 [Stores]
17:   Ranges 3->5, 7->8, 10->11, 13->14, 16->17, 19->20, 22->22 [Stores]
17:   Point 18 [Stores]
17: :  Point 3 [States]
17:  FATAL ERROR  Point 12 [States]: 
17: REQUIRE(   fetch_until_timeout(fetch_result, check_result, error_printer) ) Range 5->18 [Stores]
17:   Range 15->20 [States]
17:   Ranges 9->11, 13->14, 16->17, 19->20, 22->23 [Stores]
17:   Point 16 [Stores]
17:   Ranges 16->18 [States]
17:   Point 18 [States]
17:   Point 5 [Stores]
17:   Point 21 [States]
17:   Point 5 [States]
17:   Ranges 10->11, 13->15 [Stores]
17:   Point 8 [Stores]
17: is NOT correct!
17:   Point 5 [Stores]
17:   Range 13->14 [Stores]
17:   Range 4->13 [Stores]
17:   Point 3 [States]
17:   Range 3->11 [Stores]
17:   Range 21->21 [Stores]
17:   Range 12->22 [States]
17:   values:   REQUIRE( false )
17: Point 14 [States]
17:   Range 4->21 [Stores]
17:   Point 7 [States]
17: 
17:   Point 12 [States]
17:   Range 5->12 [Stores]
17:   Point 23 [States]
17:   Range 12->19 [Stores]
17:   Ranges 6->8, 10->11, 13->14, 16->18 [States]
17:   Ranges 8->8, 10->11, 13->15 [Stores]
17:   Range 12->21 [Stores]
17:   Range 3->17 [States]
17:   Range 4->22 [States]
17:   Point 10 [Stores]
17:   Ranges 12->15 [States]
17:   Range 8->21 [Stores]
17:   Point 19 [States]
17:   Range 3->6 [Stores]
17:   Ranges 8->8, 10->12 [States]
17:   Ranges 11->11, 13->14, 16->17, 19->20, 22->22 [Stores]
17:   Point 4 [States]
17:   Ranges 3->5, 7->8, 10->11, 13->15 [Stores]
17:   Ranges 9->11, 13->14, 16->17, 19->20, 22->22 [States]
17:   Ranges 17->17, 19->20, 22->23 [States]
17:   Range 3->19 [States]
17:   Point 12 [States]
17:   Range 4->13 [States]
17:   Range 3->17 [Stores]
17:   Ranges 4->5, 7->8, 10->11, 13->14, 16->17 [Stores]
17:   Point 9 [Stores]
17:   Range 6->10 [Stores]
17:   Point 10 [States]
17:   Range 3->9 [States]
17:   Point 19 [States]
17:   Point 6 [Stores]
17:   Range 6->10 [States]
17:   Point 20 [States]
17:   Ranges 16->17, 19->21 [States]
17:   Point 6 [Stores]
17:   Range 14->20 [Stores]
17:   Range 8->15 [Stores]
17:   Point 21 [States]
17:   Range 13->18 [States]
17:   Ranges 5->5, 7->8, 10->11, 13->14 [States]
17:   Range 11->11 [Stores]
17:   Range 5->16 [States]
17:   Point 3 [Stores]
17:   Range 5->13 [States]
17:   Range 13->20 [Stores]
17:   Range 13->20 [States]
17:   Ranges 5->6 [States]
17:   Point 12 [Stores]
17:   Point 22 [States]
17:   Range 14->20 [Stores]
17:   Point 19 [Stores]
17:   Range 17->20 [Stores]
17:   Point 7 [Stores]
17:   Point 22 [States]
17:   Ranges 10->11, 13->14, 16->18 [States]
17:   Ranges 3->5, 7->8, 10->12 [States]
17:   Range 5->12 [States]
17:   Ranges 7->8, 10->11, 13->14, 16->18 [States]
17:   Point 14 [States]
17:   Point 14 [Stores]
17:   Point 7 [Stores]
17:   Range 4->10 [Stores]
17:   Ranges 15->17, 19->20 [States]
17:   Ranges 7->8, 10->11, 13->14, 16->16 [Stores]
17:   Ranges 12->15 [Stores]
17:   Range 7->22 [Stores]
17:   Point 7 [Stores]
17:   Point 15 [Stores]
17:   Range 17->19 [Stores]
17:   Range 21->22 [States]
17:   Range 11->21 [States]
17:   Point 23 [States]
17:   Ranges 14->14, 16->17, 19->20, 22->22 [States]
17:   Ranges 5->6 [States]
17:   Point 14 [Stores]
17:   Point 14 [States]
17:   Ranges 10->11, 13->14, 16->17 [States]
17:   Range 14->22 [States]
17:   Range 4->20 [States]
17:   Range 4->11 [Stores]
17:   Ranges 8->8, 10->11, 13->14, 16->17 [States]
17:   Ranges 16->17, 19->20, 22->23 [States]
17:   Range 3->21 [Stores]
17:   Ranges 11->11, 13->14, 16->16 [Stores]
17:   Range 10->21 [States]
17:   Ranges 10->11, 13->14, 16->17, 19->19 [States]
17:   Range 7->11 [Stores]
17:   Range 16->23 [Stores]
17:   Ranges 11->11, 13->14, 16->17, 19->19 [Stores]
17:   Range 12->20 [States]
17:   Ranges 18->18 [Stores]
17:   Ranges 10->11, 13->14, 16->17, 19->20, 22->22 [States]
17:   Range 3->8 [States]
17: libc++abi: terminating due to uncaught exception of type doctest::detail::TestFailureException
17: ../src/node/test/historical_queries.cpp:1296: FATAL ERROR: REQUIRE( fetch_until_timeout(fetch_result, check_result, error_printer) ) is NOT correct!
17:   values: REQUIRE( false )
17: 
17: ../src/node/test/historical_queries.cpp:1082: FATAL ERROR: test case CRASHED: SIGABRT - Abort (abnormal termination) signal
17: 
17: ===============================================================================
17: [doctest] test cases:      6 |      5 passed | 1 failed | 3 skipped
17: [doctest] assertions: 425668 | 425666 passed | 2 failed |
17: [doctest] Status: FAILURE!
17: libc++abi: terminating due to uncaught exception of type doctest::detail::TestFailureException
    Test #17: historical_queries_test ..........Child aborted***Exception:   5.25 sec

microsoft / CCF

Timeout in historical_queries_test #5417