elastic / ml-cpp

Machine learning C++ code
Other
150 stars 62 forks source link

[ML] Windows. Flaky test - CTrendComponentTest testStepChangeForecasting #2651

Closed edsavage closed 1 week ago

edsavage commented 6 months ago

Every so often, CTrendComponentTest::testStepChangeForecasting fails. This occurs on Windows only, see.

https://buildkite.com/organizations/elastic/analytics/suites/ml-cpp-windows-x86-64/tests/flaky

The mode of failure is:

unexpected exception
system fatal error

UNCAUGHT EXCEPTION:
- file: unknown location
- line: 0

EXCEPTION STACK TRACE: --------------
memory access violation occurred at address 0x21a9672f3e4, while attempting to  read inaccessible data
-------------------------------------

Last checkpoint:
- message: ""testStepChangeForecasting" test entry"
- file: CTrendComponentTest.cc
- line: 362
edsavage commented 4 months ago

With the address sanitizer enabled (compiled with -fsanitizer=address flag) a more detailed description of the error is obtained

==6368==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x119279ea8914 at pc 0x7ffde626b6d2 bp 0x00099dded3b0 sp 0x00099dded3b8
READ of size 4 at 0x119279ea8914 thread T0
    #0 0x7ffde626b6d1 in ml::core::CFloatStorage::operator double(void) const (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\libMlCore.dll+0x18006b6d1)
    #1 0x7ffde7b16366 in ml::maths::time_series::CTrendComponent::shiftLevel(double, __int64, __int64, class std::vector<struct ml::maths::common::CBasicStatistics::SSampleCentralMoments<class ml::core::CFloatStorage, 1>, class std::allocator<struct ml::maths::common::CBasicStatistics::SSampleCentralMoments<class ml::core::CFloatStorage, 1>>> const &, class std::vector<unsigned __int64, class std::allocator<unsigned __int64>> const &, class std::vector<double, class std::allocator<double>> const &) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\libMlMathsTimeSeries.dll+0x1804f6366)
    #2 0x7ff65d9ef3ef  (C:\Users\ed_savage\src\ml-cpp\cmake-build-sanitizer\test\lib\maths\time_series\unittest\Sanitizer\ml_test_maths_time_series.exe+0x1404cf3ef)
    #3 0x7ff65d9e6853  (C:\Users\ed_savage\src\ml-cpp\cmake-build-sanitizer\test\lib\maths\time_series\unittest\Sanitizer\ml_test_maths_time_series.exe+0x1404c6853)
    #4 0x7ffdefffd2c9 in boost::execution_monitor::execute(class boost::function<(void)> const &) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x18000d2c9)
    #5 0x7ffdefffc13b in boost::execution_monitor::catch_signals(class boost::function<(void)> const &) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x18000c13b)
    #6 0x7ffdefffd0a6 in boost::execution_monitor::execute(class boost::function<(void)> const &) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x18000d0a6)
    #7 0x7ffdefffda7c in boost::execution_monitor::vexecute(class boost::function<(void)> const &) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x18000da7c)
    #8 0x7ffdf0019f3c in boost::unit_test::unit_test_monitor_t::execute_and_translate(class boost::function<(void)> const &, unsigned long) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x180029f3c)
    #9 0x7ffdf00057f0 in boost::unit_test::framework::exception_caught(class boost::execution_exception const &) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x1800157f0)
    #10 0x7ffdf00050f5 in boost::unit_test::framework::exception_caught(class boost::execution_exception const &) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x1800150f5)
    #11 0x7ffdf00050f5 in boost::unit_test::framework::exception_caught(class boost::execution_exception const &) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x1800150f5)
    #12 0x7ffdf0007b52 in boost::unit_test::framework::run(unsigned long, bool) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x180017b52)
    #13 0x7ffdf0019c2d in boost::unit_test::unit_test_main(bool (__cdecl *)(void), int, char **const) (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\boost_unit_test_framework-vc142-mt-x64-1_83.dll+0x180029c2d)
    #14 0x7ff65d54ad2d  (C:\Users\ed_savage\src\ml-cpp\cmake-build-sanitizer\test\lib\maths\time_series\unittest\Sanitizer\ml_test_maths_time_series.exe+0x14002ad2d)
    #15 0x7ff65d9f55db  (C:\Users\ed_savage\src\ml-cpp\cmake-build-sanitizer\test\lib\maths\time_series\unittest\Sanitizer\ml_test_maths_time_series.exe+0x1404d55db)
    #16 0x7ffe048c7ac3  (C:\Windows\System32\KERNEL32.DLL+0x180017ac3)
    #17 0x7ffe0719a4e0  (C:\Windows\SYSTEM32\ntdll.dll+0x18005a4e0)

Address 0x119279ea8914 is a wild pointer.
SUMMARY: AddressSanitizer: heap-buffer-overflow (C:\Users\ed_savage\src\ml-cpp\build\distribution\platform\windows-x86_64\bin\libMlCore.dll+0x18006b6d1) in ml::core::CFloatStorage::operator double(void) const
Shadow bytes around the buggy address:
  0x038cc92550d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc92550e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc92550f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc9255100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc9255110: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x038cc9255120: fa fa[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc9255130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc9255140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc9255150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc9255160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x038cc9255170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==6368==ABORTING
edsavage commented 1 week ago

Fixed by #2738