search

swiss-seismological-service / scdetect

A computationally efficient earthquake detection module for SeisComP
https://scdetect.readthedocs.io
GNU Affero General Public License v3.0
15 stars 6 forks source link

Amplitude calculation crushes #97

Closed mmesim closed 2 years ago

mmesim commented 2 years ago

Well, I was trying to run scdetect-cc as :

When running the same dataset without requiring magnitude calculation, I was able to get results, i.e 3330 origins -> 1109 unique events.

With the magnitude on 


"createAmpliutdes": true
"createMagnitudes": true

I saw this error many times 

13:58:47 [error] [detector-02::8b18b121-6d0a-4a80-86f9-f8f8e84491aa::aa538897-997e-4219-971c-eb3a9e89a256] [8D.RAW2..HHN] Failed to feed data (tw.start=2019-11-05T01:14:32.455Z, tw.end=2019-11-05T01:14:34.355Z). Reason: status=101, status_value=0.000000 (failed to feed data)

At some point it crushed 

=================================================================
==3740==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060011426e8 at pc 0x560006a50d55 bp 0x7ffc649ae000 sp 0x7ffc649adff0
READ of size 8 at 0x6060011426e8 thread T0
    #0 0x560006a50d54 in std::__shared_ptr<Seiscomp::detect::Application::DetectionItem, (__gnu_cxx::_Lock_policy)2>::get() const /usr/include/c++/9/bits/shared_ptr_base.h:1310
    #1 0x560006a3f00b in bool std::operator==<Seiscomp::detect::Application::DetectionItem, Seiscomp::detect::Application::DetectionItem>(std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) /usr/include/c++/9/bits/shared_ptr.h:384
    #2 0x560006a2019d in Seiscomp::detect::Application::removeDetection(std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:1925
    #3 0x560006a1bd39 in Seiscomp::detect::Application::publishAndRemoveDetection(std::shared_ptr<Seiscomp::detect::Application::DetectionItem>&) **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:1752
    #4 0x560006a00642 in Seiscomp::detect::Application::done() **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:507
    #5 0x7f3e9432b564 in Seiscomp::System::Application::exec() **/work/projects/seiscomp/src/base/common/libs/seiscomp/system/application.cpp:1105
    #6 0x7f3e9432b451 in Seiscomp::System::Application::operator()() **/work/projects/seiscomp/src/base/common/libs/seiscomp/system/application.cpp:1079
    #7 0x560006c1133c in main **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/main.cpp:11
    #8 0x7f3e92f9c0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x240b2)
    #9 0x5600069880dd in _start (**/seiscomp/bin/scdetect-cc+0x3960dd)

0x6060011426e8 is located 40 bytes inside of 64-byte region [0x6060011426c0,0x606001142700)
freed by thread T0 here:
    #0 0x7f3e9581a8df in operator delete(void*) (/lib/x86_64-linux-gnu/libasan.so.5+0x1108df)
    #1 0x560006a88ac7 in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> >::deallocate(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true>*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
    #2 0x560006a7f661 in std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> > >::deallocate(std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> >&, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true>*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470
    #3 0x560006a722f4 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> > >::_M_deallocate_node_ptr(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true>*) /usr/include/c++/9/bits/hashtable_policy.h:2113
    #4 0x560006a63511 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true>*) /usr/include/c++/9/bits/hashtable_policy.h:2103
    #5 0x560006a715ec in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, false> >::_M_erase(unsigned long, std::__detail::_Hash_node_base*, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true>*) (**/seiscomp/bin/scdetect-cc+0x47f5ec)
    #6 0x560006a61642 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, false> >::erase(std::__detail::_Node_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, false, true>) /usr/include/c++/9/bits/hashtable.h:1861
    #7 0x560006a50e5c in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, false> >::erase(std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, false, true>) /usr/include/c++/9/bits/hashtable.h:768
    #8 0x560006a3f0cd in std::unordered_multimap<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Seiscomp::detect::Application::DetectionItem>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> > > >::erase(std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, false, true>) /usr/include/c++/9/bits/unordered_map.h:1691
    #9 0x560006a202cc in Seiscomp::detect::Application::removeDetection(std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:1928
    #10 0x560006a1bd39 in Seiscomp::detect::Application::publishAndRemoveDetection(std::shared_ptr<Seiscomp::detect::Application::DetectionItem>&) **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:1752
    #11 0x560006a00642 in Seiscomp::detect::Application::done() **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:507
    #12 0x7f3e9432b564 in Seiscomp::System::Application::exec() **/work/projects/seiscomp/src/base/common/libs/seiscomp/system/application.cpp:1105
    #13 0x7f3e9432b451 in Seiscomp::System::Application::operator()() **/work/projects/seiscomp/src/base/common/libs/seiscomp/system/application.cpp:1079
    #14 0x560006c1133c in main **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/main.cpp:11
    #15 0x7f3e92f9c0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x240b2)

previously allocated by thread T0 here:
    #0 0x7f3e95819947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x560006a90af6 in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> >::allocate(unsigned long, void const*) (**/seiscomp/bin/scdetect-cc+0x49eaf6)
    #2 0x560006a87a2e in std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> > >::allocate(std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> >&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
    #3 0x560006a7e117 in std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true>* std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, true> > >::_M_allocate_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) (**/seiscomp/bin/scdetect-cc+0x48c117)
    #4 0x560006a710e6 in std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, false, true> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, false> >::_M_emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&>(std::__detail::_Node_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, false, true>, std::integral_constant<bool, false>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) (**/seiscomp/bin/scdetect-cc+0x47f0e6)
    #5 0x560006a6145f in std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, false, true> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, false> >::_M_emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&>(std::integral_constant<bool, false>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) (**/seiscomp/bin/scdetect-cc+0x46f45f)
    #6 0x560006a50c89 in std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, false, true> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, false> >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) (**/seiscomp/bin/scdetect-cc+0x45ec89)
    #7 0x560006a3efb3 in std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> >, false, true> std::unordered_multimap<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Seiscomp::detect::Application::DetectionItem>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> > > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) /usr/include/c++/9/bits/unordered_map.h:1530
    #8 0x560006a1fb0b in Seiscomp::detect::Application::registerDetection(std::shared_ptr<Seiscomp::detect::Application::DetectionItem> const&) **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:1905
    #9 0x560006a08dd1 in Seiscomp::detect::Application::processDetection(Seiscomp::detect::detector::Detector const*, Seiscomp::Record const*, std::unique_ptr<Seiscomp::detect::detector::Detector::Detection const, std::default_delete<Seiscomp::detect::detector::Detector::Detection const> >) **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:932
    #10 0x560006a166fc in operator() **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:1537
    #11 0x560006a25c45 in _M_invoke /usr/include/c++/9/bits/std_function.h:300
    #12 0x560006ba9668 in std::function<void (Seiscomp::detect::detector::Detector const*, Seiscomp::Record const*, std::unique_ptr<Seiscomp::detect::detector::Detector::Detection const, std::default_delete<Seiscomp::detect::detector::Detector::Detection const> >)>::operator()(Seiscomp::detect::detector::Detector const*, Seiscomp::Record const*, std::unique_ptr<Seiscomp::detect::detector::Detector::Detection const, std::default_delete<Seiscomp::detect::detector::Detector::Detection const> >) const /usr/include/c++/9/bits/std_function.h:688
    #13 0x560006ba2f71 in Seiscomp::detect::detector::Detector::emitDetection(Seiscomp::Record const*, std::unique_ptr<Seiscomp::detect::detector::Detector::Detection const, std::default_delete<Seiscomp::detect::detector::Detector::Detection const> >) **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/detector/detector.cpp:537
    #14 0x560006ba3130 in Seiscomp::detect::detector::Detector::processDetections(Seiscomp::Record const*) **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/detector/detector.cpp:543
    #15 0x560006ba0dc5 in Seiscomp::detect::detector::Detector::terminate() **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/detector/detector.cpp:419
    #16 0x560006a00506 in Seiscomp::detect::Application::done() **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/app.cpp:502
    #17 0x7f3e9432b564 in Seiscomp::System::Application::exec() **/work/projects/seiscomp/src/base/common/libs/seiscomp/system/application.cpp:1105
    #18 0x7f3e9432b451 in Seiscomp::System::Application::operator()() **/work/projects/seiscomp/src/base/common/libs/seiscomp/system/application.cpp:1079
    #19 0x560006c1133c in main **/work/projects/seiscomp/src/extras/scdetect/src/apps/cc/main.cpp:11
    #20 0x7f3e92f9c0b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x240b2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/shared_ptr_base.h:1310 in std::__shared_ptr<Seiscomp::detect::Application::DetectionItem, (__gnu_cxx::_Lock_policy)2>::get() const
Shadow bytes around the buggy address:
  0x0c0c80220480: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c0c80220490: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c802204a0: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c802204b0: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c802204c0: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
=>0x0c0c802204d0: fd fd fd fd fa fa fa fa fd fd fd fd fd[fd]fd fd
  0x0c0c802204e0: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c802204f0: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
  0x0c0c80220500: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c80220510: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c80220520: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==3740==ABORTING

Any clue? Is the VM or the code?

damb commented 2 years ago

Thanks for the report.

I saw this error many times 

13:58:47 [error] [detector-02::8b18b121-6d0a-4a80-86f9-f8f8e84491aa::aa538897-997e-4219-971c-eb3a9e89a256] [8D.RAW2..HHN] Failed to feed data (tw.start=2019-11-05T01:14:32.455Z, tw.end=2019-11-05T01:14:34.355Z). Reason: status=101, status_value=0.000000 (failed to feed data)

What's the configured waveformBufferSize? Is there data for the required waveform stream identifier (i.e. 8D.RAW2..HHN) and time window available?

damb commented 2 years ago

Any clue? Is the VM or the code?

There is a memory issue during the application clean-up. Could you provide data including the configuration in order to reproduce the issue? Thanks.

mmesim commented 2 years ago

What's the configured waveformBufferSize? Is there data for the required waveform stream identifier (i.e. 8D.RAW2..HHN) and time window available?

I didn't set any attribute waveformBufferSize, so I guess is the default (?).

mmesim commented 2 years ago

I just found it in scconfig :open_mouth:

It was 300 seconds. I think a value of 60 seconds would be ok, right?

damb commented 2 years ago

It was 300 seconds. I think a value of 60 seconds would be ok, right?

scautopick's default is 300 s (see https://docs.gempa.de/seiscomp/current/apps/scautopick.html#confval-ringBufferSize). For the config you run this shouldn't be an issue (though, less is better in order to reduce scdetect-cc's memory footprint). Note that the required data for amplitude calculation still must be buffered. Else amplitude calculation (and therefore magnitude calculation, too) fails.

damb commented 2 years ago

@mmesim, I had a closer look into the issue.

First of all, the memory issue should be fixed with #100.

Besides, I had a look into the issue causing the error log entry

13:58:47 [error] [detector-02::8b18b121-6d0a-4a80-86f9-f8f8e84491aa::aa538897-997e-4219-971c-eb3a9e89a256] [8D.RAW2..HHN] Failed to feed data (tw.start=2019-11-05T01:14:32.455Z, tw.end=2019-11-05T01:14:34.355Z). Reason: status=101, status_value=0.000000 (failed to feed data)

In order to understand what's going on behind the scenes it is important to remind that data required for amplitude calculation must be still buffered. Though, if trigger facilities are enabled the current detection is emitted (and therefore processed) only once the next detection is available. However, if the time passed by (i.e. the time between the current detection and the next one) was greater than the time specified by processing.waveformBufferSize module configuration parameter no buffered data is available anymore and thus amplitude calculation fails. As a result you see an error log entry as e.g. above. With #100 I handled this case more gracefully. Though, the fact that there is no amplitude calculated cannot be avoided. AFAIU, the issue can be partially avoided only if fetching historical data is implemented (e.g. similar to the approach by scamp).

EDIT: Although, it's just a workaround, increasing the waveformBufferSize should help. Again, note that this will increase scdetect-cc's memory footprint.