Unit Test problem with IcdTest.AnimatorPlayback

[ajm-ska]

It has been discovered that the IcdTest.AnimatorPlayback Unit Test is does not run reliably on Ubuntu 20.04. Sometimes it passes, sometimes it gives a segmentation fault, and sometimes it just hangs indefinitely. The later case makes CI difficult as it holds everything up.

To reproduce the issue, please try running the test multiple times: ./carta_backend_tests --gtest_filter=IcdTest.AnimatorPlayback --gtest_repeat=10

[veggiesaurus]

@markccchiang can you look at this?

Actually, not sure who should investigate this. I suggested @markccchiang as he wrote the test, but I can't really see anything wrong with the test itself.

[markccchiang]

@markccchiang can you look at this?

I will investigate this problem.

[veggiesaurus]

Stack trace for the segfault I get on ubuntu:

Session::WaitingFlowEvent() Session.h:199
AnimationTask::execute() OnMessageTask.cc:41
operator() Threading.cc:53
std::__invoke_impl<void, carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> >(std::__invoke_other, <> &&) invoke.h:60
std::__invoke<carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> >(<> &&) invoke.h:95
std::thread::_Invoker<std::tuple<carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<>) const thread:244
std::thread::_Invoker<std::tuple<carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> > >::operator()() const thread:251
std::thread::_State_impl<std::thread::_Invoker<std::tuple<carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> > > >::_M_run() const thread:195
<unknown> 0x00007f0544093de4
start_thread 0x00007f05446f3609
__clone 0x00007f0543ed2293

The other relevant threads are:

std::mutex::lock() std_mutex.h:103
carta::concurrent_queue<std::pair<std::vector<char, std::allocator<char> >, bool> >::try_pop(std::pair<std::vector<char, std::allocator<char> >, bool>&) Concurrency.h:32
TestSession::TryPopMessagesQueue(std::pair<std::vector<char, std::allocator<char> >, bool>&) BackendModel.cc:18
BackendModel::TryPopMessagesQueue(std::pair<std::vector<char, std::allocator<char> >, bool>&) BackendModel.cc:277
IcdTest::AnimatorPlayback() TestIcd.cc:404
IcdTest_AnimatorPlayback_Test::TestBody() TestIcd.cc:535
void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) 0x000055f1c0d84561
void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) 0x000055f1c0d8455a
testing::Test::Run() 0x000055f1c0d79226
testing::Test::Run() 0x000055f1c0d791e6
testing::TestInfo::Run() 0x000055f1c0d79385
testing::TestInfo::Run() 0x000055f1c0d79370
testing::TestSuite::Run() 0x000055f1c0d7946d
testing::TestSuite::Run() 0x000055f1c0d793d2
testing::internal::UnitTestImpl::RunAllTests() 0x000055f1c0d7998c

casacore::Allocator_private::BulkAllocatorImpl<casacore::casacore_allocator<void*, 32ul> >::deallocate(void**, unsigned long) 0x00007f05464d8344
casacore::CoordinateSystem::~CoordinateSystem() 0x00007f0546214e6b
casacore::SubImage<float>::SubImage(casacore::ImageInterface<float> const&, casacore::Slicer const&, casacore::AxesSpecifier, bool) 0x00007f054593ff2b
carta::FileLoader::GetSlice(casacore::Array<float>&, casacore::Slicer const&) FileLoader.cc:318
Frame::GetSlicerData(casacore::Slicer const&, std::vector<float, std::allocator<float> >&) Frame.cc:1633
Frame::FillImageCache() Frame.cc:336
Frame::SetImageChannels(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) Frame.cc:298
Session::ExecuteAnimationFrameInner() Session.cc:1946
Session::ExecuteAnimationFrame() Session.cc:2010
AnimationTask::execute() OnMessageTask.cc:34
operator() Threading.cc:53
std::__invoke_impl<void, carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> >(std::__invoke_other, <> &&) invoke.h:60
std::__invoke<carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> >(<> &&) invoke.h:95
std::thread::_Invoker<std::tuple<carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<>) const thread:244
std::thread::_Invoker<std::tuple<carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> > >::operator()() const thread:251
std::thread::_State_impl<std::thread::_Invoker<std::tuple<carta::ThreadManager::StartEventHandlingThreads(int)::<lambda()> > > >::_M_run() const thread:195
<unknown> 0x00007f0544093de4
start_thread 0x00007f05446f3609
__clone 0x00007f0543ed2293

[veggiesaurus]

I can't reproduce the same segfault in the initial commit that added the IcdTest.AnimatorPlayback test, but the test doesn't pass, it just gets stuck in an infinite loop (probably waiting for messages that never get sent?)