search

InsightSoftwareConsortium / ITK

Insight Toolkit (ITK) -- Official Repository. ITK builds on a proven, spatially-oriented architecture for processing, segmentation, and registration of scientific images in two, three, or more dimensions.
https://itk.org
Apache License 2.0
1.37k stars 660 forks source link

BUG: fixed race condition setting m_ANTSAssociate #4625

Open seanm opened 2 months ago

seanm commented 2 months ago

Found with thread sanitizer (TSan) running the itkANTSNeighborhoodCorrelationImageToImageRegistrationTest unit test.

There was already a m_ANTSAssociateOnceFlag flag for use with std::call_once, so I just used that in this place too (it's already used elsewhere).

Partial backtrace from TSan:

WARNING: ThreadSanitizer: data race (pid=79176)
  Write of size 8 at 0x000108f02240 by thread T2:
    #0 void itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution_impl<itk::ThreadedIndexedContainerPartitioner>(itk::IdentityHelper<itk::ThreadedIndexedContainerPartitioner>, itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.hxx:118 (ITKMetricsv4TestDriver:arm64+0x1004bfe50)
    #1 itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution(itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.h:235 (ITKMetricsv4TestDriver:arm64+0x1004bf9c0)
    #2 itk::DomainThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreaderCallback(void*) itkDomainThreader.hxx:123 (ITKMetricsv4TestDriver:arm64+0x100259b6c)
    #3 std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()::operator()() const itkThreadPool.h:92 (ITKMetricsv4TestDriver:arm64+0x1007d3228)
    #4 decltype(std::declval<void* (*&)(void*)>()(std::declval<itk::PoolMultiThreader::ThreadPoolInfoStruct*>())) std::__1::__invoke[abi:v160006]<std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()&>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&) invoke.h:394 (ITKMetricsv4TestDriver:arm64+0x1007d31a4)

  Previous write of size 8 at 0x000108f02240 by thread T14:
    #0 void itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution_impl<itk::ThreadedIndexedContainerPartitioner>(itk::IdentityHelper<itk::ThreadedIndexedContainerPartitioner>, itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.hxx:118 (ITKMetricsv4TestDriver:arm64+0x1004bfe50)
    #1 itk::ANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>, itk::ANTSNeighborhoodCorrelationImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreadedExecution(itk::Index<2u> const&, unsigned int) itkANTSNeighborhoodCorrelationImageToImageMetricv4GetValueAndDerivativeThreader.h:235 (ITKMetricsv4TestDriver:arm64+0x1004bf9c0)
    #2 itk::DomainThreader<itk::ThreadedIndexedContainerPartitioner, itk::ImageToImageMetricv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double, itk::DefaultImageToImageMetricTraitsv4<itk::Image<double, 2u>, itk::Image<double, 2u>, itk::Image<double, 2u>, double>>>::ThreaderCallback(void*) itkDomainThreader.hxx:123 (ITKMetricsv4TestDriver:arm64+0x100259b6c)
    #3 std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()::operator()() const itkThreadPool.h:92 (ITKMetricsv4TestDriver:arm64+0x1007d3228)
    #4 decltype(std::declval<void* (*&)(void*)>()(std::declval<itk::PoolMultiThreader::ThreadPoolInfoStruct*>())) std::__1::__invoke[abi:v160006]<std::__1::future<std::__1::invoke_result<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>::type> itk::ThreadPool::AddWork<void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&)::'lambda'()&>(void* (*&)(void*), itk::PoolMultiThreader::ThreadPoolInfoStruct*&&) invoke.h:394 (ITKMetricsv4TestDriver:arm64+0x1007d31a4)

PR Checklist

Refer to the ITK Software Guide for further development details if necessary.

seanm commented 2 months ago

@dzenanz @hjmjohnson @N-Dekker I don't know this code at all, but I noticed the existence of m_ANTSAssociateOnceFlag already, so this fix seems good to me.

BTW I've created a nightly cdash submission than uses Thread Sanitizer: https://open.cdash.org/viewTest.php?onlyfailed&buildid=9578838

I will triage the other test failures...

seanm commented 1 month ago

Anyone have further thoughts on this one?

seanm commented 1 week ago

Friendly ping. @dzenanz @N-Dekker @hjmjohnson @thewtex any thoughts on this one?

seanm commented 6 days ago

In the big picture, it would be a major improvement to migrate this code to modern C++ lambda's, use thread_local, etc. in terms of readability, maybe performance, and immunity to some of these subtle issues like race conditions.

I don't actually use this class, so while I agree, realistically I won't have time to volunteer for that. :)

What I have been trying to spend a bit of time on though is get all ITK tests passing with TSan.

10 to go: https://open.cdash.org/viewTest.php?onlyfailed&buildid=9719623