search

oneapi-src / oneTBB

oneAPI Threading Building Blocks (oneTBB)
https://oneapi-src.github.io/oneTBB/
Apache License 2.0
5.73k stars 1.02k forks source link

[TSAN] `tbb::queuing_rw_mutex` race condition #1226

Open natale-p opened 1 year ago

natale-p commented 1 year ago

Hello,

With a simple small test case I'm able to reproduce consistently a problem with tbb::queuing_rw_mutex, version v2021.9.0. TSAN reports a data race, that doesn't happen if I use tbb::queuing_mutex, std::shared_mutex, or std::mutex. The logic behind the test case is that there are two threads, one continuously tries to retrieve the data stored in a shared data structure (the reader), while the other adds the data (the writer). It happens with GCC (from 12.0 to 13.2), but not with clang in macOs.

The program is the following:

#include <cassert>
#include <map>
#include <iostream>
#include <shared_mutex>

#include "tbb/queuing_mutex.h"
#include "tbb/queuing_rw_mutex.h"

template <typename MutexType> class Register {
 public:
  Register() {}

  std::string getClient(uint32_t id) const {
    if constexpr (std::is_same_v<MutexType, tbb::queuing_rw_mutex>) {
      typename MutexType::scoped_lock lock(mutex_, false);
      return getClientUnsafe(id);
    } else if constexpr (std::is_same_v<MutexType, tbb::queuing_mutex>) {
      typename MutexType::scoped_lock lock(mutex_);
      return getClientUnsafe(id);
    } else if constexpr (std::is_same_v<MutexType, std::shared_mutex>) {
      std::shared_lock lock(mutex_);
      return getClientUnsafe(id);
    } else if constexpr (std::is_same_v<MutexType, std::mutex>) {
      std::unique_lock lock(mutex_);
      return getClientUnsafe(id);
    } else {
      assert(1 == 0);
    }
    assert(1 == 0);
  }

  std::string getClientUnsafe(uint32_t id) const {
    auto it = storage_.find(id);
    if (it != storage_.end()) {
      return storage_.at(id);
    }

    return "";
  }

  void addClient(uint32_t id, const std::string& name) {
    if constexpr (std::is_same_v<MutexType, tbb::queuing_rw_mutex>) {
      typename MutexType::scoped_lock lock(mutex_, true);
      addClientUnsafe(id, name);
    } else if constexpr (std::is_same_v<MutexType, tbb::queuing_mutex>) {
      typename MutexType::scoped_lock lock(mutex_);
      addClientUnsafe(id, name);
    } else if constexpr (std::is_same_v<MutexType, std::shared_mutex>) {
      std::unique_lock lock(mutex_);
      addClientUnsafe(id, name);
    } else if constexpr (std::is_same_v<MutexType, std::mutex>) {
      std::unique_lock lock(mutex_);
      addClientUnsafe(id, name);
    } else {
      assert(1 == 0);
    }
  }

  void addClientUnsafe(uint32_t id, const std::string& name) {
    if (!getClientUnsafe(id).empty()) {
      return;
    }

    storage_[id] = name;
  }

 private:
  std::map<uint32_t, std::string> storage_;
  mutable MutexType mutex_;
};

template <typename MutexType> class TestRegister {
 public:
  TestRegister() {
    std::vector<std::thread> threads;
    threads.emplace_back(get, std::ref(register_), 0);
    threads.emplace_back(add, std::ref(register_), 0);

    for (auto& thread : threads) {
      thread.join();
    }
  }

  static void add(Register<MutexType>& r, uint32_t id) { r.addClient(id, std::to_string(id)); }

  static std::string get(Register<MutexType>& r, uint32_t id) {
    std::string name;
    while (name.empty()) {
      name = r.getClient(id);
    }
    return name;
  }

  Register<MutexType> register_;
};

using TestRegisterWithRw = TestRegister<tbb::queuing_rw_mutex>;
using TestRegisterWithNormal = TestRegister<tbb::queuing_mutex>;
using TestRegisterWithStdShared = TestRegister<std::shared_mutex>;
using TestRegisterWithStdMutex = TestRegister<std::mutex>;

static void testWithNormal() {
  std::cout << "Testing tbb::queuing_mutex: " << std::endl;
  TestRegisterWithNormal();
  std::cout << "Done." << std::endl << std::endl;
}

static void testWithRw() {
  std::cout << "Testing tbb::queuing_rw_mutex: " << std::endl;
  TestRegisterWithRw();
  std::cout << "Done." << std::endl << std::endl;
}

static void testWithStdShared() {
  std::cout << "Testing std::shared_mutex: " << std::endl;
  TestRegisterWithStdShared();
  std::cout << "Done." << std::endl << std::endl;
}

static void testWithStdMutex() {
  std::cout << "Testing std::mutex: " << std::endl;
  TestRegisterWithStdMutex();
  std::cout << "Done." << std::endl << std::endl;
}

int main(int argc, char** argv) {
  testWithNormal();
  testWithRw();
  testWithStdShared();
  testWithStdMutex();

  return 0;
}

The output is the following:

Testing tbb::queuing_mutex:
Done.

Testing tbb::queuing_rw_mutex:
==================
WARNING: ThreadSanitizer: data race (pid=8958)
  Read of size 4 at 0x7b1400002870 by thread T3:
    #0 std::less<unsigned int>::operator()(unsigned int const&, unsigned int const&) const <null> (TbbThreadedTest+0x1d3cc) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #1 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_lower_bound(std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const*, std::_Rb_tree_node_base const*, unsigned int const&) const <null> (TbbThreadedTest+0x1e144) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #2 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::find(unsigned int const&) const <null> (TbbThreadedTest+0x1d143) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #3 std::map<unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::find(unsigned int const&) const <null> (TbbThreadedTest+0x1c04b) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 Register<tbb::detail::d1::queuing_rw_mutex>::getClientUnsafe[abi:cxx11](unsigned int) const <null> (TbbThreadedTest+0x1ba89) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 Register<tbb::detail::d1::queuing_rw_mutex>::getClient[abi:cxx11](unsigned int) const <null> (TbbThreadedTest+0x1981c) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 TestRegister<tbb::detail::d1::queuing_rw_mutex>::get[abi:cxx11](Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int) <null> (TbbThreadedTest+0x16eaa) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x23641) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x22be2) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) <null> (TbbThreadedTest+0x22362) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #10 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::operator()() <null> (TbbThreadedTest+0x21ef6) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> > >::_M_run() <null> (TbbThreadedTest+0x21b8e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #12 execute_native_thread_routine /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xe1942) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)

  Previous write of size 8 at 0x7b1400002870 by thread T4:
    #0 operator new(unsigned long) /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_new_delete.cpp:64 (libtsan.so.2+0x932ff) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e)
    #1 std::__new_allocator<std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::allocate(unsigned long, void const*) <null> (TbbThreadedTest+0x21230) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #2 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_get_node() <null> (TbbThreadedTest+0x201d8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #3 std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >* std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1f5e0) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_Auto_node::_Auto_node<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1e77a) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 std::_Rb_tree_iterator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1da34) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 std::map<unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](unsigned int const&) <null> (TbbThreadedTest+0x1c59b) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 Register<tbb::detail::d1::queuing_rw_mutex>::addClientUnsafe(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (TbbThreadedTest+0x1bc34) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 Register<tbb::detail::d1::queuing_rw_mutex>::addClient(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (TbbThreadedTest+0x19e63) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 TestRegister<tbb::detail::d1::queuing_rw_mutex>::add(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int) <null> (TbbThreadedTest+0x17276) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #10 void std::__invoke_impl<void, void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__invoke_other, void (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x23547) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #11 std::__invoke_result<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>::type std::__invoke<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(void (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x22a7e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #12 void std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) <null> (TbbThreadedTest+0x222b8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #13 std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::operator()() <null> (TbbThreadedTest+0x21e96) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> > >::_M_run() <null> (TbbThreadedTest+0x21b2e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #15 execute_native_thread_routine /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xe1942) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)

  Location is heap block of size 72 at 0x7b1400002850 allocated by thread T4:
    #0 operator new(unsigned long) /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_new_delete.cpp:64 (libtsan.so.2+0x932ff) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e)
    #1 std::__new_allocator<std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::allocate(unsigned long, void const*) <null> (TbbThreadedTest+0x21230) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #2 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_get_node() <null> (TbbThreadedTest+0x201d8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #3 std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >* std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1f5e0) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_Auto_node::_Auto_node<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1e77a) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 std::_Rb_tree_iterator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1da34) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 std::map<unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](unsigned int const&) <null> (TbbThreadedTest+0x1c59b) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 Register<tbb::detail::d1::queuing_rw_mutex>::addClientUnsafe(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (TbbThreadedTest+0x1bc34) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 Register<tbb::detail::d1::queuing_rw_mutex>::addClient(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (TbbThreadedTest+0x19e63) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 TestRegister<tbb::detail::d1::queuing_rw_mutex>::add(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int) <null> (TbbThreadedTest+0x17276) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #10 void std::__invoke_impl<void, void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__invoke_other, void (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x23547) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #11 std::__invoke_result<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>::type std::__invoke<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(void (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x22a7e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #12 void std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) <null> (TbbThreadedTest+0x222b8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #13 std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::operator()() <null> (TbbThreadedTest+0x21e96) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> > >::_M_run() <null> (TbbThreadedTest+0x21b2e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #15 execute_native_thread_routine /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xe1942) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)

  Thread T3 (tid=8962, running) created by main thread at:
    #0 pthread_create /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1036 (libtsan.so.2+0x44219) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e)
    #1 __gthread_create /usr/src/debug/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663 (libstdc++.so.6+0xe1a29) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:172 (libstdc++.so.6+0xe1a29)
    #3 void std::vector<std::thread, std::allocator<std::thread> >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(__gnu_cxx::__normal_iterator<std::thread*, std::vector<std::thread, std::allocator<std::thread> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x19b43) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 std::thread& std::vector<std::thread, std::allocator<std::thread> >::emplace_back<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x171e6) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 TestRegister<tbb::detail::d1::queuing_rw_mutex>::TestRegister() <null> (TbbThreadedTest+0x15774) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test::TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test() <null> (TbbThreadedTest+0x15024) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 testing::internal::TestFactoryImpl<TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test>::CreateTest() <null> (TbbThreadedTest+0x15099) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /home/nat/Work/tbb/build-arch/ext/gtest/src/buildgtest/googletest/src/gtest.cc:2621 (TbbThreadedTest+0x66090) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 main /home/nat/Work/tbb/test/registry/TbbThreadedTest.cpp:111 (TbbThreadedTest+0x108c8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)

  Thread T4 (tid=8963, finished) created by main thread at:
    #0 pthread_create /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1036 (libtsan.so.2+0x44219) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e)
    #1 __gthread_create /usr/src/debug/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663 (libstdc++.so.6+0xe1a29) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:172 (libstdc++.so.6+0xe1a29)
    #3 void std::vector<std::thread, std::allocator<std::thread> >::_M_realloc_insert<void (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(__gnu_cxx::__normal_iterator<std::thread*, std::vector<std::thread, std::allocator<std::thread> > >, void (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x1a123) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 std::thread& std::vector<std::thread, std::allocator<std::thread> >::emplace_back<void (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(void (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x1751e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 TestRegister<tbb::detail::d1::queuing_rw_mutex>::TestRegister() <null> (TbbThreadedTest+0x157b6) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test::TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test() <null> (TbbThreadedTest+0x15024) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 testing::internal::TestFactoryImpl<TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test>::CreateTest() <null> (TbbThreadedTest+0x15099) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /home/nat/Work/tbb/build-arch/ext/gtest/src/buildgtest/googletest/src/gtest.cc:2621 (TbbThreadedTest+0x66090) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 main /home/nat/Work/tbb/test/registry/TbbThreadedTest.cpp:111 (TbbThreadedTest+0x108c8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)

SUMMARY: ThreadSanitizer: data race (/home/nat/Work/tbb/build-arch/test/TbbThreadedTest+0x1d3cc) (BuildId: 2587178523925e7d620936b69c23077933c789f0) in std::less<unsigned int>::operator()(unsigned int const&, unsigned int const&) const
==================
==================
WARNING: ThreadSanitizer: data race (pid=8958)
  Read of size 8 at 0x7b1400002860 by thread T3:
    #0 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_S_left(std::_Rb_tree_node_base const*) <null> (TbbThreadedTest+0x1f2d5) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #1 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_lower_bound(std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const*, std::_Rb_tree_node_base const*, unsigned int const&) const <null> (TbbThreadedTest+0x1e15f) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #2 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::find(unsigned int const&) const <null> (TbbThreadedTest+0x1d143) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #3 std::map<unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::find(unsigned int const&) const <null> (TbbThreadedTest+0x1c04b) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 Register<tbb::detail::d1::queuing_rw_mutex>::getClientUnsafe[abi:cxx11](unsigned int) const <null> (TbbThreadedTest+0x1ba89) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 Register<tbb::detail::d1::queuing_rw_mutex>::getClient[abi:cxx11](unsigned int) const <null> (TbbThreadedTest+0x1981c) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 TestRegister<tbb::detail::d1::queuing_rw_mutex>::get[abi:cxx11](Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int) <null> (TbbThreadedTest+0x16eaa) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::__invoke_impl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__invoke_other, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x23641) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 std::__invoke_result<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>::type std::__invoke<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x22be2) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) <null> (TbbThreadedTest+0x22362) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #10 std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::operator()() <null> (TbbThreadedTest+0x21ef6) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> > >::_M_run() <null> (TbbThreadedTest+0x21b8e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #12 execute_native_thread_routine /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xe1942) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)

  Previous write of size 8 at 0x7b1400002860 by thread T4:
    #0 operator new(unsigned long) /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_new_delete.cpp:64 (libtsan.so.2+0x932ff) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e)
    #1 std::__new_allocator<std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::allocate(unsigned long, void const*) <null> (TbbThreadedTest+0x21230) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #2 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_get_node() <null> (TbbThreadedTest+0x201d8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #3 std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >* std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1f5e0) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_Auto_node::_Auto_node<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1e77a) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 std::_Rb_tree_iterator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1da34) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 std::map<unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](unsigned int const&) <null> (TbbThreadedTest+0x1c59b) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 Register<tbb::detail::d1::queuing_rw_mutex>::addClientUnsafe(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (TbbThreadedTest+0x1bc34) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 Register<tbb::detail::d1::queuing_rw_mutex>::addClient(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (TbbThreadedTest+0x19e63) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 TestRegister<tbb::detail::d1::queuing_rw_mutex>::add(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int) <null> (TbbThreadedTest+0x17276) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #10 void std::__invoke_impl<void, void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__invoke_other, void (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x23547) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #11 std::__invoke_result<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>::type std::__invoke<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(void (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x22a7e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #12 void std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) <null> (TbbThreadedTest+0x222b8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #13 std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::operator()() <null> (TbbThreadedTest+0x21e96) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> > >::_M_run() <null> (TbbThreadedTest+0x21b2e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #15 execute_native_thread_routine /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xe1942) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)

  Location is heap block of size 72 at 0x7b1400002850 allocated by thread T4:
    #0 operator new(unsigned long) /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_new_delete.cpp:64 (libtsan.so.2+0x932ff) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e)
    #1 std::__new_allocator<std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::allocate(unsigned long, void const*) <null> (TbbThreadedTest+0x21230) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #2 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_get_node() <null> (TbbThreadedTest+0x201d8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #3 std::_Rb_tree_node<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >* std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1f5e0) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_Auto_node::_Auto_node<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1e77a) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 std::_Rb_tree_iterator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<unsigned int const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::piecewise_construct_t const&, std::tuple<unsigned int const&>&&, std::tuple<>&&) <null> (TbbThreadedTest+0x1da34) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 std::map<unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::operator[](unsigned int const&) <null> (TbbThreadedTest+0x1c59b) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 Register<tbb::detail::d1::queuing_rw_mutex>::addClientUnsafe(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (TbbThreadedTest+0x1bc34) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 Register<tbb::detail::d1::queuing_rw_mutex>::addClient(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) <null> (TbbThreadedTest+0x19e63) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 TestRegister<tbb::detail::d1::queuing_rw_mutex>::add(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int) <null> (TbbThreadedTest+0x17276) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #10 void std::__invoke_impl<void, void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__invoke_other, void (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x23547) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #11 std::__invoke_result<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>::type std::__invoke<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(void (*&&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x22a7e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #12 void std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) <null> (TbbThreadedTest+0x222b8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #13 std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> >::operator()() <null> (TbbThreadedTest+0x21e96) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int> > >::_M_run() <null> (TbbThreadedTest+0x21b2e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #15 execute_native_thread_routine /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104 (libstdc++.so.6+0xe1942) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)

  Thread T3 (tid=8962, running) created by main thread at:
    #0 pthread_create /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1036 (libtsan.so.2+0x44219) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e)
    #1 __gthread_create /usr/src/debug/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663 (libstdc++.so.6+0xe1a29) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:172 (libstdc++.so.6+0xe1a29)
    #3 void std::vector<std::thread, std::allocator<std::thread> >::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(__gnu_cxx::__normal_iterator<std::thread*, std::vector<std::thread, std::allocator<std::thread> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x19b43) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 std::thread& std::vector<std::thread, std::allocator<std::thread> >::emplace_back<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x171e6) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 TestRegister<tbb::detail::d1::queuing_rw_mutex>::TestRegister() <null> (TbbThreadedTest+0x15774) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test::TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test() <null> (TbbThreadedTest+0x15024) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 testing::internal::TestFactoryImpl<TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test>::CreateTest() <null> (TbbThreadedTest+0x15099) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /home/nat/Work/tbb/build-arch/ext/gtest/src/buildgtest/googletest/src/gtest.cc:2621 (TbbThreadedTest+0x66090) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 main /home/nat/Work/tbb/test/registry/TbbThreadedTest.cpp:111 (TbbThreadedTest+0x108c8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)

  Thread T4 (tid=8963, finished) created by main thread at:
    #0 pthread_create /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1036 (libtsan.so.2+0x44219) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e)
    #1 __gthread_create /usr/src/debug/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663 (libstdc++.so.6+0xe1a29) (BuildId: 207eb738c5976dd9aac1ae0640fc4de5946b547e)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:172 (libstdc++.so.6+0xe1a29)
    #3 void std::vector<std::thread, std::allocator<std::thread> >::_M_realloc_insert<void (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(__gnu_cxx::__normal_iterator<std::thread*, std::vector<std::thread, std::allocator<std::thread> > >, void (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x1a123) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #4 std::thread& std::vector<std::thread, std::allocator<std::thread> >::emplace_back<void (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >, int>(void (&)(Register<tbb::detail::d1::queuing_rw_mutex>&, unsigned int), std::reference_wrapper<Register<tbb::detail::d1::queuing_rw_mutex> >&&, int&&) <null> (TbbThreadedTest+0x1751e) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #5 TestRegister<tbb::detail::d1::queuing_rw_mutex>::TestRegister() <null> (TbbThreadedTest+0x157b6) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #6 TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test::TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test() <null> (TbbThreadedTest+0x15024) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #7 testing::internal::TestFactoryImpl<TestRegisterWithRw_queuing_rw_mutex_withoutProblems_Test>::CreateTest() <null> (TbbThreadedTest+0x15099) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #8 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /home/nat/Work/tbb/build-arch/ext/gtest/src/buildgtest/googletest/src/gtest.cc:2621 (TbbThreadedTest+0x66090) (BuildId: 2587178523925e7d620936b69c23077933c789f0)
    #9 main /home/nat/Work/tbb/test/registry/TbbThreadedTest.cpp:111 (TbbThreadedTest+0x108c8) (BuildId: 2587178523925e7d620936b69c23077933c789f0)

SUMMARY: ThreadSanitizer: data race (/home/nat/Work/tbb/build-arch/test/TbbThreadedTest+0x1f2d5) (BuildId: 2587178523925e7d620936b69c23077933c789f0) in std::_Rb_tree<unsigned int, std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::_Select1st<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::_S_left(std::_Rb_tree_node_base const*)
==================
Done.

Testing std::shared_mutex:
Done.

Testing std::mutex:
Done.

ThreadSanitizer: reported 2 warnings

TSAN is complaining about the access to the map while modifying the map itself when adding the element, as it wasn't protected by the rw_mutex. Do you think I can safely assume that the rw_mutex is race-free, and this is a false positive? Thanks in advance.

JhaShweta1 commented 1 year ago

Hi, could you please share the details of the system you are using for your application. Were you able to find a workaround using other available mutexes?

natale-p commented 1 year ago

Hello @JhaShweta1, the OS is Debian 12 (when I tried GCC 12.1) and ArchLinux when I tried with GCC 13.2. I'm trying clang with macOs, and clang is 15.0.0.

Replacing tbb::queuing_rw_mutex with tbb::queuing_mutex (losing the r/w capability) solves the race. However, this is not something I would like to do. Thanks!

pavelkumbrasev commented 1 year ago

Replacing tbb::queuing_rw_mutex with tbb::queuing_mutex (losing the r/w capability) solves the race. However, this is not something I would like to do. Thanks!

Hi @natale-p, Is there any specific architecture you are targeting? I believe tbb::queuing_rw_mutex should work without issues on x86 and there is possible problems on weaker memory models such as ARM. This particular issue in tbb::queuing_rw_mutex is quite tricky (there are set of complex synchronizations) we still working on it.

natale-p commented 1 year ago

Hello @pavelkumbrasev , this is happening on x86_64.

natale-p commented 9 months ago

Any news on this? Do I need to raise the bug in GCC, as it's not happening with clang?

pavelkumbrasev commented 9 months ago

Hi @natale-p, the issue is still here. I was not able to fix it yet. tbb::queuing_rw_mutex should work fine on x86_64 (because of strict memory model) so you can safely use it on your platform. I will try to return to this issue later.