search

PowerDNS / pdns

PowerDNS Authoritative, PowerDNS Recursor, dnsdist
https://www.powerdns.com/
GNU General Public License v2.0
3.73k stars 914 forks source link

[tsan] Possible data race in AuthQueryCache::cleanupIfNeeded() and AuthPacketCache::cleanupIfNeeded() #11816

Open darnuria opened 2 years ago

darnuria commented 2 years ago

Short description

Get 2 possible datarace (tsan report in AuthQueryCache + AuthPacketCache)

Environment

Steps to reproduce

  1. ./configure --with-dynmodules= --with-modules="gsqlite3" --enable-malloc-trace --enable-ubsan CXXFLAGS='-ggdb -DHAVE_CXX11' CFLAGS='-DHAVE_CXX11' YAHTTP_CFLAGS='-DHAVE_CXX11' --enable-coverage --enable-tsan --enable-unit-tests
  2. setup an empty sqlite3 database
  3. run with ./pdns/pdns_server --launch=gsqlite3 --module-dir=./modules/gsqlite3backend.libs --socket-dir=/tmp/sqlite3 --local-port=5354 --allow-axfr-ips='0.0.0.0/0,::/0' --loglevel=9 --no-config --zone-cache-refresh-interval=0 --gsqlite3-database=./pdns_sqlite3.db (database will be empty)
  4. run dnspyre -n 1 -c 8 -s localhost:5354 --recurse -t SOA @domains.txt --distribution --color --plot . (no domain will be found)

Also reproducible with our backend I only reported the minimal case with sqlite3 same report from tsan.

Expected behaviour

No datarace or indication that's it's an Okayish one.

Actual behaviour

Get datarace in Tsan

Tests Regression/unit

Not observable in tests or default regressionnal test (maybe I did'nt managed to run them all).

Other information

Possible datarace on QC

Edited the paste to lastest master commit

SUMMARY: ThreadSanitizer: data race $HOME/pdns/pdns/auth-querycache.cc:212
in AuthQueryCache::cleanupIfNeeded()

see: https://github.com/PowerDNS/pdns/blob/master/pdns/auth-querycache.cc#L212

==================
WARNING: ThreadSanitizer: data race (pid=1449746)
  Read of size 8 at 0x555e0d5b56c8 by thread T6:
    #0 AuthQueryCache::cleanupIfNeeded() $HOME/pdns/pdns/auth-querycache.cc:212 (pdns_server+0x1e9d312)
    #1 AuthQueryCache::getEntry(DNSName const&, QType const&, std::vector<DNSZoneRecord, std::allocator<DNSZoneRecord> >&, int) $HOME/pdns/pdns/auth-querycache.cc:57 (pdns_server+0x1e9a93a)
    #2 UeberBackend::cacheHas(UeberBackend::Question const&, std::vector<DNSZoneRecord, std::allocator<DNSZoneRecord> >&) $HOME/pdns/pdns/ueberbackend.cc:609 (pdns_server+0x2e6c767)
    #3 UeberBackend::getAuth(DNSName const&, QType const&, SOAData*, bool) $HOME/pdns/pdns/ueberbackend.cc:413 (pdns_server+0x2e65cea)
    #4 PacketHandler::doQuestion(DNSPacket&) $HOME/pdns/pdns/packethandler.cc:1476 (pdns_server+0x2b894d3)
    #5 PacketHandler::question(DNSPacket&) $HOME/pdns/pdns/packethandler.cc:1161 (pdns_server+0x2b81766)
    #6 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::distribute(int) <null> (pdns_server+0x2080fef)
    #7 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}::operator()() const <null> (pdns_server+0x2076753)
    #8 void std::__invoke_impl<void, MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>(std::__invoke_other, MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}&&) <null> (pdns_server+0x20a8c03)
    #9 std::__invoke_result<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>::type std::__invoke<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>(MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}&&) <null> (pdns_server+0x20a70f4)
    #10 void std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (pdns_server+0x20a5426)
    #11 std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> >::operator()() <null> (pdns_server+0x209ec8c)
    #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> > >::_M_run() /usr/include/c++/12.1.0/bits/std_thread.h:210 (pdns_server+0x209b4b3)
    #13 execute_native_thread_routine /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:82 (libstdc++.so.6+0xd6182)

  Previous write of size 8 at 0x555e0d5b56c8 by thread T5:
    #0 AuthQueryCache::cleanupIfNeeded() $HOME/pdns/pdns/auth-querycache.cc:237 (pdns_server+0x1e9de48)
    #1 AuthQueryCache::getEntry(DNSName const&, QType const&, std::vector<DNSZoneRecord, std::allocator<DNSZoneRecord> >&, int) $HOME/pdns/pdns/auth-querycache.cc:57 (pdns_server+0x1e9a93a)
    #2 UeberBackend::cacheHas(UeberBackend::Question const&, std::vector<DNSZoneRecord, std::allocator<DNSZoneRecord> >&) $HOME/pdns/pdns/ueberbackend.cc:609 (pdns_server+0x2e6c767)
    #3 UeberBackend::getAuth(DNSName const&, QType const&, SOAData*, bool) $HOME/pdns/pdns/ueberbackend.cc:413 (pdns_server+0x2e65cea)
    #4 PacketHandler::doQuestion(DNSPacket&) $HOME/pdns/pdns/packethandler.cc:1476 (pdns_server+0x2b894d3)
    #5 PacketHandler::question(DNSPacket&) $HOME/pdns/pdns/packethandler.cc:1161 (pdns_server+0x2b81766)
    #6 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::distribute(int) <null> (pdns_server+0x2080fef)
    #7 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}::operator()() const <null> (pdns_server+0x2076753)
    #8 void std::__invoke_impl<void, MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>(std::__invoke_other, MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}&&) <null> (pdns_server+0x20a8c03)
    #9 std::__invoke_result<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>::type std::__invoke<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>(MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}&&) <null> (pdns_server+0x20a70f4)
    #10 void std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (pdns_server+0x20a5426)
    #11 std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> >::operator()() <null> (pdns_server+0x209ec8c)
    #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> > >::_M_run() /usr/include/c++/12.1.0/bits/std_thread.h:210 (pdns_server+0x209b4b3)
    #13 execute_native_thread_routine /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:82 (libstdc++.so.6+0xd6182)

  Location is global 'QC' of size 88 at 0x555e0d5b5680 (pdns_server+0x49116c8)

  Thread T6 'pdns/distributo' (tid=1449755, running) created by thread T3 at:
    #0 pthread_create /usr/src/debug/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x670c9)
    #1 __gthread_create /usr/src/debug/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663 (libstdc++.so.6+0xd6269)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:147 (libstdc++.so.6+0xd6269)
    #3 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int) <null> (pdns_server+0x207794e)
    #4 Distributor<DNSPacket, DNSPacket, PacketHandler>::Create(int) <null> (pdns_server+0x206df60)
    #5 qthread $HOME/pdns/pdns/common_startup.cc:474 (pdns_server+0x20538db)
    #6 void std::__invoke_impl<void, void (*)(unsigned int), unsigned int>(std::__invoke_other, void (*&&)(unsigned int), unsigned int&&) <null> (pdns_server+0x20a8ef8)
    #7 std::__invoke_result<void (*)(unsigned int), unsigned int>::type std::__invoke<void (*)(unsigned int), unsigned int>(void (*&&)(unsigned int), unsigned int&&) <null> (pdns_server+0x20a74a7)
    #8 void std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) <null> (pdns_server+0x20a5668)
    #9 std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> >::operator()() <null> (pdns_server+0x209ed30)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> > >::_M_run() /usr/include/c++/12.1.0/bits/std_thread.h:210 (pdns_server+0x209b5b5)
    #11 execute_native_thread_routine /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:82 (libstdc++.so.6+0xd6182)

  Thread T5 'pdns/distributo' (tid=1449754, running) created by thread T3 at:
    #0 pthread_create /usr/src/debug/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x670c9)
    #1 __gthread_create /usr/src/debug/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663 (libstdc++.so.6+0xd6269)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:147 (libstdc++.so.6+0xd6269)
    #3 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int) <null> (pdns_server+0x207794e)
    #4 Distributor<DNSPacket, DNSPacket, PacketHandler>::Create(int) <null> (pdns_server+0x206df60)
    #5 qthread $HOME/pdns/pdns/common_startup.cc:474 (pdns_server+0x20538db)
    #6 void std::__invoke_impl<void, void (*)(unsigned int), unsigned int>(std::__invoke_other, void (*&&)(unsigned int), unsigned int&&) <null> (pdns_server+0x20a8ef8)
    #7 std::__invoke_result<void (*)(unsigned int), unsigned int>::type std::__invoke<void (*)(unsigned int), unsigned int>(void (*&&)(unsigned int), unsigned int&&) <null> (pdns_server+0x20a74a7)
    #8 void std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) <null> (pdns_server+0x20a5668)
    #9 std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> >::operator()() <null> (pdns_server+0x209ed30)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> > >::_M_run() /usr/include/c++/12.1.0/bits/std_thread.h:210 (pdns_server+0x209b5b5)
    #11 execute_native_thread_routine /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:82 (libstdc++.so.6+0xd6182)

SUMMARY: ThreadSanitizer: data race $HOME/pdns/pdns/auth-querycache.cc:212 in AuthQueryCache::cleanupIfNeeded()
==================

Possible datarace on: PC

SUMMARY: ThreadSanitizer: data race $HOME/pdns/pdns/auth-packetcache.cc:254 
in AuthPacketCache::cleanupIfNeeded()

See: https://github.com/PowerDNS/pdns/blob/master/pdns/auth-packetcache.cc#L254

==================
WARNING: ThreadSanitizer: data race (pid=1449746)
  Read of size 8 at 0x555e0d5b5668 by thread T5:
    #0 AuthPacketCache::cleanupIfNeeded() $HOME/pdns/pdns/auth-packetcache.cc:254 (pdns_server+0x1e3b232)
    #1 AuthPacketCache::insert(DNSPacket&, DNSPacket&, unsigned int) $HOME/pdns/pdns/auth-packetcache.cc:108 (pdns_server+0x1e38d9d)
    #2 PacketHandler::doQuestion(DNSPacket&) $HOME/pdns/pdns/packethandler.cc:1784 (pdns_server+0x2b90e1e)
    #3 PacketHandler::question(DNSPacket&) $HOME/pdns/pdns/packethandler.cc:1161 (pdns_server+0x2b81766)
    #4 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::distribute(int) <null> (pdns_server+0x2080fef)
    #5 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}::operator()() const <null> (pdns_server+0x2076753)
    #6 void std::__invoke_impl<void, MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>(std::__invoke_other, MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}&&) <null> (pdns_server+0x20a8c03)
    #7 std::__invoke_result<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>::type std::__invoke<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}>(MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}&&) <null> (pdns_server+0x20a70f4)
    #8 void std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (pdns_server+0x20a5426)
    #9 std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> >::operator()() <null> (pdns_server+0x209ec8c)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int)::{lambda()#1}> > >::_M_run() /usr/include/c++/12.1.0/bits/std_thread.h:210 (pdns_server+0x209b4b3)
    #11 execute_native_thread_routine /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:82 (libstdc++.so.6+0xd6182)

  Previous write of size 8 at 0x555e0d5b5668 by thread T3:
    #0 AuthPacketCache::cleanupIfNeeded() $HOME/pdns/pdns/auth-packetcache.cc:279 (pdns_server+0x1e3bd68)
    #1 AuthPacketCache::get(DNSPacket&, DNSPacket&) $HOME/pdns/pdns/auth-packetcache.cc:59 (pdns_server+0x1e3770e)
    #2 qthread $HOME/pdns/pdns/common_startup.cc:552 (pdns_server+0x205525e)
    #3 void std::__invoke_impl<void, void (*)(unsigned int), unsigned int>(std::__invoke_other, void (*&&)(unsigned int), unsigned int&&) <null> (pdns_server+0x20a8ef8)
    #4 std::__invoke_result<void (*)(unsigned int), unsigned int>::type std::__invoke<void (*)(unsigned int), unsigned int>(void (*&&)(unsigned int), unsigned int&&) <null> (pdns_server+0x20a74a7)
    #5 void std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) <null> (pdns_server+0x20a5668)
    #6 std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> >::operator()() <null> (pdns_server+0x209ed30)
    #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> > >::_M_run() /usr/include/c++/12.1.0/bits/std_thread.h:210 (pdns_server+0x209b5b5)
    #8 execute_native_thread_routine /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:82 (libstdc++.so.6+0xd6182)

  Location is global 'PC' of size 96 at 0x555e0d5b5620 (pdns_server+0x4911668)

  Thread T5 'pdns/distributo' (tid=1449754, running) created by thread T3 at:
    #0 pthread_create /usr/src/debug/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x670c9)
    #1 __gthread_create /usr/src/debug/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663 (libstdc++.so.6+0xd6269)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:147 (libstdc++.so.6+0xd6269)
    #3 MultiThreadDistributor<DNSPacket, DNSPacket, PacketHandler>::MultiThreadDistributor(int) <null> (pdns_server+0x207794e)
    #4 Distributor<DNSPacket, DNSPacket, PacketHandler>::Create(int) <null> (pdns_server+0x206df60)
    #5 qthread $HOME/pdns/pdns/common_startup.cc:474 (pdns_server+0x20538db)
    #6 void std::__invoke_impl<void, void (*)(unsigned int), unsigned int>(std::__invoke_other, void (*&&)(unsigned int), unsigned int&&) <null> (pdns_server+0x20a8ef8)
    #7 std::__invoke_result<void (*)(unsigned int), unsigned int>::type std::__invoke<void (*)(unsigned int), unsigned int>(void (*&&)(unsigned int), unsigned int&&) <null> (pdns_server+0x20a74a7)
    #8 void std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) <null> (pdns_server+0x20a5668)
    #9 std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> >::operator()() <null> (pdns_server+0x209ed30)
    #10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(unsigned int), unsigned int> > >::_M_run() /usr/include/c++/12.1.0/bits/std_thread.h:210 (pdns_server+0x209b5b5)
    #11 execute_native_thread_routine /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:82 (libstdc++.so.6+0xd6182)

  Thread T3 'pdns/receiver' (tid=1449752, running) created by main thread at:
    #0 pthread_create /usr/src/debug/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x670c9)
    #1 __gthread_create /usr/src/debug/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663 (libstdc++.so.6+0xd6269)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /usr/src/debug/gcc/libstdc++-v3/src/c++11/thread.cc:147 (libstdc++.so.6+0xd6269)
    #3 mainthread() $HOME/pdns/pdns/common_startup.cc:772 (pdns_server+0x205c9a8)
    #4 main $HOME/pdns/pdns/receiver.cc:677 (pdns_server+0x2bff0ee)

SUMMARY: ThreadSanitizer: data race $HOME/pdns/pdns/auth-packetcache.cc:254 in AuthPacketCache::cleanupIfNeeded()
==================
darnuria commented 2 years ago

Sorry for the edits had the strange idea to do not update the logs, mixed AuthPacketCache with AuthQueryCache while writing the issue (they look similar).