search

apache / trafficserver

Apache Traffic Server™ is a fast, scalable and extensible HTTP/1.1 and HTTP/2 compliant caching proxy server.
https://trafficserver.apache.org/
Apache License 2.0
1.82k stars 804 forks source link

TSan: ink atomic queue not so atomic #11640

Open JosiahWI opened 3 months ago

JosiahWI commented 3 months ago 
WARNING: ThreadSanitizer: data race (pid=9915)
  Atomic write of size 8 at 0xffff933d5bb0 by main thread (mutexes: write M0, write M1, write M2, write M3, write M4):
    #0 ink_atomiclist_push(InkAtomicList*, void*) /trafficserver/src/tscore/ink_queue.cc:559 (traffic_server+0x93d17c) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #1 ProtectedQueue::enqueue(Event*) /trafficserver/src/iocore/eventsystem/ProtectedQueue.cc:52 (traffic_server+0xecee84) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 EventProcessor::schedule(Event*, int) /trafficserver/src/iocore/dns/../eventsystem/P_UnixEventProcessor.h:125 (traffic_server+0xc2c5d0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 EventProcessor::schedule_every(Continuation*, long, int, int, void*) /trafficserver/src/iocore/dns/../eventsystem/P_UnixEventProcessor.h:200 (traffic_server+0xc2ca90) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 RecProcessStart() /trafficserver/src/iocore/eventsystem/RecProcess.cc:189 (traffic_server+0xedc834) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 main /trafficserver/src/traffic_server/traffic_server.cc:2157 (traffic_server+0x8fcca4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)

  Previous read of size 8 at 0xffff933d5bb0 by thread T4 (mutexes: write M5):
    #0 ProtectedQueue::wait(long) /trafficserver/src/iocore/eventsystem/ProtectedQueue.cc:94 (traffic_server+0xecf0b4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #1 EThread::DefaultTailHandler::waitForActivity(long) /trafficserver/include/iocore/eventsystem/EThread.h:393 (traffic_server+0xed33c0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 EThread::execute_regular() /trafficserver/src/iocore/eventsystem/UnixEThread.cc:300 (traffic_server+0xed2510) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 EThread::execute() /trafficserver/src/iocore/eventsystem/UnixEThread.cc:350 (traffic_server+0xed2780) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 spawn_thread_internal /trafficserver/src/iocore/eventsystem/Thread.cc:75 (traffic_server+0xed0220) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 <null> <null> (libtsan.so.2+0x555e4) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)

  Location is heap block of size 1101696 at 0xffff932d5000 allocated by main thread:
    #0 operator new(unsigned long) <null> (libtsan.so.2+0xa7418) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)
    #1 EventProcessor::spawn_event_threads(int, int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:451 (traffic_server+0xed7b7c) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 TasksProcessor::start(int, unsigned long) /trafficserver/src/iocore/eventsystem/Tasks.cc:42 (traffic_server+0xecfa08) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 main /trafficserver/src/traffic_server/traffic_server.cc:2155 (traffic_server+0x8fcca0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)

  Mutex M0 (0xffff97803f90) created at:
    #0 pthread_mutex_init <null> (libtsan.so.2+0x5f218) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)
    #1 ink_mutex_init(pthread_mutex_t*) /trafficserver/src/tscore/ink_mutex.cc:54 (traffic_server+0x93aa0c) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 ProxyMutex::init() /trafficserver/include/iocore/eventsystem/Lock.h:245 (traffic_server+0x8e4df4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 new_ProxyMutex() /trafficserver/include/iocore/eventsystem/Lock.h:677 (traffic_server+0x8e5490) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 Thread::Thread() /trafficserver/src/iocore/eventsystem/Thread.cc:44 (traffic_server+0xecfff4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 EThread::EThread() /trafficserver/src/iocore/eventsystem/UnixEThread.cc:84 (traffic_server+0xed0958) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #6 main /trafficserver/src/traffic_server/traffic_server.cc:1936 (traffic_server+0x8fb688) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)

  Mutex M1 (0xffff97803d10) created at:
    #0 pthread_mutex_init <null> (libtsan.so.2+0x5f218) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)
    #1 ink_mutex_init(pthread_mutex_t*) /trafficserver/src/tscore/ink_mutex.cc:54 (traffic_server+0x93aa0c) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 ProxyMutex::init() /trafficserver/include/iocore/eventsystem/Lock.h:245 (traffic_server+0x8e4df4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 new_ProxyMutex() /trafficserver/include/iocore/eventsystem/Lock.h:677 (traffic_server+0x8e5490) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 Thread::Thread() /trafficserver/src/iocore/eventsystem/Thread.cc:44 (traffic_server+0xecfff4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 EThread::EThread(ThreadType, int) /trafficserver/src/iocore/eventsystem/UnixEThread.cc:89 (traffic_server+0xed0c38) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #6 EventProcessor::spawn_event_threads(int, int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:451 (traffic_server+0xed7bbc) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #7 EventProcessor::start(int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:550 (traffic_server+0xed84f0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #8 main /trafficserver/src/traffic_server/traffic_server.cc:2110 (traffic_server+0x8fc9ac) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)

  Mutex M2 (0xffff97803c90) created at:
    #0 pthread_mutex_init <null> (libtsan.so.2+0x5f218) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)
    #1 ink_mutex_init(pthread_mutex_t*) /trafficserver/src/tscore/ink_mutex.cc:54 (traffic_server+0x93aa0c) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 ProxyMutex::init() /trafficserver/include/iocore/eventsystem/Lock.h:245 (traffic_server+0x8e4df4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 new_ProxyMutex() /trafficserver/include/iocore/eventsystem/Lock.h:677 (traffic_server+0x8e5490) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 Thread::Thread() /trafficserver/src/iocore/eventsystem/Thread.cc:44 (traffic_server+0xecfff4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 EThread::EThread(ThreadType, int) /trafficserver/src/iocore/eventsystem/UnixEThread.cc:89 (traffic_server+0xed0c38) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #6 EventProcessor::spawn_event_threads(int, int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:451 (traffic_server+0xed7bbc) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #7 EventProcessor::start(int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:550 (traffic_server+0xed84f0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #8 main /trafficserver/src/traffic_server/traffic_server.cc:2110 (traffic_server+0x8fc9ac) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)

  Mutex M3 (0xffff97803990) created at:
    #0 pthread_mutex_init <null> (libtsan.so.2+0x5f218) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)
    #1 ink_mutex_init(pthread_mutex_t*) /trafficserver/src/tscore/ink_mutex.cc:54 (traffic_server+0x93aa0c) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 ProxyMutex::init() /trafficserver/include/iocore/eventsystem/Lock.h:245 (traffic_server+0x8e4df4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 new_ProxyMutex() /trafficserver/include/iocore/eventsystem/Lock.h:677 (traffic_server+0x8e5490) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 Thread::Thread() /trafficserver/src/iocore/eventsystem/Thread.cc:44 (traffic_server+0xecfff4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 EThread::EThread(ThreadType, int) /trafficserver/src/iocore/eventsystem/UnixEThread.cc:89 (traffic_server+0xed0c38) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #6 EventProcessor::spawn_event_threads(int, int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:451 (traffic_server+0xed7bbc) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #7 TasksProcessor::start(int, unsigned long) /trafficserver/src/iocore/eventsystem/Tasks.cc:42 (traffic_server+0xecfa08) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #8 main /trafficserver/src/traffic_server/traffic_server.cc:2155 (traffic_server+0x8fcca0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)

  Mutex M4 (0xffff97803910) created at:
    #0 pthread_mutex_init <null> (libtsan.so.2+0x5f218) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)
    #1 ink_mutex_init(pthread_mutex_t*) /trafficserver/src/tscore/ink_mutex.cc:54 (traffic_server+0x93aa0c) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 ProxyMutex::init() /trafficserver/include/iocore/eventsystem/Lock.h:245 (traffic_server+0x8e4df4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 new_ProxyMutex() /trafficserver/include/iocore/eventsystem/Lock.h:677 (traffic_server+0x8e5490) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 Thread::Thread() /trafficserver/src/iocore/eventsystem/Thread.cc:44 (traffic_server+0xecfff4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 EThread::EThread(ThreadType, int) /trafficserver/src/iocore/eventsystem/UnixEThread.cc:89 (traffic_server+0xed0c38) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #6 EventProcessor::spawn_event_threads(int, int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:451 (traffic_server+0xed7bbc) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #7 TasksProcessor::start(int, unsigned long) /trafficserver/src/iocore/eventsystem/Tasks.cc:42 (traffic_server+0xecfa08) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #8 main /trafficserver/src/traffic_server/traffic_server.cc:2155 (traffic_server+0x8fcca0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)

  Mutex M5 (0xffff933d5bd0) created at:
    #0 pthread_mutex_init <null> (libtsan.so.2+0x5f218) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)
    #1 ink_mutex_init(pthread_mutex_t*) /trafficserver/src/tscore/ink_mutex.cc:54 (traffic_server+0x93aa0c) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 ProtectedQueue::ProtectedQueue() /trafficserver/src/iocore/dns/../eventsystem/P_ProtectedQueue.h:38 (traffic_server+0xc2bbd4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 EThread::EThread(ThreadType, int) /trafficserver/src/iocore/eventsystem/UnixEThread.cc:89 (traffic_server+0xed0cc0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 EventProcessor::spawn_event_threads(int, int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:451 (traffic_server+0xed7bbc) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 TasksProcessor::start(int, unsigned long) /trafficserver/src/iocore/eventsystem/Tasks.cc:42 (traffic_server+0xecfa08) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #6 main /trafficserver/src/traffic_server/traffic_server.cc:2155 (traffic_server+0x8fcca0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)

  Thread T4 '[ET_TASK 0]' (tid=9924, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x6073c) (BuildId: 4fc6d4b1d08e85e4139b0e07fb335a013192e924)
    #1 ink_thread_create /trafficserver/include/tscore/ink_thread.h:129 (traffic_server+0xecfdb4) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #2 Thread::start(char const*, void*, unsigned long, std::function<void ()> const&) /trafficserver/src/iocore/eventsystem/Thread.cc:92 (traffic_server+0xed0354) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #3 EventProcessor::spawn_event_threads(int, int, unsigned long) /trafficserver/src/iocore/eventsystem/UnixEventProcessor.cc:469 (traffic_server+0xed7e70) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #4 TasksProcessor::start(int, unsigned long) /trafficserver/src/iocore/eventsystem/Tasks.cc:42 (traffic_server+0xecfa08) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
    #5 main /trafficserver/src/traffic_server/traffic_server.cc:2155 (traffic_server+0x8fcca0) (BuildId: 7206bc40bbaaebae6ec8626a5203b22a73f3cecf)
ywkaras commented 3 months ago

Related issue: https://github.com/apache/trafficserver/issues/7398