search

abseil / abseil-cpp

Abseil Common Libraries (C++)
https://abseil.io
Apache License 2.0
14.77k stars 2.59k forks source link

Failed to lock `absl::Mutex` dring static storage duration #1068

Closed gou4shi1 closed 2 years ago

gou4shi1 commented 2 years ago

Describe the bug I know that static init order is eval, but my company project has copy-and-pasted some files from tensorflow long time ago, if I replace the mutex in the link with absl::Mutex, ASAN fail:

=================================================================
==79194==ERROR: AddressSanitizer: initialization-order-fiasco on address 0x7f0e30e000e0 at pc 0x7f0e30debac4 bp 0x7ffe06a36b90 sp 0x7ffe06a36b88
READ of size 4 at 0x7f0e30e000e0 thread T0
    #0 0x7f0e30debac3 in std::__atomic_base<unsigned int>::load(std::memory_order) const /usr/gcc_toolchain/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/atomic_base.h:419:9
    #1 0x7f0e30debac3 in absl::lts_2020_02_25::base_internal::SpinLock::TryLockImpl() /proc/self/cwd/external/com_google_absl/absl/base/internal/spinlock.h:171:37
    #2 0x7f0e30debac3 in absl::lts_2020_02_25::base_internal::SpinLock::Lock() /proc/self/cwd/external/com_google_absl/absl/base/internal/spinlock.h:85:10
    #3 0x7f0e30debac3 in absl::lts_2020_02_25::base_internal::SpinLockHolder::SpinLockHolder(absl::lts_2020_02_25::base_internal::SpinLock*) /proc/self/cwd/external/com_google_absl/absl/base/internal/spinlock.h:187:8
    #4 0x7f0e30debac3 in absl::lts_2020_02_25::synchronization_internal::NewThreadIdentity() /proc/self/cwd/external/com_google_absl/absl/synchronization/internal/create_thread_identity.cc:102:35
    #5 0x7f0e30debac3 in absl::lts_2020_02_25::synchronization_internal::CreateThreadIdentity() /proc/self/cwd/external/com_google_absl/absl/synchronization/internal/create_thread_identity.cc:129:45
    #6 0x7f0e30deff2b in absl::lts_2020_02_25::synchronization_internal::GetOrCreateCurrentThreadIdentity() /proc/self/cwd/external/com_google_absl/absl/synchronization/internal/create_thread_identity.h:51:12
    #7 0x7f0e30deff2b in absl::lts_2020_02_25::Synch_GetPerThread() /proc/self/cwd/external/com_google_absl/absl/synchronization/mutex.cc:525:30
    #8 0x7f0e30deff2b in absl::lts_2020_02_25::Synch_GetAllLocks() /proc/self/cwd/external/com_google_absl/absl/synchronization/mutex.cc:541:23
    #9 0x7f0e30deff2b in absl::lts_2020_02_25::DeadlockCheck(absl::lts_2020_02_25::Mutex*) /proc/self/cwd/external/com_google_absl/absl/synchronization/mutex.cc:1318:31
    #10 0x7f0e30deff2b in absl::lts_2020_02_25::DebugOnlyDeadlockCheck(absl::lts_2020_02_25::Mutex*) /proc/self/cwd/external/com_google_absl/absl/synchronization/mutex.cc:1401:12
    #11 0x7f0e30def142 in absl::lts_2020_02_25::Mutex::Lock() /proc/self/cwd/external/com_google_absl/absl/synchronization/mutex.cc:1460:16
    #12 0x7f0e3c998231 in absl::lts_2020_02_25::MutexLock::MutexLock(absl::lts_2020_02_25::Mutex*) /proc/self/cwd/external/com_google_absl/absl/synchronization/mutex.h:542:16
    #13 0x7f0e3c998231 in base::FileSystemRegistryImpl::Register(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<base::FileSystem* ()>) /proc/self/cwd/.../env.cc:45:19
    #14 0x7f0e3c999b11 in base::Env::RegisterFileSystem(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<base::FileSystem* ()>) /proc/self/cwd/.../env.cc:92:33
    #15 0x7f0e3c9af6c3 in base::register_file_system::Register<base::PosixFileSystem>::Register(base::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /proc/self/cwd/.../env.h:412:10
    #16 0x7f0e3c9b0b66 in __cxx_global_var_init.1 /proc/self/cwd/.../posix_env.cc:113:1
    #17 0x7f0e3c9b0b66 in _GLOBAL__sub_I_posix_env.cc /proc/self/cwd/.../posix_env.cc
    #18 0x7f0e3cf636f9  (/lib64/ld-linux-x86-64.so.2+0x106f9)

0x7f0e30e000e0 is located 0 bytes inside of global variable 'absl::lts_2020_02_25::synchronization_internal::freelist_lock' defined in 'external/com_google_absl/absl/synchronization/internal/create_thread_identity.cc:35:32' (0x7f0e30e000e0) of size 4
  registered at:
    #0 0x55a6ebd1074d in __asan_register_globals (...)
    #1 0x7f0e30debc0b in asan.module_ctor (/.../libexternal_Scom_Ugoogle_Uabsl_Sabsl_Ssynchronization_Slibsynchronization.so+0xfc0b)

SUMMARY: AddressSanitizer: initialization-order-fiasco /usr/gcc_toolchain/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/atomic_base.h:419:9 in std::__atomic_base<unsigned int>::load(std::memory_order) const
Shadow bytes around the buggy address:
  0x0fe2461b7fc0: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x0fe2461b7fd0: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x0fe2461b7fe0: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x0fe2461b7ff0: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x0fe2461b8000: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
=>0x0fe2461b8010: f9 f9 f9 f9 f9 f9 f9 f9 00 00 00 00[f6]f6 f6 f6
  0x0fe2461b8020: f6 f6 f6 f6 00 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x0fe2461b8030: 04 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 f9 f9 f9 f9
  0x0fe2461b8040: 00 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 00 f9 f9 f9
  0x0fe2461b8050: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe2461b8060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==79194==ABORTING

Steps to reproduce the bug bazel test -c dbg --config asan in my project.

What version of Abseil are you using? lts_2020_02_25

What operating system and version are you using Linux 4.10.0-42-generic Ubuntu 16.04.1

What compiler and version are you using?

Ubuntu clang version 11.1.0-++20210314110124+1fdec59bffc1-1~exp1~20210314220751.162
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/6
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/6.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/6
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/6.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5.5.0
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Candidate multilib: x32;@mx32
Selected multilib: .;@m64
Found CUDA installation: /opt/.../third_party/cuda_artifact/cuda-11.2.tar/content/cuda-11.2, version 11.0

What build system are you using? bazel 3.7.2

derekmauro commented 2 years ago

Static/global absl::Mutex objects either need to be absl::kConstInit, or you have to use a safe idiom like

absl::Mutex& GetGlobalMutex() {
  static absl::Mutex* mu = new absl::Mutex;
  return *mu;
}
gou4shi1 commented 2 years ago

Thanks for the reply, but in this case, the mutex is a member var of a class, and the static obj of that class is init with safe idiom, the problem is that it also calls a func other than ctor and the mutex is locked in that func

derekmauro commented 2 years ago

We need a complete reproduction before we can investigate.