search

simsong / bulk_extractor

This is the development tree. Production downloads are at:
https://github.com/simsong/bulk_extractor/releases
Other
1.11k stars 187 forks source link

find_ngram_size is being called in multiple threads on the same sbuf #305

Closed simsong closed 2 years ago

simsong commented 2 years ago

Something is wrong here. Multiple calls on sbuf at 0x1065e4663.

Sampling process 60175 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling bulk_extractor (pid 60175) every 1 millisecond
Process:         bulk_extractor [60175]
Path:            /Users/USER/*/bulk_extractor
Load Address:    0x1064d6000
Identifier:      bulk_extractor
Version:         0
Code Type:       X86-64
Platform:        macOS
Parent Process:  zsh [55495]

Date/Time:       2021-12-08 14:21:30.844 -0500
Launch Time:     2021-12-08 14:17:57.561 -0500
OS Version:      macOS 12.0.1 (21A559)
Report Version:  7
Analysis Tool:   /usr/bin/sample

Physical footprint:         1.0G
Physical footprint (peak):  1.0G
----

Call graph:
    1927 Thread_7180493   DispatchQueue_1: com.apple.main-thread  (serial)
    + 1927 start  (in dyld) + 462  [0x1130ec4fe]
    +   1927 main  (in bulk_extractor) + 71  [0x1069362f7]  main.cpp:15
    +     1927 bulk_extractor_main(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_ostream<char, std::__1::char_traits<char> >&, int, char* const*)  (in bulk_extractor) + 49544  [0x10667a2f8]  bulk_extractor.cpp:590
    +       1927 Phase1::phase1_run()  (in bulk_extractor) + 489  [0x10672e8d9]  phase1.cpp:286
    +         1927 Phase1::read_process_sbufs()  (in bulk_extractor) + 3209  [0x10672c879]  phase1.cpp:202
    +           1927 scanner_set::schedule_sbuf(sbuf_t const*)  (in bulk_extractor) + 333  [0x1065e26fd]  scanner_set.cpp:739
    +             1927 thread_pool::push_task(sbuf_t const*)  (in bulk_extractor) + 58  [0x10662053a]  threadpool.cpp:106
    +               1927 thread_pool::push_task(sbuf_t const*, void (*)(scanner_params&))  (in bulk_extractor) + 413  [0x106620a0d]  threadpool.cpp:92
    +                 1927 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)  (in libc++.1.dylib) + 18  [0x7ff8003ccec2]
    +                   1927 wrap_pthread_cond_wait  (in libclang_rt.tsan_osx_dynamic.dylib) + 525  [0x107b6331d]
    +                     1927 __tsan::call_pthread_cancel_with_cleanup(int (*)(void*), void (*)(void*), void*)  (in libclang_rt.tsan_osx_dynamic.dylib) + 62  [0x107ba970e]
    +                       1927 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1224  [0x7ff80046da89]
    +                         1927 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff800433506]
    1927 Thread_7180524
    + 1927 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
    +   1927 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
    +     1923 __tsan::BackgroundThread(void*)  (in libclang_rt.tsan_osx_dynamic.dylib) + 436  [0x107b9b764]
    +     ! 1923 usleep  (in libsystem_c.dylib) + 53  [0x7ff8003487df]
    +     !   1923 nanosleep  (in libsystem_c.dylib) + 196  [0x7ff8003488a9]
    +     !     1923 __semwait_signal  (in libsystem_kernel.dylib) + 10  [0x7ff8004333da]
    +     3 __tsan::BackgroundThread(void*)  (in libclang_rt.tsan_osx_dynamic.dylib) + 441  [0x107b9b769]
    +     ! 2 __sanitizer::internal_memset(void*, int, unsigned long)  (in libclang_rt.tsan_osx_dynamic.dylib) + 353  [0x107b42391]
    +     ! 1 __sanitizer::NanoTime()  (in libclang_rt.tsan_osx_dynamic.dylib) + 38  [0x107b45376]
    +     !   1 gettimeofday  (in libsystem_c.dylib) + 45  [0x7ff80034164a]
    +     !     1 __commpage_gettimeofday_internal  (in libsystem_kernel.dylib) + 31  [0x7ff8004322bb]
    +     1 __tsan::BackgroundThread(void*)  (in libclang_rt.tsan_osx_dynamic.dylib) + 416  [0x107b9b750]
    1927 Thread_7180525
    + 1927 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
    +   1927 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
    +     1927 __tsan_thread_start_func  (in libclang_rt.tsan_osx_dynamic.dylib) + 141  [0x107b62a2d]
    +       1927 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*> >(void*)  (in bulk_extractor) + 118  [0x106629756]  thread:297
    +         1927 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*>&, std::__1::__tuple_indices<2ul>)  (in bulk_extractor) + 79  [0x10662a5ef]  thread:286
    +           1927 decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<void*>(fp0))) std::__1::__invoke<void* (*)(void*), void*>(void* (*&&)(void*), void*&&)  (in bulk_extractor) + 107  [0x10662a72b]  type_traits:3694
    +             1927 worker::start_worker(void*)  (in bulk_extractor) + 46  [0x10661f88e]  threadpool.cpp:142
    +               1927 worker::run()  (in bulk_extractor) + 1388  [0x1066214fc]  threadpool.cpp:195
    +                 1927 scanner_set::process_sbuf(sbuf_t const*, void (*)(scanner_params&))  (in bulk_extractor) + 2035  [0x1065e4663]  scanner_set.cpp:802
    +                   1927 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 68  [0x1065b93f4]  sbuf.cpp:511
    +                     1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 54  [0x1065b5576]  __mutex_base:91
    +                       1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 90  [0x1065c0f2a]  __mutex_base:91
    +                         1927 std::__1::mutex::lock()  (in libc++.1.dylib) + 9  [0x7ff8003ce899]
    +                           1927 wrap_pthread_mutex_lock  (in libclang_rt.tsan_osx_dynamic.dylib) + 228  [0x107b72b54]
    +                             1927 _pthread_mutex_firstfit_lock_slow  (in libsystem_pthread.dylib) + 204  [0x7ff800468d78]
    +                               1927 _pthread_mutex_firstfit_lock_wait  (in libsystem_pthread.dylib) + 76  [0x7ff80046aeaf]
    +                                 1927 __psynch_mutexwait  (in libsystem_kernel.dylib) + 10  [0x7ff800432cf2]
    1927 Thread_7180526
    + 1927 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
    +   1927 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
    +     1927 __tsan_thread_start_func  (in libclang_rt.tsan_osx_dynamic.dylib) + 141  [0x107b62a2d]
    +       1927 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*> >(void*)  (in bulk_extractor) + 118  [0x106629756]  thread:297
    +         1927 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*>&, std::__1::__tuple_indices<2ul>)  (in bulk_extractor) + 79  [0x10662a5ef]  thread:286
    +           1927 decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<void*>(fp0))) std::__1::__invoke<void* (*)(void*), void*>(void* (*&&)(void*), void*&&)  (in bulk_extractor) + 107  [0x10662a72b]  type_traits:3694
    +             1927 worker::start_worker(void*)  (in bulk_extractor) + 46  [0x10661f88e]  threadpool.cpp:142
    +               1927 worker::run()  (in bulk_extractor) + 1388  [0x1066214fc]  threadpool.cpp:195
    +                 1927 scanner_set::process_sbuf(sbuf_t const*, void (*)(scanner_params&))  (in bulk_extractor) + 2035  [0x1065e4663]  scanner_set.cpp:802
    +                   1927 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 68  [0x1065b93f4]  sbuf.cpp:511
    +                     1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 54  [0x1065b5576]  __mutex_base:91
    +                       1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 90  [0x1065c0f2a]  __mutex_base:91
    +                         1927 std::__1::mutex::lock()  (in libc++.1.dylib) + 9  [0x7ff8003ce899]
    +                           1927 wrap_pthread_mutex_lock  (in libclang_rt.tsan_osx_dynamic.dylib) + 228  [0x107b72b54]
    +                             1927 _pthread_mutex_firstfit_lock_slow  (in libsystem_pthread.dylib) + 204  [0x7ff800468d78]
    +                               1927 _pthread_mutex_firstfit_lock_wait  (in libsystem_pthread.dylib) + 76  [0x7ff80046aeaf]
    +                                 1927 __psynch_mutexwait  (in libsystem_kernel.dylib) + 10  [0x7ff800432cf2]
    1927 Thread_7180528
    + 1927 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
    +   1927 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
    +     1927 __tsan_thread_start_func  (in libclang_rt.tsan_osx_dynamic.dylib) + 141  [0x107b62a2d]
    +       1927 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*> >(void*)  (in bulk_extractor) + 118  [0x106629756]  thread:297
    +         1927 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*>&, std::__1::__tuple_indices<2ul>)  (in bulk_extractor) + 79  [0x10662a5ef]  thread:286
    +           1927 decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<void*>(fp0))) std::__1::__invoke<void* (*)(void*), void*>(void* (*&&)(void*), void*&&)  (in bulk_extractor) + 107  [0x10662a72b]  type_traits:3694
    +             1927 worker::start_worker(void*)  (in bulk_extractor) + 46  [0x10661f88e]  threadpool.cpp:142
    +               1927 worker::run()  (in bulk_extractor) + 1388  [0x1066214fc]  threadpool.cpp:195
    +                 1927 scanner_set::process_sbuf(sbuf_t const*, void (*)(scanner_params&))  (in bulk_extractor) + 2035  [0x1065e4663]  scanner_set.cpp:802
    +                   1927 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 68  [0x1065b93f4]  sbuf.cpp:511
    +                     1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 54  [0x1065b5576]  __mutex_base:91
    +                       1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 90  [0x1065c0f2a]  __mutex_base:91
    +                         1927 std::__1::mutex::lock()  (in libc++.1.dylib) + 9  [0x7ff8003ce899]
    +                           1927 wrap_pthread_mutex_lock  (in libclang_rt.tsan_osx_dynamic.dylib) + 228  [0x107b72b54]
    +                             1927 _pthread_mutex_firstfit_lock_slow  (in libsystem_pthread.dylib) + 204  [0x7ff800468d78]
    +                               1927 _pthread_mutex_firstfit_lock_wait  (in libsystem_pthread.dylib) + 76  [0x7ff80046aeaf]
    +                                 1927 __psynch_mutexwait  (in libsystem_kernel.dylib) + 10  [0x7ff800432cf2]
    1927 Thread_7180531
    + 1927 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
    +   1927 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
    +     1927 __tsan_thread_start_func  (in libclang_rt.tsan_osx_dynamic.dylib) + 141  [0x107b62a2d]
    +       1927 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*> >(void*)  (in bulk_extractor) + 118  [0x106629756]  thread:297
    +         1927 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*>&, std::__1::__tuple_indices<2ul>)  (in bulk_extractor) + 79  [0x10662a5ef]  thread:286
    +           1927 decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<void*>(fp0))) std::__1::__invoke<void* (*)(void*), void*>(void* (*&&)(void*), void*&&)  (in bulk_extractor) + 107  [0x10662a72b]  type_traits:3694
    +             1927 worker::start_worker(void*)  (in bulk_extractor) + 46  [0x10661f88e]  threadpool.cpp:142
    +               1927 worker::run()  (in bulk_extractor) + 1388  [0x1066214fc]  threadpool.cpp:195
    +                 1927 scanner_set::process_sbuf(sbuf_t const*, void (*)(scanner_params&))  (in bulk_extractor) + 2035  [0x1065e4663]  scanner_set.cpp:802
    +                   446 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 338  [0x1065b9502]  sbuf.cpp:516
    +                   ! 356 __tsan_read1  (in libclang_rt.tsan_osx_dynamic.dylib) + 159,186,...  [0x107b931af,0x107b931ca,...]
    +                   ! 87 __tsan_read1  (in libclang_rt.tsan_osx_dynamic.dylib) + 22  [0x107b93126]
    +                   ! : 55 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 21,5,...  [0x107ba8aa5,0x107ba8a95,...]
    +                   ! : 32 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 28  [0x107ba8aac]
    +                   ! :   29 pthread_self  (in libsystem_pthread.dylib) + 0,4,...  [0x7ff80046992d,0x7ff800469931,...]
    +                   ! :   3 DYLD-STUB$$pthread_self  (in libclang_rt.tsan_osx_dynamic.dylib) + 0  [0x107baf0c6]
    +                   ! 2 DYLD-STUB$$__tsan_read1  (in bulk_extractor) + 0  [0x106939ffa]
    +                   ! 1 __tsan_read1  (in libclang_rt.tsan_osx_dynamic.dylib) + 435  [0x107b932c3]
    +                   !   1 __tsan::TraceSwitch(__tsan::ThreadState*)  (in libclang_rt.tsan_osx_dynamic.dylib) + 85  [0x107b90775]
    +                   407 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 270  [0x1065b94be]  sbuf.cpp:516
    +                   ! 327 __tsan_read1  (in libclang_rt.tsan_osx_dynamic.dylib) + 63,1602,...  [0x107b9314f,0x107b93752,...]
    +                   ! 74 __tsan_read1  (in libclang_rt.tsan_osx_dynamic.dylib) + 22  [0x107b93126]
    +                   ! : 56 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 78,21,...  [0x107ba8ade,0x107ba8aa5,...]
    +                   ! : 18 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 28  [0x107ba8aac]
    +                   ! :   18 pthread_self  (in libsystem_pthread.dylib) + 0,4,...  [0x7ff80046992d,0x7ff800469931,...]
    +                   ! 6 DYLD-STUB$$__tsan_read1  (in bulk_extractor) + 0  [0x106939ffa]
    +                   361 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 255,292,...  [0x1065b94af,0x1065b94d4,...]  sbuf.cpp:516
    +                   224 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 307  [0x1065b94e3]  sbuf.cpp:516
    +                   ! 139 __tsan_read8  (in libclang_rt.tsan_osx_dynamic.dylib) + 158,86,...  [0x107b948de,0x107b94896,...]
    +                   ! 83 __tsan_read8  (in libclang_rt.tsan_osx_dynamic.dylib) + 22  [0x107b94856]
    +                   ! : 53 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 4,21,...  [0x107ba8a94,0x107ba8aa5,...]
    +                   ! : 30 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 28  [0x107ba8aac]
    +                   ! :   29 pthread_self  (in libsystem_pthread.dylib) + 0,29,...  [0x7ff80046992d,0x7ff80046994a,...]
    +                   ! :   1 DYLD-STUB$$pthread_self  (in libclang_rt.tsan_osx_dynamic.dylib) + 0  [0x107baf0c6]
    +                   ! 2 DYLD-STUB$$__tsan_read8  (in bulk_extractor) + 0  [0x10693a012]
    +                   215 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 225  [0x1065b9491]  sbuf.cpp:516
    +                   ! 137 __tsan_read8  (in libclang_rt.tsan_osx_dynamic.dylib) + 158,136,...  [0x107b948de,0x107b948c8,...]
    +                   ! 71 __tsan_read8  (in libclang_rt.tsan_osx_dynamic.dylib) + 22  [0x107b94856]
    +                   ! : 49 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 84,21,...  [0x107ba8ae4,0x107ba8aa5,...]
    +                   ! : 22 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 28  [0x107ba8aac]
    +                   ! :   21 pthread_self  (in libsystem_pthread.dylib) + 28,0,...  [0x7ff800469949,0x7ff80046992d,...]
    +                   ! :   1 DYLD-STUB$$pthread_self  (in libclang_rt.tsan_osx_dynamic.dylib) + 0  [0x107baf0c6]
    +                   ! 7 DYLD-STUB$$__tsan_read8  (in bulk_extractor) + 0  [0x10693a012]
    +                   204 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 191  [0x1065b946f]  sbuf.cpp:515
    +                   ! 135 __tsan_read8  (in libclang_rt.tsan_osx_dynamic.dylib) + 158,141,...  [0x107b948de,0x107b948cd,...]
    +                   ! 69 __tsan_read8  (in libclang_rt.tsan_osx_dynamic.dylib) + 22  [0x107b94856]
    +                   !   41 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 67,41,...  [0x107ba8ad3,0x107ba8ab9,...]
    +                   !   28 __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 28  [0x107ba8aac]
    +                   !     24 pthread_self  (in libsystem_pthread.dylib) + 0,29,...  [0x7ff80046992d,0x7ff80046994a,...]
    +                   !     4 DYLD-STUB$$pthread_self  (in libclang_rt.tsan_osx_dynamic.dylib) + 0  [0x107baf0c6]
    +                   40 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 182,195,...  [0x1065b9466,0x1065b9473,...]  sbuf.cpp:515
    +                   27 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 209,170  [0x1065b9481,0x1065b945a]  sbuf.cpp:0
    +                   3 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 370  [0x1065b9522]  sbuf.cpp:520
    1927 Thread_7180537
    + 1927 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
    +   1927 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
    +     1927 __tsan_thread_start_func  (in libclang_rt.tsan_osx_dynamic.dylib) + 141  [0x107b62a2d]
    +       1927 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*> >(void*)  (in bulk_extractor) + 118  [0x106629756]  thread:297
    +         1927 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*>&, std::__1::__tuple_indices<2ul>)  (in bulk_extractor) + 79  [0x10662a5ef]  thread:286
    +           1927 decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<void*>(fp0))) std::__1::__invoke<void* (*)(void*), void*>(void* (*&&)(void*), void*&&)  (in bulk_extractor) + 107  [0x10662a72b]  type_traits:3694
    +             1927 worker::start_worker(void*)  (in bulk_extractor) + 46  [0x10661f88e]  threadpool.cpp:142
    +               1927 worker::run()  (in bulk_extractor) + 1388  [0x1066214fc]  threadpool.cpp:195
    +                 1927 scanner_set::process_sbuf(sbuf_t const*, void (*)(scanner_params&))  (in bulk_extractor) + 2035  [0x1065e4663]  scanner_set.cpp:802
    +                   1927 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 68  [0x1065b93f4]  sbuf.cpp:511
    +                     1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 54  [0x1065b5576]  __mutex_base:91
    +                       1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 90  [0x1065c0f2a]  __mutex_base:91
    +                         1927 std::__1::mutex::lock()  (in libc++.1.dylib) + 9  [0x7ff8003ce899]
    +                           1927 wrap_pthread_mutex_lock  (in libclang_rt.tsan_osx_dynamic.dylib) + 228  [0x107b72b54]
    +                             1927 _pthread_mutex_firstfit_lock_slow  (in libsystem_pthread.dylib) + 204  [0x7ff800468d78]
    +                               1927 _pthread_mutex_firstfit_lock_wait  (in libsystem_pthread.dylib) + 76  [0x7ff80046aeaf]
    +                                 1927 __psynch_mutexwait  (in libsystem_kernel.dylib) + 10  [0x7ff800432cf2]
    1927 Thread_7180538
    + 1927 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
    +   1927 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
    +     1927 __tsan_thread_start_func  (in libclang_rt.tsan_osx_dynamic.dylib) + 141  [0x107b62a2d]
    +       1927 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*> >(void*)  (in bulk_extractor) + 118  [0x106629756]  thread:297
    +         1927 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*>&, std::__1::__tuple_indices<2ul>)  (in bulk_extractor) + 79  [0x10662a5ef]  thread:286
    +           1927 decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<void*>(fp0))) std::__1::__invoke<void* (*)(void*), void*>(void* (*&&)(void*), void*&&)  (in bulk_extractor) + 107  [0x10662a72b]  type_traits:3694
    +             1927 worker::start_worker(void*)  (in bulk_extractor) + 46  [0x10661f88e]  threadpool.cpp:142
    +               1927 worker::run()  (in bulk_extractor) + 1388  [0x1066214fc]  threadpool.cpp:195
    +                 1927 scanner_set::process_sbuf(sbuf_t const*, void (*)(scanner_params&))  (in bulk_extractor) + 2035  [0x1065e4663]  scanner_set.cpp:802
    +                   1927 sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 68  [0x1065b93f4]  sbuf.cpp:511
    +                     1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 54  [0x1065b5576]  __mutex_base:91
    +                       1927 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 90  [0x1065c0f2a]  __mutex_base:91
    +                         1927 std::__1::mutex::lock()  (in libc++.1.dylib) + 9  [0x7ff8003ce899]
    +                           1927 wrap_pthread_mutex_lock  (in libclang_rt.tsan_osx_dynamic.dylib) + 228  [0x107b72b54]
    +                             1927 _pthread_mutex_firstfit_lock_slow  (in libsystem_pthread.dylib) + 204  [0x7ff800468d78]
    +                               1927 _pthread_mutex_firstfit_lock_wait  (in libsystem_pthread.dylib) + 76  [0x7ff80046aeaf]
    +                                 1927 __psynch_mutexwait  (in libsystem_kernel.dylib) + 10  [0x7ff800432cf2]
    1927 Thread_7180541
      1927 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
        1927 _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
          1927 __tsan_thread_start_func  (in libclang_rt.tsan_osx_dynamic.dylib) + 141  [0x107b62a2d]
            1927 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (notify_thread::*)(), notify_thread*> >(void*)  (in bulk_extractor) + 118  [0x106727176]  thread:297
              1927 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (notify_thread::*)(), notify_thread*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (notify_thread::*)(), notify_thread*>&, std::__1::__tuple_indices<2ul>)  (in bulk_extractor) + 79  [0x10672801f]  thread:286
                1927 decltype(*(std::__1::forward<notify_thread*>(fp0)).*fp()) std::__1::__invoke<void* (notify_thread::*)(), notify_thread*, void>(void* (notify_thread::*&&)(), notify_thread*&&)  (in bulk_extractor) + 205  [0x1067281bd]  type_traits:3635
                  1926 notify_thread::run()  (in bulk_extractor) + 4372  [0x1067200a4]  notify_thread.cpp:123
                  ! 1926 void std::__1::this_thread::sleep_for<long long, std::__1::ratio<1l, 1l> >(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&)  (in bulk_extractor) + 295  [0x1065dcf17]  thread:386
                  !   1926 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&)  (in libc++.1.dylib) + 73  [0x7ff8003d6c11]
                  !     1926 wrap_nanosleep  (in libclang_rt.tsan_osx_dynamic.dylib) + 354  [0x107b619d2]
                  !       1926 nanosleep  (in libsystem_c.dylib) + 196  [0x7ff8003488a9]
                  !         1926 __semwait_signal  (in libsystem_kernel.dylib) + 10  [0x7ff8004333da]
                  1 notify_thread::run()  (in bulk_extractor) + 3277  [0x10671fc5d]  notify_thread.cpp:101
                    1 std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<<<std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*)  (in bulk_extractor) + 78  [0x1064d726e]  ostream:868
                      1 std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long)  (in bulk_extractor) + 471  [0x1064dffa7]  ostream:729
                        1 std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> > std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char)  (in bulk_extractor) + 590  [0x1064e043e]  locale:1417
                          1 std::__1::basic_streambuf<char, std::__1::char_traits<char> >::sputn(char const*, long)  (in bulk_extractor) + 103  [0x1064e07f7]  streambuf:229
                            1 wrap_fwrite  (in libclang_rt.tsan_osx_dynamic.dylib) + 201  [0x107b694b9]
                              1 __tsan::FuncExit(__tsan::ThreadState*)  (in libclang_rt.tsan_osx_dynamic.dylib) + 71  [0x107b93057]

Total number in stack (recursive counted multiple, when >=5):
        8       _pthread_start  (in libsystem_pthread.dylib) + 125  [0x7ff80046d514]
        8       thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80046902f]
        7       __tsan_thread_start_func  (in libclang_rt.tsan_osx_dynamic.dylib) + 141  [0x107b62a2d]
        6       decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<void*>(fp0))) std::__1::__invoke<void* (*)(void*), void*>(void* (*&&)(void*), void*&&)  (in bulk_extractor) + 107  [0x10662a72b]  type_traits:3694
        6       scanner_set::process_sbuf(sbuf_t const*, void (*)(scanner_params&))  (in bulk_extractor) + 2035  [0x1065e4663]  scanner_set.cpp:802
        6       void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*>&, std::__1::__tuple_indices<2ul>)  (in bulk_extractor) + 79  [0x10662a5ef]  thread:286
        6       void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), void*> >(void*)  (in bulk_extractor) + 118  [0x106629756]  thread:297
        6       worker::run()  (in bulk_extractor) + 1388  [0x1066214fc]  threadpool.cpp:195
        6       worker::start_worker(void*)  (in bulk_extractor) + 46  [0x10661f88e]  threadpool.cpp:142
        5       __psynch_mutexwait  (in libsystem_kernel.dylib) + 0  [0x7ff800432ce8]
        5       __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 0  [0x107ba8a90]
        5       __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib) + 28  [0x107ba8aac]
        5       _pthread_mutex_firstfit_lock_slow  (in libsystem_pthread.dylib) + 204  [0x7ff800468d78]
        5       _pthread_mutex_firstfit_lock_wait  (in libsystem_pthread.dylib) + 76  [0x7ff80046aeaf]
        5       pthread_self  (in libsystem_pthread.dylib) + 0  [0x7ff80046992d]
        5       sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor) + 68  [0x1065b93f4]  sbuf.cpp:511
        5       std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 54  [0x1065b5576]  __mutex_base:91
        5       std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&)  (in bulk_extractor) + 90  [0x1065c0f2a]  __mutex_base:91
        5       std::__1::mutex::lock()  (in libc++.1.dylib) + 9  [0x7ff8003ce899]
        5       wrap_pthread_mutex_lock  (in libclang_rt.tsan_osx_dynamic.dylib) + 228  [0x107b72b54]

Sort by top of stack, same collapsed (when >= 5):
        __psynch_mutexwait  (in libsystem_kernel.dylib)        9635
        __semwait_signal  (in libsystem_kernel.dylib)        3849
        __psynch_cvwait  (in libsystem_kernel.dylib)        1927
        __tsan_read1  (in libclang_rt.tsan_osx_dynamic.dylib)        683
        sbuf_t::find_ngram_size(unsigned long) const  (in bulk_extractor)        431
        __tsan_read8  (in libclang_rt.tsan_osx_dynamic.dylib)        411
        __tsan::cur_thread()  (in libclang_rt.tsan_osx_dynamic.dylib)        254
        pthread_self  (in libsystem_pthread.dylib)        121
        DYLD-STUB$$__tsan_read8  (in bulk_extractor)        9
        DYLD-STUB$$pthread_self  (in libclang_rt.tsan_osx_dynamic.dylib)        9
        DYLD-STUB$$__tsan_read1  (in bulk_extractor)        8

Binary Images:
       0x1064d6000 -        0x106d59ffb +bulk_extractor (0) <F574FF68-5410-3A83-82E1-474952D14162> /Users/*/bulk_extractor
       0x1076b6000 -        0x1077bdff7 +libewf.2.dylib (0) <8A4AA325-3333-375E-B0D4-0D5EB7601315> /usr/local/Cellar/libewf/20140812/lib/libewf.2.dylib
       0x107b36000 -        0x107bc1fff +libclang_rt.tsan_osx_dynamic.dylib (1300.0.29.3) <313B87E2-8905-3C81-8AAD-B10F50F777EB> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.tsan_osx_dynamic.dylib
       0x1130e7000 -        0x11314fc07  dyld (940) <1A6AE033-9438-33C0-8077-988FD885250A> /usr/lib/dyld
    0x7ff8001a3000 -     0x7ff8001a4ff2  libsystem_blocks.dylib (79) <326DD91B-4EF8-3D80-B90B-F50EB7D7FDB8> /usr/lib/system/libsystem_blocks.dylib
    0x7ff8001a5000 -     0x7ff8001dbfff  libxpc.dylib (2235.40.11) <900E1F8B-6333-3E4D-B64C-751264BB7223> /usr/lib/system/libxpc.dylib
    0x7ff8001dc000 -     0x7ff8001f4ffe  libsystem_trace.dylib (1375.40.2) <21CA8683-068C-3B8E-95F1-8DF502725FAA> /usr/lib/system/libsystem_trace.dylib
    0x7ff8001f5000 -     0x7ff800287fff  libcorecrypto.dylib (1217.40.11) <33AD70A8-4EBE-3B93-B442-00C77664628F> /usr/lib/system/libcorecrypto.dylib
    0x7ff800288000 -     0x7ff8002b3fff  libsystem_malloc.dylib (374.40.6) <896704B7-96B0-32B4-986D-A6BCD5C816B4> /usr/lib/system/libsystem_malloc.dylib
    0x7ff8002b4000 -     0x7ff8002faff3  libdispatch.dylib (1324.41.2) <BE53A13C-8CE1-3E40-B9BC-98473D3EED3E> /usr/lib/system/libdispatch.dylib
    0x7ff8002fb000 -     0x7ff800334ff2  libobjc.A.dylib (838) <7CFCDC89-2BEE-3784-B7AF-C676F60FE211> /usr/lib/libobjc.A.dylib
    0x7ff800335000 -     0x7ff800337ff7  libsystem_featureflags.dylib (56) <E8DF746B-D811-3752-853A-27CB4F1202B3> /usr/lib/system/libsystem_featureflags.dylib
    0x7ff800338000 -     0x7ff8003c0fff  libsystem_c.dylib (1506.40.4) <991F58B7-B4C0-3C3A-84A8-C9C571DE5A27> /usr/lib/system/libsystem_c.dylib
    0x7ff8003c1000 -     0x7ff800419ff7  libc++.1.dylib (1200.3) <91A6D960-2646-34E2-A81E-4E34E3697A92> /usr/lib/libc++.1.dylib
    0x7ff80041a000 -     0x7ff80042fff7  libc++abi.dylib (1200.3) <A851E310-475A-3C94-B547-8EB5AE90191A> /usr/lib/libc++abi.dylib
    0x7ff800430000 -     0x7ff800466fff  libsystem_kernel.dylib (8019.41.5) <12BD6F13-C452-35EE-9069-51BEFEF29F1A> /usr/lib/system/libsystem_kernel.dylib
    0x7ff800467000 -     0x7ff800472ff7  libsystem_pthread.dylib (485.40.4) <29A2750E-F31B-3630-8761-242A6BC3E99E> /usr/lib/system/libsystem_pthread.dylib
    0x7ff800473000 -     0x7ff80047eff7  libdyld.dylib (940) <9CFEF356-4205-3D24-A961-45F5C2B581B5> /usr/lib/system/libdyld.dylib
    0x7ff80047f000 -     0x7ff800488fef  libsystem_platform.dylib (273.40.1) <E443252E-7302-3580-B8C8-9DACD0BE818D> /usr/lib/system/libsystem_platform.dylib
    0x7ff800489000 -     0x7ff8004b3fff  libsystem_info.dylib (554) <04B85DBC-7A5E-3BED-8F23-9BC24BE5528D> /usr/lib/system/libsystem_info.dylib
    0x7ff802a99000 -     0x7ff802aa2fff  libsystem_darwin.dylib (1506.40.4) <907D9E83-BA0D-31BA-A637-749CDB79C527> /usr/lib/system/libsystem_darwin.dylib
    0x7ff802ec1000 -     0x7ff802ecffff  libsystem_notify.dylib (301) <68B9582E-F269-329E-88B7-2EC598AE7A8C> /usr/lib/system/libsystem_notify.dylib
    0x7ff805368000 -     0x7ff80537dffb  libsystem_networkextension.dylib (1471.41.2) <A248F2DB-27D3-3ECE-B24C-01D8FAF4F6A7> /usr/lib/system/libsystem_networkextension.dylib
    0x7ff8053cb000 -     0x7ff8053e1ff7  libsystem_asl.dylib (390) <19AAA44A-0A38-3BF4-8B3E-3E5CB1EA3CE3> /usr/lib/system/libsystem_asl.dylib
    0x7ff80683e000 -     0x7ff806a02fef  libsqlite3.dylib (329) <77C6709C-2CB3-35A5-B637-0113D6792EE6> /usr/lib/libsqlite3.dylib
    0x7ff806c1c000 -     0x7ff806c23fff  libsystem_symptoms.dylib (1617) <879662E4-EB0A-3B24-884D-0D1822807A4F> /usr/lib/system/libsystem_symptoms.dylib
    0x7ff808d56000 -     0x7ff808d72ffb  libsystem_containermanager.dylib (382.40.7) <22962274-0321-3D75-96CA-3BD5DB4E2495> /usr/lib/system/libsystem_containermanager.dylib
    0x7ff809a94000 -     0x7ff809a97ffb  libsystem_configuration.dylib (1163.40.8) <62783DC9-4ED4-3B96-A208-0998051EA725> /usr/lib/system/libsystem_configuration.dylib
    0x7ff809a98000 -     0x7ff809a9cffb  libsystem_sandbox.dylib (1656.40.33) <05681E28-AAE5-3FDA-8ED4-4AEFF859DD72> /usr/lib/system/libsystem_sandbox.dylib
    0x7ff80a8ad000 -     0x7ff80a8afff7  libquarantine.dylib (133) <84113EA9-6172-3953-9C6D-F3062A330F53> /usr/lib/system/libquarantine.dylib
    0x7ff80af55000 -     0x7ff80af59fff  libsystem_coreservices.dylib (133) <FCB3B98E-929B-3E71-9FCB-BB7C3DA4F464> /usr/lib/system/libsystem_coreservices.dylib
    0x7ff80b1a1000 -     0x7ff80b1b3ffb  libz.1.dylib (77) <3C798945-234F-37CD-A07A-5E123899B7FB> /usr/lib/libz.1.dylib
    0x7ff80b1b4000 -     0x7ff80b214fff  libsystem_m.dylib (3204) <346B4244-B344-32C7-B975-000E86FF648A> /usr/lib/system/libsystem_m.dylib
    0x7ff80b216000 -     0x7ff80b21bfff  libmacho.dylib (987) <C2ACCFF7-E9C1-3BC7-986C-A5279F8B9464> /usr/lib/system/libmacho.dylib
    0x7ff80b238000 -     0x7ff80b243fff  libcommonCrypto.dylib (60190) <E2E69DDD-7F00-375C-90AC-3DC7D2BE6418> /usr/lib/system/libcommonCrypto.dylib
    0x7ff80b244000 -     0x7ff80b24eff7  libunwind.dylib (201) <DF7F7CF8-44EF-3FBB-B105-B42AAD811AFB> /usr/lib/system/libunwind.dylib
    0x7ff80b24f000 -     0x7ff80b256fff  liboah.dylib (254.5.3) <E41F3963-25ED-33B9-A850-AF31BAFACD1C> /usr/lib/liboah.dylib
    0x7ff80b257000 -     0x7ff80b260ff7  libcopyfile.dylib (179.40.2) <E7487E27-2927-3A6D-8AC5-BAF619582180> /usr/lib/system/libcopyfile.dylib
    0x7ff80b261000 -     0x7ff80b268fff  libcompiler_rt.dylib (102.2) <2DA02427-9E4D-348A-95AB-48B70942D6FE> /usr/lib/system/libcompiler_rt.dylib
    0x7ff80b269000 -     0x7ff80b26dff7  libsystem_collections.dylib (1506.40.4) <7C664A30-3982-3902-AFDA-EB0381A8EB9D> /usr/lib/system/libsystem_collections.dylib
    0x7ff80b26e000 -     0x7ff80b270ff7  libsystem_secinit.dylib (106.40.3) <7616D11B-CFB2-3BC0-9AA9-89C50D24B8A9> /usr/lib/system/libsystem_secinit.dylib
    0x7ff80b271000 -     0x7ff80b272fff  libremovefile.dylib (60) <130FA7F6-46C2-3582-9FAA-A0B01307C12E> /usr/lib/system/libremovefile.dylib
    0x7ff80b273000 -     0x7ff80b273ffb  libkeymgr.dylib (31) <1B400A14-A175-3109-9F3A-81E1018A38AE> /usr/lib/system/libkeymgr.dylib
    0x7ff80b274000 -     0x7ff80b27bff3  libsystem_dnssd.dylib (1556.40.29) <E7108DF9-08B2-3275-988A-EAF5C968C50D> /usr/lib/system/libsystem_dnssd.dylib
    0x7ff80b27c000 -     0x7ff80b280ff3  libcache.dylib (85) <57A4C696-A65F-3C83-BF2F-BEA125B7695B> /usr/lib/system/libcache.dylib
    0x7ff80b281000 -     0x7ff80b282ff3  libSystem.B.dylib (1311) <234AB609-FE3D-38A2-9D1A-DB639349EB94> /usr/lib/libSystem.B.dylib
    0x7ff80beb2000 -     0x7ff80becbff7  libexpat.1.dylib (28) <0E956C6D-2A58-3650-96B6-B450E6EC5D80> /usr/lib/libexpat.1.dylib
    0x7ff8113b2000 -     0x7ff8113b2fff  libsystem_product_info_filter.dylib (10) <966D59F0-C745-307B-86D2-54AF02A9E6B9> /usr/lib/system/libsystem_product_info_filter.dylib
    0x7ff9098d6000 -     0x7ff909905ff7  libncurses.5.4.dylib (57) <00077BF2-57A6-32BB-B0CD-8A300F0E2EAA> /usr/lib/libncurses.5.4.dylib
Sample analysis of process 60175 written to file /dev/stdout
simsong commented 2 years ago

Also look at this scoreboard display:

bulk_extractor      Wed Dec  8 14:25:10 2021

available_memory: 54595584
bytes_queued: 134217728
depth0_bytes_queued: 134217728
depth0_sbufs_queued: 4
elapsed_time:  0:07:11
estimated_date_completion: 2021-12-08 14:27:44
estimated_time_remaining:  0:02:34
fraction_read: 73.600000 %
max_offset: 285212672
sbufs_created: 295125
sbufs_queued: 4
sbufs_remaining: 29
tasks_queued: 234
thread-1: 167772160: accts (33554432 bytes)
thread-2: 234881024 release_sbuf
thread-3: 234881024 release_sbuf
thread-4: 167772160: email (33554432 bytes)
thread-5: 234881024 release_sbuf
thread-6: 234881024: httplogs (33554432 bytes)
thread_count: 6
===========================================================================================================>......................................|
simsong commented 2 years ago

And a little later:

thread-1: 234881024 release_sbuf
thread-2: 234881024 release_sbuf
thread-3: 234881024 release_sbuf
thread-4: 234881024 release_sbuf
thread-5: 234881024 release_sbuf
thread-6: 234881024 release_sbuf
thread_count: 6
simsong commented 2 years ago

Fixed. The problem was that the caching logic for ngram size was reversed.