Segmentation Fault (core dump) when running on Linux

MarkCRay commented 3 months ago

Documentation said it was developed on MacOS, but I thought I would try on Linux. I'm certainly not an expert on C++ programming.

OS: Red Hat Enterprise Linux release 8.2 Kernel Version: 4.18.0-193.14.3.el8_2.x86_64 Processor model: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)

I compiled the program as follows:

$ g++ -g -std=c++17 -pthread *.cpp -o main.exe

The program starts up and runs, but core dumps after ~3 seconds. The amount of time is variable, but its always a multiple of 3 seconds. I think this is because the Trader thread wakes up every 3 seconds. Below is a typical run...

$ time ./main.exe
Setting up simulation!
Initializing Traders!
Initializing Stocks!
Starting Simulation!
Market Event: Nothing!
Market Fluctuated!
Market Event: Interest Rate Change!
Warning: Will not simulate negative interest rate. Adjusting interest rate to 0.01
Market Fluctuated!
Stock Bought!
Stock Bought!
Market Event: Nothing!
Stock Bought!
Stock Bought!
Stock Bought!
Stock Bought!
Stock Bought!
Market Fluctuated!
Stock Bought!
Buy Order for Trader 12 Placed!
Stock Bought!
Stock Bought!
Buy Order for Trader 13 Placed!
Stock Bought!
Stock Bought!
Stock Bought!
Stock Bought!
Stock Bought!
Buy Order for Trader 19 Placed!
Buy Order for Trader 20 Placed!
Stock Bought!
Stock Bought!
Stock Bought!
Stock Bought!
Stock Bought!
Segmentation fault (core dumped)

real 0m3.047s user 0m0.012s sys 0m0.047s

GDB on the core dump shows

$ gdb main.exe core.52350
...
warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./main.exe'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000041a628 in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=<error reading variable: Cannot access memory at address 0x7fd582ffefa8>,
    __urng=<error reading variable: Cannot access memory at address 0x7fd582ffefa0>, __param=<error reading variable: Cannot access memory at address 0x7fd582ffef98>)
    at /usr/include/c++/8/bits/uniform_int_dist.h:218

warning: Source file is more recent than executable.
218           uniform_int_distribution<_IntType>::
[Current thread is 1 (Thread 0x7fd5837fe700 (LWP 52368))]

The stack trace of the thread seems to show a recursive loop in uniform_int_distribution. Note the depth of the trace shows over 52K stack frames. This is with a stack size of 8192K. I can increase the stack size, but the coredump still occurs, there just more stack frames on the stack.

(gdb) bt
#0  0x000000000041a628 in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=<error reading variable: Cannot access memory at address 0x7fd582ffefa8>,
    __urng=<error reading variable: Cannot access memory at address 0x7fd582ffefa0>, __param=<error reading variable: Cannot access memory at address 0x7fd582ffef98>)
    at /usr/include/c++/8/bits/uniform_int_dist.h:218
#1  0x000000000041a72c in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=0x7fd5837fdc98, __urng=..., __param=...) at /usr/include/c++/8/bits/uniform_int_dist.h:268
#2  0x000000000041a72c in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=0x7fd5837fdc98, __urng=..., __param=...) at /usr/include/c++/8/bits/uniform_int_dist.h:268
#3  0x000000000041a72c in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=0x7fd5837fdc98, __urng=..., __param=...) at /usr/include/c++/8/bits/uniform_int_dist.h:268
#4  0x000000000041a72c in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=0x7fd5837fdc98, __urng=..., __param=...) at /usr/include/c++/8/bits/uniform_int_dist.h:268
...
#52396 0x000000000041a72c in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=0x7fd5837fdc98, __urng=..., __param=...) at /usr/include/c++/8/bits/uniform_int_dist.h:268
#52397 0x000000000041a72c in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=0x7fd5837fdc98, __urng=..., __param=...) at /usr/include/c++/8/bits/uniform_int_dist.h:268
#52398 0x000000000041a4b1 in std::uniform_int_distribution<int>::operator()<std::mersenne_twister_engine<unsigned long, 32ul, 624ul, 397ul, 31ul, 2567483615ul, 11ul, 4294967295ul, 7ul, 2636928640ul, 15ul, 4022730752ul, 18ul, 1812433253ul> > (this=0x7fd5837fdc98, __urng=...) at /usr/include/c++/8/bits/uniform_int_dist.h:166
#52399 0x000000000041a129 in Trader::doAction (this=0x1c7e9d0, stocks=std::vector of length 65, capacity 65 = {...}, trademtx=...) at Trader.cpp:119
#52400 0x0000000000404d63 in Market::<lambda()>::operator()(void) const (__closure=0x1c5b8c8) at Market.cpp:238
#52401 0x00000000004060f9 in std::__invoke_impl<void, Market::run()::<lambda()> >(std::__invoke_other, Market::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#52402 0x0000000000405ace in std::__invoke<Market::run()::<lambda()> >(Market::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#52403 0x0000000000406838 in std::thread::_Invoker<std::tuple<Market::run()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x1c5b8c8) at /usr/include/c++/8/thread:244
#52404 0x00000000004067c0 in std::thread::_Invoker<std::tuple<Market::run()::<lambda()> > >::operator()(void) (this=0x1c5b8c8) at /usr/include/c++/8/thread:253
#52405 0x0000000000406744 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<Market::run()::<lambda()> > > >::_M_run(void) (this=0x1c5b8c0) at /usr/include/c++/8/thread:196
#52406 0x00007fd5c20b6ba3 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#52407 0x00007fd5c18422de in start_thread () from /lib64/libpthread.so.0
#52408 0x00007fd5c1573e83 in clone () from /lib64/libc.so.6

alexnicholsamu commented 3 months ago

Thanks for this. I'll spend some time this weekend looking into it and get back to you! How does changing the sleep time / int distribution range affect this (if at all?)

MarkCRay commented 3 months ago

In Trader.cpp for Trader::doAction(), if I change the SleepDuration from 3 to 20 seconds, then it coredumps in 20 secords or a multiple of 20 seconds, or if I change it to 2, then it coredumps in 2 seconds or some multiple of 2 seconds.

From the stack trace, it would see to originate from the Order Cancellation code:

else{
    std::uniform_int_distribution<int> orderCancelDistribution(0, active_orders.size()-1);
    choice = orderCancelDistribution(generator);
    if(!(active_orders.size() == 0)){
        cancelOrder(active_orders[choice], trademtx);
    }
}

}

But even if I comment this out, it still coredumps somewhere else where uniform_int_distribution is used.

alexnicholsamu / HFT-market-sim

Segmentation Fault (core dump) when running on Linux #1