SegFault on AddOrder

[xanderdunn]

I am consistently getting a segfault after some millions of calls to AddOrder:

Thread 6 "my_server" received signal SIGSEGV, Segmentation fault.
(gdb) backtrace
#0  0x00007fff803c9b2c in std::__1::pair<unsigned long, CppTrader::Matching::OrderNode*>::pair<unsigned long&, CppTrader::Matching::OrderNode*&, false>(unsigned long&, CppTrader::Matching::OrderNode*&) ()
#1  0x00007fff803bad6e in std::__1::pair<std::__1::__unwrap_ref_decay<unsigned long&>::type, std::__1::__$nwrap_ref_decay<CppTrader::Matching::OrderNode*&>::type> std::__1::make_pair<unsigned long&, CppTrader::M$tching::OrderNode*&>(unsigned long&, CppTrader::Matching::OrderNode*&) ()
#2  0x00007fff803b74c3 in CppTrader::Matching::MarketManager::AddLimitOrder(CppTrader::Matching::Order co$st&, bool) ()
#3  0x00007fff803b6bad in CppTrader::Matching::MarketManager::AddOrder(CppTrader::Matching::Order const&)
    ()

It looks like it is this call.

I have many threads that are receiving orders, creating CppTrader::Matching::Order objects, and placing them into a FIFO concurrent queue: orders_queue.enqueue(order). However, reads from the queue and calls to AddOrder are only ever done by a single thread:

         Order order;
          while (orders_queue.try_dequeue(order)) { // Empty the queue
              if (std::this_thread::get_id() != this_thread_id) {
                  abort();
              }
              market.AddOrder(order);
        }

Something strange I notice is that the segfault occurs at about the same point in execution on every run. I print some order book stats every 5 seconds, and it's always around the same number of orders processed before segfault:

// Run 1:
Matching Engine Stats:
    Order Queue Depth: 0
    Total Orders Count: 2027629
    Total Executed Orders Count: 3170152
    Orders Since Last Stats: 86721
    OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=237; Asks=240; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)

// Run 2:
Matching Engine Stats:
    Order Queue Depth: 0
    Total Orders Count: 2070242
    Total Executed Orders Count: 3236818
    Orders Since Last Stats: 88030
    OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=257; Asks=238; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)

// Run 3:
Matching Engine Stats:
    Order Queue Depth: 0
    Total Orders Count: 2048591
    Total Executed Orders Count: 3202352
    Orders Since Last Stats: 87628
    OrderBook(Symbol=Symbol(Id=1; Name="MYSYMBOL"); Bids=254; Asks=239; BuyStop=0; SellStop=0; TrailingBuyStop=0; TrailingSellStop=0)

All orders are either buy or sell limit orders. The price and quantity values are randomly generated ints from 1 to 1000.

Do you have any recommendations on how I might be incorrectly handling the memory management of my Orders, or other ideas on what might cause this?

[xanderdunn]

This was a mistake on my part - the environment I was running it in had a limit on heap page allocation and it was being killed when it attempted to exceed that memory limit. Simply lifting the memory limit prevents the segfault.