search

cmu-db / peloton

The Self-Driving Database Management System
http://pelotondb.io
Apache License 2.0
2.03k stars 624 forks source link

psql -c to create and drop database segfaults on benchmarking machine #1431

Closed lmwnshn closed 6 years ago

lmwnshn commented 6 years ago

Issue Latest version of Peloton segfaults on both DEBUG and RELEASE builds on a Ubuntu 16.04.4 LTS box, when you use -c or -f to CREATE and DROP database.

Reproducibility Unable to reproduce on other lab machines, including both Linux and Mac.

Setup:

cmake -DCMAKE_BUILD_TYPE=Release ..
bash packages.sh
make -j32
bash create_certificates.sh

Running:

psql -h localhost -p 15721 -U postgres default_database -c 'CREATE DATABASE tpcc;'
psql -h localhost -p 15721 -U postgres default_database -c 'DROP DATABASE tpcc;'

Backtrace (-c)

(gdb) run
Starting program: /home/wanshenl/peloton/build/bin/peloton
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffedfff700 (LWP 2340)]
2018-06-26 12:28:26 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread main-pool-worker-0 starting ...
[New Thread 0x7fffed7fe700 (LWP 2341)]
2018-06-26 12:28:26 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread main-pool-worker-1 starting ...
[New Thread 0x7fffecffd700 (LWP 2342)]
2018-06-26 12:28:26 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread main-pool-worker-2 starting ...
[New Thread 0x7fffe7fff700 (LWP 2343)]
2018-06-26 12:28:26 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread main-pool-worker-3 starting ...
[New Thread 0x7fffe77fe700 (LWP 2344)]
2018-06-26 12:28:26 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread executor-pool-worker-0 starting ...
[New Thread 0x7fffe6ffd700 (LWP 2345)]
2018-06-26 12:28:26 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread executor-pool-worker-1 starting ...
[New Thread 0x7fffe67fc700 (LWP 2346)]
2018-06-26 12:28:26 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread executor-pool-worker-2 starting ...
[New Thread 0x7fffe5ffb700 (LWP 2347)]
2018-06-26 12:28:26 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread executor-pool-worker-3 starting ...
[New Thread 0x7fffe57fa700 (LWP 2348)]
[New Thread 0x7fffe4ff9700 (LWP 2349)]
2018-06-26 12:28:26 [src/catalog/system_catalogs.cpp:89:Bootstrap] DEBUG - Bootstrapping database: peloton
2018-06-26 12:28:26 [src/catalog/system_catalogs.cpp:89:Bootstrap] DEBUG - Bootstrapping database: default_database
2018-06-26 12:28:26 [src/network/peloton_server.cpp:124:SSLInit] WARN  - Exception when loading root_crt!
2018-06-26 12:28:26 [src/network/peloton_server.cpp:133:SSLInit] INFO  - certificate file path /home/wanshenl/peloton/data/peloton_insecure_server.crt
2018-06-26 12:28:26 [src/network/peloton_server.cpp:143:SSLInit] INFO  - private key file path /home/wanshenl/peloton/data/peloton_insecure_server.key
[New Thread 0x7ff4f16d9700 (LWP 2350)]
[New Thread 0x7ff4f0ed8700 (LWP 2351)]
[New Thread 0x7ff4f06d7700 (LWP 2352)]
[New Thread 0x7ff4efed6700 (LWP 2353)]
[New Thread 0x7ff4ef6d5700 (LWP 2354)]
[New Thread 0x7ff4eeed4700 (LWP 2355)]
[New Thread 0x7ff4ee6d3700 (LWP 2356)]
[New Thread 0x7ff4eded2700 (LWP 2357)]
[New Thread 0x7ff4ed6d1700 (LWP 2358)]
[New Thread 0x7ff4eced0700 (LWP 2359)]
[New Thread 0x7ff4ec6cf700 (LWP 2360)]
[New Thread 0x7ff4ebece700 (LWP 2361)]
[New Thread 0x7ff4eb6cd700 (LWP 2362)]
[New Thread 0x7ff4eaecc700 (LWP 2363)]
[New Thread 0x7ff4ea6cb700 (LWP 2364)]
[New Thread 0x7ff4e9eca700 (LWP 2365)]
[New Thread 0x7ff4e96c9700 (LWP 2366)]
[New Thread 0x7ff4e8ec8700 (LWP 2367)]
[New Thread 0x7ff4e86c7700 (LWP 2368)]
[New Thread 0x7ff4e7ec6700 (LWP 2369)]
[New Thread 0x7ff4e76c5700 (LWP 2370)]
[New Thread 0x7ff4e6ec4700 (LWP 2371)]
[New Thread 0x7ff4e66c3700 (LWP 2372)]
[New Thread 0x7ff4e5ec2700 (LWP 2373)]
[New Thread 0x7ff4e56c1700 (LWP 2374)]
[New Thread 0x7ff4e4ec0700 (LWP 2375)]
[New Thread 0x7ff4e46bf700 (LWP 2376)]
[New Thread 0x7ff4e3ebe700 (LWP 2377)]
[New Thread 0x7ff4e36bd700 (LWP 2378)]
[New Thread 0x7ff4e2ebc700 (LWP 2379)]
[New Thread 0x7ff4e26bb700 (LWP 2380)]
[New Thread 0x7ff4e1eba700 (LWP 2381)]
2018-06-26 12:28:27 [src/network/peloton_server.cpp:267:SetupServer] INFO  - Listening on port 15721
2018-06-26 12:28:28 [src/network/connection_dispatcher_task.cpp:63:DispatchConnection] DEBUG - Dispatching connection to worker 0
2018-06-26 12:28:28 [src/network/postgres_protocol_handler.cpp:992:ProcessInitialPacket] INFO  - protocol version: 80877103
2018-06-26 12:28:28 [src/network/postgres_protocol_handler.cpp:992:ProcessInitialPacket] INFO  - protocol version: 196608
2018-06-26 12:28:28 [src/catalog/system_catalogs.cpp:89:Bootstrap] DEBUG - Bootstrapping database: tpcc
2018-06-26 12:28:28 [src/network/connection_handle.cpp:235:TryCloseConnection] DEBUG - Attempt to close the connection 200
2018-06-26 12:28:28 [src/network/network_io_wrappers.cpp:186:Close] ERROR - Error shutting down ssl session, err: 0
2018-06-26 12:28:33 [src/network/connection_dispatcher_task.cpp:63:DispatchConnection] DEBUG - Dispatching connection to worker 1

Thread 13 "peloton" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff4f0ed8700 (LWP 2351)]
0x00007ffff6686e04 in peloton::network::Buffer::Reset (this=0x0)
    at /home/wanshenl/peloton/src/include/network/marshal.h:48
48      size_ = 0;
(gdb) bt
#0  0x00007ffff6686e04 in peloton::network::Buffer::Reset (this=0x0)
    at /home/wanshenl/peloton/src/include/network/marshal.h:48
#1  0x00007ffff6687235 in peloton::network::NetworkIoWrapper::NetworkIoWrapper
    (this=0x7ff04c000a80, sock_fd=200, rbuf=std::shared_ptr (empty) 0x0,
    wbuf=std::shared_ptr (empty) 0x0)
    at /home/wanshenl/peloton/src/include/network/network_io_wrappers.h:56
#2  0x00007ffff6685dea in peloton::network::PosixSocketIoWrapper::PosixSocketIoWrapper (this=0x7ff04c000a80, sock_fd=200,
    rbuf=<error reading variable: Cannot access memory at address 0xc8dc004b48>, wbuf=std::shared_ptr (expired, weak 0) 0x7ff4f0ed7630)
    at /home/wanshenl/peloton/src/network/network_io_wrappers.cpp:61
#3  0x00007ffff6684f6d in __gnu_cxx::new_allocator<peloton::network::PosixSocketIoWrapper>::construct<peloton::network::PosixSocketIoWrapper, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff4f0ed7797, __p=0x7ff04c000a80)
    at /usr/include/c++/5/ext/new_allocator.h:120
#4  0x00007ffff66847b8 in std::allocator_traits<std::allocator<peloton::network::PosixSocketIoWrapper> >::construct<peloton::network::PosixSocketIoWrapper, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (__a=..., __p=0x7ff04c000a80)
    at /usr/include/c++/5/bits/alloc_traits.h:530
#5  0x00007ffff6683e7f in std::_Sp_counted_ptr_inplace<peloton::network::PosixSocketIoWrapper, std::allocator<peloton::network::PosixSocketIoWrapper>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<int&, std::shared_ptr<peloton::netwo---Type <return> to continue, or q <return> to quit---
rk::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (
    this=0x7ff04c000a70, __a=...)
    at /usr/include/c++/5/bits/shared_ptr_base.h:522
#6  0x00007ffff668326b in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<peloton::network::PosixSocketIoWrapper, std::allocator<peloton::network::PosixSocketIoWrapper>, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff4f0ed7a28, __a=...)
    at /usr/include/c++/5/bits/shared_ptr_base.h:617
#7  0x00007ffff6682733 in std::__shared_ptr<peloton::network::PosixSocketIoWrapper, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<peloton::network::PosixSocketIoWrapper>, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff4f0ed7a20, __tag=...,
    __a=...) at /usr/include/c++/5/bits/shared_ptr_base.h:1097
#8  0x00007ffff6681ccc in std::shared_ptr<peloton::network::PosixSocketIoWrapper>::shared_ptr<std::allocator<peloton::network::PosixSocketIoWrapper>, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff4f0ed7a20, __tag=..., __a=...)
    at /usr/include/c++/5/bits/shared_ptr.h:319
#9  0x00007ffff66813cb in std::allocate_shared<peloton::network::PosixSocketIoWrapper, std::allocator<peloton::network::PosixSocketIoWrapper>, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (__a=...) at /usr/include/c++/5/bits/shared_ptr.h:620
#10 0x00007ffff6680969 in std::make_shared<peloton::network::PosixSocketIoWrapper, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton---Type <return> to continue, or q <return> to quit---
::network::WriteBuffer>&> () at /usr/include/c++/5/bits/shared_ptr.h:636
#11 0x00007ffff667f73a in peloton::network::NetworkIoWrapperFactory::NewNetworkIoWrapper (
    this=0x7ffff7dc5d00 <peloton::network::NetworkIoWrapperFactory::GetInstance()::factory>, conn_fd=200)
    at /home/wanshenl/peloton/src/network/network_io_wrapper_factory.cpp:37
#12 0x00007ffff66afec5 in peloton::network::ConnectionHandle::ConnectionHandle
    (this=0x7ff04c0008c0, sock_fd=200, handler=0x1346110)
    at /home/wanshenl/peloton/src/network/connection_handle.cpp:172
#13 0x00007ffff667ec50 in peloton::network::ConnectionHandlerTask::HandleDispatch (this=0x1346110, new_conn_recv_fd=18)
    at /home/wanshenl/peloton/src/network/connection_handler_task.cpp:61
#14 0x00007ffff667e906 in peloton::network::ConnectionHandlerTask::ConnectionHandlerTask(int)::{lambda(int, short, void*)#1}::operator()(int, short, void*) const () at /home/wanshenl/peloton/src/network/connection_handler_task.cpp:29
#15 0x00007ffff667e936 in peloton::network::ConnectionHandlerTask::ConnectionHandlerTask(int)::{lambda(int, short, void*)#1}::_FUN(int, short, void*) ()
    at /home/wanshenl/peloton/src/network/connection_handler_task.cpp:29
#16 0x00007ffff4b5d4c9 in event_base_loop ()
   from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#17 0x0000000000421d9b in peloton::EventUtil::EventBaseDispatch (
    base=0x13461b0)
    at /home/wanshenl/peloton/src/include/common/event_util.h:81
#18 0x0000000000421e1a in peloton::NotifiableTask::EventLoop (this=0x1346110)
---Type <return> to continue, or q <return> to quit---
    at /home/wanshenl/peloton/src/include/common/notifiable_task.h:172
#19 0x00007ffff6687a01 in peloton::network::ConnectionDispatcherTask::ConnectionDispatcherTask(int, int)::{lambda()#3}::operator()() const ()
    at /home/wanshenl/peloton/src/network/connection_dispatcher_task.cpp:43
#20 0x00007ffff66892d0 in std::_Bind_simple<peloton::network::ConnectionDispatcherTask::ConnectionDispatcherTask(int, int)::<lambda()>()>::_M_invoke<>(std::_Index_tuple<>) (this=0x1346988) at /usr/include/c++/5/functional:1531
#21 0x00007ffff6689226 in std::_Bind_simple<peloton::network::ConnectionDispatcherTask::ConnectionDispatcherTask(int, int)::<lambda()>()>::operator()(void) (
    this=0x1346988) at /usr/include/c++/5/functional:1520
#22 0x00007ffff66891b6 in std::thread::_Impl<std::_Bind_simple<peloton::network::ConnectionDispatcherTask::ConnectionDispatcherTask(int, int)::<lambda()>()> >::_M_run(void) (this=0x1346970) at /usr/include/c++/5/thread:115
#23 0x00007ffff4683c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#24 0x00007ffff39976ba in start_thread (arg=0x7ff4f0ed8700)
    at pthread_create.c:333
#25 0x00007ffff40f241d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Backtrace (-f)

(gdb) run
Starting program: /home/wanshenl/peloton/build/bin/peloton
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffedfff700 (LWP 2734)]
2018-06-26 12:33:05 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread main-pool-worker-0 starting ...
[New Thread 0x7fffed7fe700 (LWP 2735)]
2018-06-26 12:33:05 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread main-pool-worker-1 starting ...
[New Thread 0x7fffecffd700 (LWP 2736)]
2018-06-26 12:33:05 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread main-pool-worker-2 starting ...
[New Thread 0x7fffdffff700 (LWP 2737)]
2018-06-26 12:33:05 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread main-pool-worker-3 starting ...
[New Thread 0x7fffdf7fe700 (LWP 2738)]
2018-06-26 12:33:05 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread executor-pool-worker-0 starting ...
[New Thread 0x7fffdeffd700 (LWP 2739)]
2018-06-26 12:33:05 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread executor-pool-worker-1 starting ...
[New Thread 0x7fffde7fc700 (LWP 2740)]
2018-06-26 12:33:05 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread executor-pool-worker-2 starting ...
[New Thread 0x7fffddffb700 (LWP 2741)]
2018-06-26 12:33:05 [src/threadpool/worker_pool.cpp:27:WorkerFunc] INFO  - Thread executor-pool-worker-3 starting ...
[New Thread 0x7fffdd7fa700 (LWP 2742)]
[New Thread 0x7fffdcff9700 (LWP 2743)]
2018-06-26 12:33:05 [src/catalog/system_catalogs.cpp:89:Bootstrap] DEBUG - Bootstrapping database: peloton
2018-06-26 12:33:05 [src/catalog/system_catalogs.cpp:89:Bootstrap] DEBUG - Bootstrapping database: default_database
2018-06-26 12:33:05 [src/network/peloton_server.cpp:124:SSLInit] WARN  - Exception when loading root_crt!
2018-06-26 12:33:05 [src/network/peloton_server.cpp:133:SSLInit] INFO  - certificate file path /home/wanshenl/peloton/data/peloton_insecure_server.crt
2018-06-26 12:33:05 [src/network/peloton_server.cpp:143:SSLInit] INFO  - private key file path /home/wanshenl/peloton/data/peloton_insecure_server.key
[New Thread 0x7ff4f1ad1700 (LWP 2744)]
[New Thread 0x7ff4f12d0700 (LWP 2745)]
[New Thread 0x7ff4f0acf700 (LWP 2746)]
[New Thread 0x7ff4f02ce700 (LWP 2747)]
[New Thread 0x7ff4efacd700 (LWP 2748)]
[New Thread 0x7ff4ef2cc700 (LWP 2749)]
[New Thread 0x7ff4eeacb700 (LWP 2750)]
[New Thread 0x7ff4ee2ca700 (LWP 2751)]
[New Thread 0x7ff4edac9700 (LWP 2752)]
[New Thread 0x7ff4ed2c8700 (LWP 2753)]
[New Thread 0x7ff4ecac7700 (LWP 2754)]
[New Thread 0x7ff4ec2c6700 (LWP 2755)]
[New Thread 0x7ff4ebac5700 (LWP 2756)]
[New Thread 0x7ff4eb2c4700 (LWP 2757)]
[New Thread 0x7ff4eaac3700 (LWP 2758)]
[New Thread 0x7ff4ea2c2700 (LWP 2759)]
[New Thread 0x7ff4e9ac1700 (LWP 2760)]
[New Thread 0x7ff4e92c0700 (LWP 2761)]
[New Thread 0x7ff4e8abf700 (LWP 2762)]
[New Thread 0x7ff4e82be700 (LWP 2763)]
[New Thread 0x7ff4e7abd700 (LWP 2765)]
[New Thread 0x7ff4e72bc700 (LWP 2766)]
[New Thread 0x7ff4e6abb700 (LWP 2767)]
[New Thread 0x7ff4e62ba700 (LWP 2768)]
[New Thread 0x7ff4e5ab9700 (LWP 2769)]
[New Thread 0x7ff4e52b8700 (LWP 2770)]
[New Thread 0x7ff4e4ab7700 (LWP 2771)]
[New Thread 0x7ff4e42b6700 (LWP 2772)]
[New Thread 0x7ff4e3ab5700 (LWP 2773)]
[New Thread 0x7ff4e32b4700 (LWP 2774)]
[New Thread 0x7ff4e2ab3700 (LWP 2775)]
[New Thread 0x7ff4e22b2700 (LWP 2776)]
2018-06-26 12:33:05 [src/network/peloton_server.cpp:267:SetupServer] INFO  - Listening on port 15721
2018-06-26 12:33:06 [src/network/connection_dispatcher_task.cpp:63:DispatchConnection] DEBUG - Dispatching connection to worker 0
2018-06-26 12:33:06 [src/network/postgres_protocol_handler.cpp:992:ProcessInitialPacket] INFO  - protocol version: 80877103
2018-06-26 12:33:06 [src/network/postgres_protocol_handler.cpp:992:ProcessInitialPacket] INFO  - protocol version: 196608
2018-06-26 12:33:07 [src/catalog/system_catalogs.cpp:89:Bootstrap] DEBUG - Bootstrapping database: tpcc
2018-06-26 12:33:07 [src/network/connection_handle.cpp:235:TryCloseConnection] DEBUG - Attempt to close the connection 200
2018-06-26 12:33:07 [src/network/network_io_wrappers.cpp:186:Close] ERROR - Error shutting down ssl session, err: 0
2018-06-26 12:33:10 [src/network/connection_dispatcher_task.cpp:63:DispatchConnection] DEBUG - Dispatching connection to worker 1

Thread 13 "peloton" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff4f12d0700 (LWP 2745)]
0x00007ffff6686e04 in peloton::network::Buffer::Reset (this=0x0)
    at /home/wanshenl/peloton/src/include/network/marshal.h:48
48      size_ = 0;
(gdb) bt
#0  0x00007ffff6686e04 in peloton::network::Buffer::Reset (this=0x0)
    at /home/wanshenl/peloton/src/include/network/marshal.h:48
#1  0x00007ffff6687235 in peloton::network::NetworkIoWrapper::NetworkIoWrapper
    (this=0x7ff050000a80, sock_fd=200, rbuf=std::shared_ptr (empty) 0x0,
    wbuf=std::shared_ptr (empty) 0x0)
    at /home/wanshenl/peloton/src/include/network/network_io_wrappers.h:56
#2  0x00007ffff6685dea in peloton::network::PosixSocketIoWrapper::PosixSocketIoWrapper (this=0x7ff050000a80, sock_fd=200,
    rbuf=<error reading variable: Cannot access memory at address 0xc8dc004b48>, wbuf=std::shared_ptr (expired, weak 0) 0x7ff4f12cf630)
    at /home/wanshenl/peloton/src/network/network_io_wrappers.cpp:61
#3  0x00007ffff6684f6d in __gnu_cxx::new_allocator<peloton::network::PosixSocketIoWrapper>::construct<peloton::network::PosixSocketIoWrapper, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff4f12cf797, __p=0x7ff050000a80)
    at /usr/include/c++/5/ext/new_allocator.h:120
#4  0x00007ffff66847b8 in std::allocator_traits<std::allocator<peloton::network::PosixSocketIoWrapper> >::construct<peloton::network::PosixSocketIoWrapper, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (__a=..., __p=0x7ff050000a80)
    at /usr/include/c++/5/bits/alloc_traits.h:530
#5  0x00007ffff6683e7f in std::_Sp_counted_ptr_inplace<peloton::network::PosixSocketIoWrapper, std::allocator<peloton::network::PosixSocketIoWrapper>, (__gnu_cxx::_Lock_---Type <return> to continue, or q <return> to quit---
rk::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff050000a70, __a=...) at /usr/include/c++/5/bits/shared_ptr_base.h:522
#6  0x00007ffff668326b in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<peloton::network::PosixSocketIoWrapper, std::allocator<peloton::network::PosixSocketIoWrapper>, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff4f12cfa28, __a=...)
    at /usr/include/c++/5/bits/shared_ptr_base.h:617
#7  0x00007ffff6682733 in std::__shared_ptr<peloton::network::PosixSocketIoWrapper, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<peloton::network::PosixSocketIoWrapper>, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff4f12cfa20, __tag=..., __a=...)
    at /usr/include/c++/5/bits/shared_ptr_base.h:1097
#8  0x00007ffff6681ccc in std::shared_ptr<peloton::network::PosixSocketIoWrapper>::shared_ptr<std::allocator<peloton::network::PosixSocketIoWrapper>, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (this=0x7ff4f12cfa20, __tag=..., __a=...)
    at /usr/include/c++/5/bits/shared_ptr.h:319
#9  0x00007ffff66813cb in std::allocate_shared<peloton::network::PosixSocketIoWrapper, std::allocator<peloton::network::PosixSocketIoWrapper>, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> (__a=...) at /usr/include/c++/5/bits/shared_ptr.h:620
#10 0x00007ffff6680969 in std::make_shared<peloton::network::PosixSocketIoWrapper, int&, std::shared_ptr<peloton::network::ReadBuffer>&, std::shared_ptr<peloton::network::WriteBuffer>&> () at /usr/include/c++/5/bits/shared_ptr.h:636
#11 0x00007ffff667f73a in peloton::network::NetworkIoWrapperFactory::NewNetworkIoWrapper (
    this=0x7ffff7dc5d00 <peloton::network::NetworkIoWrapperFactory::GetInstance()::factory>, conn_fd=200)
    at /home/wanshenl/peloton/src/network/network_io_wrapper_factory.cpp:37

#12 0x00007ffff66afec5 in peloton::network::ConnectionHandle::ConnectionHandle (this=0x7ff0500008c0, sock_fd=200, handler=0x1344400)
    at /home/wanshenl/peloton/src/network/connection_handle.cpp:172
#13 0x00007ffff667ec50 in peloton::network::ConnectionHandlerTask::HandleDispatch (this=0x1344400, new_conn_recv_fd=18)
    at /home/wanshenl/peloton/src/network/connection_handler_task.cpp:61
#14 0x00007ffff667e906 in peloton::network::ConnectionHandlerTask::ConnectionHandlerTask(int)::{lambda(int, short, void*)#1}::operator()(int, short, void*) const ()
    at /home/wanshenl/peloton/src/network/connection_handler_task.cpp:29
#15 0x00007ffff667e936 in peloton::network::ConnectionHandlerTask::ConnectionHandlerTask(int)::{lambda(int, short, void*)#1}::_FUN(int, short, void*) ()
    at /home/wanshenl/peloton/src/network/connection_handler_task.cpp:29
#16 0x00007ffff4b5d4c9 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#17 0x0000000000421d9b in peloton::EventUtil::EventBaseDispatch (base=0x13444a0) at /home/wanshenl/peloton/src/include/common/event_util.h:81

#18 0x0000000000421e1a in peloton::NotifiableTask::EventLoop (this=0x1344400) at /home/wanshenl/peloton/src/include/common/notifiable_task.h:172
#19 0x00007ffff6687a01 in peloton::network::ConnectionDispatcherTask::ConnectionDispatcherTask(int, int)::{lambda()#3}::operator()() const ()
    at /home/wanshenl/peloton/src/network/connection_dispatcher_task.cpp:43
#20 0x00007ffff66892d0 in std::_Bind_simple<peloton::network::ConnectionDispatcherTask::ConnectionDispatcherTask(int, int)::<lambda()>()>::_M_invoke<>(std::_Index_tuple<>) (this=0x1344c78) at /usr/include/c++/5/functional:1531
#21 0x00007ffff6689226 in std::_Bind_simple<peloton::network::ConnectionDispatcherTask::ConnectionDispatcherTask(int, int)::<lambda()>()>::operator()(void) (
    this=0x1344c78) at /usr/include/c++/5/functional:1520
#22 0x00007ffff66891b6 in std::thread::_Impl<std::_Bind_simple<peloton::network::ConnectionDispatcherTask::ConnectionDispatcherTask(int, int)::<lambda()>()> >::_M_run(void) (this=0x1344c60) at /usr/include/c++/5/thread:115
#23 0x00007ffff4683c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#24 0x00007ffff39976ba in start_thread (arg=0x7ff4f12d0700) at pthread_create.c:333
#25 0x00007ffff40f241d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109