multithreaded websocket++ server crashes

[manuel-schiller]

I modified the websocket++ echo server example to use multiple threads:

#include <iostream>
#include <boost/thread/thread.hpp>
#include <websocketpp/config/asio_no_tls.hpp>
#include <websocketpp/server.hpp>

typedef websocketpp::server<websocketpp::config::asio> server;

using websocketpp::lib::placeholders::_1;
using websocketpp::lib::placeholders::_2;
using websocketpp::lib::bind;

// pull out the type of messages sent by our config
typedef server::message_ptr message_ptr;

// Define a callback to handle incoming messages
void on_message(server* s, websocketpp::connection_hdl hdl, message_ptr msg)
{
    try {
        s->send(hdl, msg->get_payload(), msg->get_opcode());
    } catch (const websocketpp::lib::error_code& e) {
        std::cout << "Echo failed because: " << e << "(" << e.message() << ")" << std::endl;
    }
}

int main()
{
    // Create a server endpoint
    server echo_server;
    boost::asio::io_service io_service;
    try {

        // Initialize Asio
        echo_server.init_asio(&io_service);

        // Register our message handler
        echo_server.set_message_handler(bind(&on_message, &echo_server, ::_1, ::_2));

        // Listen on port 9002
        echo_server.set_reuse_addr(true);
        echo_server.listen(9002);

        // Start the server accept loop
        echo_server.start_accept();

        boost::thread_group threadpool;
        threadpool.create_thread(boost::bind(&boost::asio::io_service::run, &io_service));
        threadpool.create_thread(boost::bind(&boost::asio::io_service::run, &io_service));
        threadpool.join_all();
    } catch (websocketpp::exception const& e) {
        std::cout << e.what() << std::endl;
    } catch (...) {
        std::cout << "other exception" << std::endl;
    }
}

I connect using one client which sends many messages asynchronously. Then the server crashes with:

2016-08-31 13:05:44] [info] asio async_read_at_least error: system:125 (Operation canceled)
[2016-08-31 13:05:44] [error] handle_read_frame error: websocketpp.transport:2 (Underlying Transport Error)
terminate called after throwing an instance of 'std::bad_weak_ptr'
  what():  bad_weak_ptr

Judging from the websocket++ manual on Thread Safety, what I am doing should be threadsafe:

The Asio transport provides full thread safety for endpoint. Works with an io_service thread pool where multiple threads are calling io_service.run();

...

All core transports guarantee that the handlers for a given connection will be serialized. When the transport and concurrency policies support endpoint concurrency, anything involving a connection_hdl should be thread safe. i.e. It is safe to pass connection_hdls to other threads, store them indefinitely, and call endpoint methods that take them as parameters from any thread at any time.

The client I am using is based on NodeJS:

client.js

var port =  9002;
var times = 10000;
var WebSocket = require("ws");
var ws = new WebSocket("ws://localhost:" + port);

ws.on('open', function open() {
    for(var i = 0; i < times; ++i) {
        ws.send(i);
    }
});

start via:

$ npm install --save ws
$ node client.js

[CryptoCurrencyStuff]

I seem to have run into a similar issue when spawning many async requests from js ws client: namely a bad this pointer in get_message.

Thread 11 "provablyfair_ga" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff1f28700 (LWP 20369)] websocketpp::message_buffer::alloc::con_msg_managerwebsocketpp::message_buffer::message::get_message (this=0x0, op=op@entry=websocketpp::frame::opcode::text, size=141) at /home/ebola/Projects/ProvablyFair/libprovablyfair/../../websocketpp/websocketpp/message_buffer/alloc.hpp:68 68 return message_ptr(lib::make_shared(type::shared_from_this(),op,size));

[CryptoCurrencyStuff]

#0 0x00000000004aea9d in websocketpp::connectionwebsocketpp::config::asio::send ( op=websocketpp::frame::opcode::text, payload="{\"roll\":{\"success\":true,\"nonce\":5,\"roll\":909,\"target\":1998,\"condition_high\":true,\"amount\":1.000000,\"profit\":-1.000000,\"balance\":1000.249200}}", this=0x7fffe40307b0) at /home/ebola/Projects/ProvablyFair/libprovablyfair/../../websocketpp/websocketpp/impl/connection_impl.hpp:86

1 websocketpp::endpointwebsocketpp::connection<websocketpp::config::asio, websocketpp::config::asio>::send (ec=, op=websocketpp::frame::opcode::text,

payload="{\"roll\":{\"success\":true,\"nonce\":5,\"roll\":909,\"target\":1998,\"condition_high\":true,\"amount\":1.000000,\"profit\":-1.000000,\"balance\":1000.249200}}", 
hdl=error reading variable: access outside bounds of object referenced via synthetic pointer, this=0x7fffec000a00)
at /home/ebola/Projects/ProvablyFair/libprovablyfair/../../websocketpp/websocketpp/impl/endpoint_impl.hpp:168

2 websocketpp::endpointwebsocketpp::connection<websocketpp::config::asio, websocketpp::config::asio>::send (this=this@entry=0x7fffffffd5b0,

hdl=std::weak_ptr (count 2, weak 5007) 0x7fffe40307b0, 
payload="{\"roll\":{\"success\":true,\"nonce\":5,\"roll\":909,\"target\":1998,\"condition_high\":true,\"amount\":1.000000,\"profit\":-1.000000,\"balance\":1000.249200}}", 
op=op@entry=websocketpp::frame::opcode::text)
at /home/ebola/Projects/ProvablyFair/libprovablyfair/../../websocketpp/websocketpp/impl/endpoint_impl.hpp:176

3 0x0000000000491a91 in server_manager::send_sock_message (this=this@entry=0x7fffffffd510,

hdl=std::weak_ptr (count 2, weak 5007) 0x7fffe40307b0, 
msg="{\"roll\":{\"success\":true,\"nonce\":5,\"roll\":909,\"target\":1998,\"condition_high\":true,\"amount\":1.000000,\"profit\":-1.000000,\"balance\":1000.249200}}")
at /home/ebola/Projects/ProvablyFair/libprovablyfair/src/websock_server.cpp:152

updated to reflect the error on hdl in stack frame 1

[CryptoCurrencyStuff]

The above happens if I create a std::thread backed pool and call send from threads other then the main thread, if I handle all message processing in my on_message handler it has no issues dealing with many async writes.

[jhurliman]

I'm seeing this issue as well, is there a good workaround?

[MiroFurtado]

Likewise encountering this issue. Is send not thread safe?

[szorbasc]

I also have similar issues. When I move connection_hdl to a thread pool and try to send there sometimes it crashes doesnt seem to have thread safety.