How to set SO_REUSEPORT

[ChristianRobl3D]

I've encountered another issue in our server project: We need to set SO_REUSEPORT. We already use endpoint:set_reuse_addr(true) with our TCP server (SOCK_STREAM), which partially helps. To solve the last issues we need to set SO_REUSEPORT.

The project uses the latest websocketpp 0.7.0 on Mac OS X.

As suggested I already uses the socket_init_handler ([http://stackoverflow.com/questions/23023317/proper-set-socket-init-handler-syntax-or-modify-source-to-turn-on-tcp-nodelay-wi/23031124#23031124] and [https://groups.google.com/forum/#!msg/websocketpp/rvBcIJ940Bc/zxpZf9AOb0IJ]). I've also set the tcp_pre_init_handler.

The modified websocketpp-provided example "echo_server.cpp" shows the issue quite nicely; I've added the following lines: ` void on_socket_init(websocketpp::connection_hdl hdl, boost::asio::ip::tcp::socket & s) { std::cerr << "on_socket_init...\n"; int reuse = 1; int ihdl = s.lowest_layer().native_handle();

if (setsockopt(ihdl, SOL_SOCKET, SO_REUSEPORT, (const char*)&reuse, sizeof(reuse)) < 0) perror(“setsockopt(SO_REUSEADDR) failed"); }

... echo_server.set_socket_init_handler(bind(&on_socket_init, ::_1, ::_2)); //echo_server.set_tcp_pre_init_handler(bind(&on_tcp_init, ::_1)); ... std::cerr << "pre listen...\n"; echo_server.listen(9002); std::cerr << "post listen...\n"; ... `

Starting a second instance of the server, it fails with [info] asio listen error: system:48 (Address already in use) Underlying Transport Error Unfortunately the handlers are called when the first client connects and therefore after endpoint.listen(), which actually creates the socket, sets SO_REUSEADDR and bind()s the socket. Trying to set SO_REUSEPORT in the handler has no effect.

The output before a client has connected: pre listen... post listen... As soon as a client connects: on_socket_init...

When adding tcp_pre_init_handler as well, I can observe it being called even after on_socket_init.

By modifying (as a test) the boost::asio code to also set SO_REUSEPORT for TCP/non-datagram oriented sockets, I can see that starting two (or more) instances of the server then is possible.

(How) is it possible to set SO_REUSEPORT before the bind() takes place?

If this is currently not possible, can you please add a callback / another handler (preferably already providing the socket like the init_socket_handler), which gets called in endpoint.listen() after m_acceptor->open(ep.protocol(),bec); and before m_acceptor->bind(ep,bec);

Many thanks in advance!

[zaphoyd]

I pushed a change to develop branch that moves the socket_init_handler hook from the pre_init phase (which is after bind/listen/accept) to the init_asio phase, which is before. Can you test if this helps?

[ChristianRobl3D]

I just tested it: The handler is now called early enough, which should do the job, but setting the option still fails. Even worse, setting the no_delay option as shown in some comments and examples now also fails.

Printing the native handle's value s.lowest_layer().native_handle() in the handler shows "-1". Thus, setting the SO_REUSEADDR in on_socket_init() fails.

For a test I also set an asio option to see the values of the error code: boost::system::error_code ec; boost::asio::ip::tcp::no_delay option(true); s.lowest_layer().set_option(option, ec); if (ec) std::cerr << "Error: " << ec.value() << " " << ec.message().c_str() << "\n";

ec result is Error: 9 Bad file descriptor

So IMHO the reason is the socket, that is not initialised at that time.

[ChristianRobl3D]

FYI I've now modified a local copy of websocket++ 0.7.0, which enables me to set SO_REUSEPORT just at the right time. I needed to modify transport/asio/endpoint.hpp: void listen(lib::asio::ip::tcp::endpoint const & ep, lib::error_code & ec): { ... m_acceptor->open(ep.protocol(),bec); if (!bec) { m_acceptor->set_option(lib::asio::socket_base::reuse_address(m_reuse_addr),bec); } // ------------
{ int reuse = 1; if (::setsockopt(m_acceptor->native_handle(), SOL_SOCKET, SO_REUSEPORT, (const char*)&reuse, sizeof(reuse)) < 0) std::cerr << "setsockopt(SO_REUSEPORT) failed!!!, handle=" << m_acceptor->native_handle() << "\n"; } // ------------ if (!bec) { m_acceptor->bind(ep,bec); } ...

Its the only location I found that: a) the socket is present b) bind(9 has not been called.

Can you change to have a signal handler called in the spot, where I added - admittedly not very nice - the setsockop() call? This way we could add SO_REUSEPORT, etc. in a much cleaner way.

Thank you in advance!

[vadz]

Please notice that the commit 6f128a549976601bba938e62a841bf2d6a6eb175 broke my code using set_socket_init_handler() in the client: I just do socket.lowest_layer().set_option(boost::asio::ip::tcp::no_delay(true)) in it (my socket is a TLS one, hence lowest_layer()) and it now fails because the socket is invalid. I simply reverted it locally for now, but hopefully a better solution can be found for 0.8.0.

[vincedupuis]

@vadz Did you manage to set the no_delay option?

[vadz]

I didn't return to this code since then, so I only "managed" to do it by reverting the commit that broke it, as written above.

[vans163]

Why is this still unfixed? I have same problem in latest master. NO_DELAY is pretty much the point of websockets in the first place, otherwise just use http polling.

MatusKysel patch fixes it for me BTW: https://github.com/SophiaTX/websocketpp/commit/c7783e2711d3b24030bf0e9a4edd68947e4c1131

Cleanest solution is probably to have a pre_init and post_init handler and also a way to determine if the passed socket is the actual listen_socket or not, since some options you wanna set on the listen_socket others on the accepted sockets.

[zaphoyd]

This regression introduced in 0.8.0 that broke setting socket options on already accepted connections has been fixed in the develop branch. For clarification, the socket_init handler should present a valid, accepted, socket fd that options like TCP_NODELAY can be set on. To set socket options on the listening socket use the tcp_pre_bind handler, which runs during the call to listen, after the acceptor has initialized the listen socket but before bind and listen have been called. Socket options like SO_REUSEPORT or IPV6_ONLY should be set here.