chriskohlhoff / asio

Asio C++ Library
http://think-async.com/Asio
4.96k stars 1.22k forks source link

Asio not working with asynchronous Unix Domain sockets in VMware ESXi 6.7 #579

Open Shempp opened 3 years ago

Shempp commented 3 years ago

Faced the following problem - Asio does not support asynchronous work with Unix Domain sockets (asio::local::stream_protocol::socket) in VMware ESXi 6.7 (and below i think). It all started with the fact that I received error "0x26 (Function not implemented)" when I tried to asynchronously connect to a socket (async_connect). During debugging, I found out that the problem is in the function: int result = epoll_ctl (epoll_fd_, EPOLL_CTL_ADD, descriptor, & ev); in method epoll_reactor::register_descriptor

Ok, most likely this is not supported at the kernel level of ESXi (VMkernel) (but it works for ipv4 sockets).

I turned off the epoll flag (BOOST_ASIO_DISABLE_EPOLL), tried it again and got another error: "0x19 (Inappropriate ioctl for device)". Problem function: int result = error_wrapper(::ioctl(s, FIONBIO, &arg), ec); in method

bool set_internal_non_blocking(socket_type s,
    state_type& state, bool value, asio::error_code& ec)

Asio uses ioctl instead of fcntl. I tried use fcntl manually and it seems to work (in any case, no error occurs). I understand that most likely Asio does not officially support ESXi, but suddenly some solution from the developers may appear. Thanks for attention

Shempp commented 3 years ago

Also, I should add that asynchronous work with unix sockets on Python works for ESXi 6.7. Looking at the socket implementation for python (https://hg.python.org/cpython/file/3.5/Modules/socketmodule.c), I noticed this:

static int
internal_setblocking(PySocketSockObject *s, int block)
{
#ifdef MS_WINDOWS
    u_long arg;
#endif
#if !defined(MS_WINDOWS) \
    && !((defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO)))
    int delay_flag, new_delay_flag;
#endif
#ifdef SOCK_NONBLOCK
    if (block)
        s->sock_type &= (~SOCK_NONBLOCK);
    else
        s->sock_type |= SOCK_NONBLOCK;
#endif

    Py_BEGIN_ALLOW_THREADS
#ifndef MS_WINDOWS
#if (defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO))
    block = !block;
    ioctl(s->sock_fd, FIONBIO, (unsigned int *)&block);
#else
    delay_flag = fcntl(s->sock_fd, F_GETFL, 0);
    if (block)
        new_delay_flag = delay_flag & (~O_NONBLOCK);
    else
        new_delay_flag = delay_flag | O_NONBLOCK;
    if (new_delay_flag != delay_flag)
        fcntl(s->sock_fd, F_SETFL, new_delay_flag);
#endif
#else /* MS_WINDOWS */
    arg = !block;
    ioctlsocket(s->sock_fd, FIONBIO, &arg);
#endif /* MS_WINDOWS */
    Py_END_ALLOW_THREADS

    /* Since these don't return anything */

    return 1;
}

We are interested in this block:

#if (defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO))
    block = !block;
    ioctl(s->sock_fd, FIONBIO, (unsigned int *)&block);
#else
    delay_flag = fcntl(s->sock_fd, F_GETFL, 0);
    if (block)
        new_delay_flag = delay_flag & (~O_NONBLOCK);
    else
        new_delay_flag = delay_flag | O_NONBLOCK;
    if (new_delay_flag != delay_flag)
        fcntl(s->sock_fd, F_SETFL, new_delay_flag);

Would it be correct to also consider this preprocessor directive (HAVE_SYS_IOCTL_H) along with the existing logic? Like that:

#if defined(ASIO_WINDOWS) || defined(__CYGWIN__)
  ioctl_arg_type arg = (value ? 1 : 0);
  int result = error_wrapper(::ioctlsocket(s, FIONBIO, &arg), ec);
#elif defined(__SYMBIAN32__) || !defined(HAVE_SYS_IOCTL_H)
  int result = error_wrapper(::fcntl(s, F_GETFL, 0), ec);
  if (result >= 0)
  {
    clear_last_error();
    int flag = (value ? (result | O_NONBLOCK) : (result & ~O_NONBLOCK));
    result = error_wrapper(::fcntl(s, F_SETFL, flag), ec);
  }
#else