Reference for nonblocking epoll on regular files

[nh2]

Linux has limited support for using epoll as a mechanism for asynchronous I/O. For reads to a file opened in buffered mode (that is, without O_DIRECT), if the file is opened as O_NONBLOCK, then a read will return EAGAIN until the relevant part is in memory.

Do you have a reference for that, or source code pointer?

Most locations discussing epoll on regular files (such as [1] [2] [3]) simply summarise with "epoll will always block on regular files".

It would be great to be able to extend them with this detail.

• [1] https://stackoverflow.com/questions/8057892/epoll-on-regular-files
• [2] https://jvns.ca/blog/2017/06/03/async-io-on-linux--select--poll--and-epoll/
• [3] http://davmac.org/davpage/linux/async-io.html#epoll

Even https://groups.google.com/forum/#!topic/comp.os.linux.development.system/K-fC-G6P4EA doesn't seem to discuss this (maybe this was written before what you stated was implemented? it would be great to know).

[nh2]

I tried with this code nonblocking-file-io-test.c:

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char const *argv[])
{
  if (argc != 2)
  {
    fprintf(stderr, "Usage: %s FILENAME\n", argv[0]);
    exit(1);
  }

  const char* filename = argv[1];

  int fd = open(filename, O_NONBLOCK);
  if (fd == -1)
  {
    perror(filename);
    exit(1);
  }

  const size_t bufsize = 1000000000;
  char* buf = malloc(bufsize); // 1 GB
  if (buf == NULL)
  {
    perror("malloc");
    exit(1);
  }

  ssize_t n = read(fd, buf, bufsize);

  if (n == -1)
  {
    printf("errno is EAGAIN: %d\n", errno == EAGAIN);
  }
  else
  {
    printf("read %ld bytes\n", n);
  }

  return 0;
}

(Compiled with clang-5.0 -std=c11 -Wall -Wextra nonblocking-file-io-test.c -o nonblocking-file-io-test on my Ubuntu 16.04.)

When run on a 15 GB file on my slow spinning disk like ./nonblocking-file-io-test mylargefile after sync; echo 3 | sudo tee /proc/sys/vm/drop_caches, this always prints read 1000000000 bytes and never seems to return EAGAIN. Same results for an even slower file system like sshfs.

Am I missing something?

[littledan]

Maybe I am remembering wrong, but IIRC this depends on filesystem support. Did you try on a raw block device?

[nh2]

@littledan I have now tried it on a raw block device (just a primary partition on my disk) with O_DIRECT, and without, using this code:

#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char const *argv[])
{
  if (argc != 2)
  {
    fprintf(stderr, "Usage: %s FILENAME\n", argv[0]);
    exit(1);
  }

  const char* filename = argv[1];

  // int fd = open(filename, O_NONBLOCK | O_DIRECT);
  int fd = open(filename, O_NONBLOCK);
  if (fd == -1)
  {
    perror(filename);
    exit(1);
  }

  const size_t bufsize = 1024LU * 1024LU * 1024LU;
  char* buf = aligned_alloc(512, bufsize); // 1 GiB
  if (buf == NULL)
  {
    perror("malloc");
    exit(1);
  }

  while (1)
  {
    ssize_t n = read(fd, buf, bufsize);

    if (n == -1)
    {
      if (errno != EAGAIN)
      {
        perror(filename);
        break;
      }
      printf("errno is EAGAIN\n");
    }
    else
    {
      printf("read %ld bytes\n", n);
      break;
    }
  }

  return 0;
}

But no matter whether I gave O_DIRECT or not, errno is EAGAIN would still never happen.

Note I had to change the malloc() to aligned_alloc() because otherwise I would get EINVAL (as described in man 2 open).

So even with working on a raw block device, I couldn't manage to make an asynchronous read.

[littledan]

Sorry about the misdirection. Interested in making a PR to fix the article?

[Mart-Bogdan]

Does an body know, are there any plans to add kernel support for async files? Found out, that FreeBSD supports it :-(

[sitsofe]

@Mart-Bogdan that is a complicated and loaded question. The Linux kernel does support async I/O via libaio/KAIO (hence the reason for thi repo :-) but there is a very long list of caveats. More recently (2019) there is the Linux kernel 5.1 io_uring interface that supports file I/O asynchrony more efficiently and in more scenarios. However none of the above doing async via conventional means + epoll.

[xnervwang]

This article is the only one I saw which says epoll supports regular file (or limited support). So I investigated and read a lot of other blogs until I found this discussion...

[kindofblue]

The Linux manpages say very clearly that NONBLOCKING does not apply to regluar files, so what is the issue here?

[rhapsodyn]

After some investigation, found that: the EPERM error was thrown on epoll_ctl, not on read.

Like this