Provide an abstraction for a select-able Event

[c7858af7-74bc-4d36-863f-de6e03572614]

BPO	40485
Nosy	@tiran, @FFY00, @paravoid
Dependencies	bpo-41001: Provide wrapper for eventfd

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature', 'library', '3.10'] title = 'Provide an abstraction for a select-able Event' updated_at = user = 'https://github.com/paravoid' ``` bugs.python.org fields: ```python activity = actor = 'paravoid' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'paravoid' dependencies = ['41001'] files = [] hgrepos = [] issue_num = 40485 keywords = [] message_count = 6.0 messages = ['367975', '380923', '390635', '390644', '390651', '390655'] nosy_count = 3.0 nosy_names = ['christian.heimes', 'FFY00', 'paravoid'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue40485' versions = ['Python 3.10'] ```

[c7858af7-74bc-4d36-863f-de6e03572614]

In certain codebases, it's useful to be able to wait for input from one or more file descriptors (e.g. a socket), while at the same time waiting for an event triggered by another thread, or perhaps multiprocessing process.

To wait for one or more file descriptors to get ready, the select module can be used. However, neither threading.Event() nor multiprocessing.Event() are select-able, i.e. they provide no fileno() method.

The standard way one can implement this on Unix is with os.pipe(), but it can be tricky (forgetting to use non-blocking I/O etc.). It is also limited to a pair of processes at a time. On Linux systems from the past decade, one can also implement this much more efficiently using the eventfd() system calls. I think similar functionality exists in other Unixes with kqueue etc.

It'd be great if stdlib provided an abstraction over this mechanism. In fact, multiprocessing.Event() itself could probably be a thin abstraction over eventfd() on Linux? Perhaps even multiprocessing.Semaphore with EFD_SEMAPHORE, although I admit I'm less familiar with how all that works.

(Select-able Queues would be even neater, but that's a story for a different issue :)

[tiran]

os.eventfd() has landed in Python 3.10.

[c7858af7-74bc-4d36-863f-de6e03572614]

Thanks so much for all the work on os.eventfd(), it's exciting to see it come to fruition.

An eventfd variant of Event (compatible with the threading & multiprocessing APIs) is now as simple as:

class Event:
    _ONE = (1).to_bytes(8, byteorder=sys.byteorder)

    def __init__(self):
        self._event_fd = os.eventfd(0, os.EFD_NONBLOCK)
        self._selector = selectors.DefaultSelector()
        self._selector.register(self._event_fd, selectors.EVENT_READ)

    def is_set(self):
        return self.wait(timeout=0)

    def set(self):
        try:
            os.write(self._event_fd, self._ONE)
        except BlockingIOError:
            pass

    def clear(self):
        try:
            os.read(self._event_fd, 8)
        except BlockingIOError:
            pass

    def wait(self, timeout=None):
        return bool(self._selector.select(timeout=timeout))

    def fileno(self):
        return self._event_fd

Given this now has a fileno() method, it is now possible to wait for the event as part of a broader selector, among other events (e.g. a file or socket becoming available to read or write).

I don't know where (or how) such a variant would fit into stdlib. It's simpler and more lightweight (less fds) than both threading's and multiprocessing's, and could be used from both threads and processes. I'd love some guidance here. (If a maintainer or anyone else reading this wants to use the above code in a PR, feel free -- no copyright claimed or expected for this trivial piece of code above)

[c7858af7-74bc-4d36-863f-de6e03572614]

I missed that there is now also an os.eventfd_{write,read}(), so in the above os.write(self._event_fd, self._ONE) can become os.eventfd_write(self._event_fd, 1)

and: os.read(self._event_fd, 8) can become: os.eventfd_read(self._event_fd)

(the _ONE declaration will then be unnecessary and can be removed)

[tiran]

Do you want to work on a feature for 3.10? Feature freeze is in less than 4 weeks.

[c7858af7-74bc-4d36-863f-de6e03572614]

Not sure if I understand the question! I'd like to see that happen, I don't particularly care if it makes to 3.10 or a later version, although of course the earlier the better :)

As an idea of a path forward, would it make sense to take the code above and push it to multiprocessing as an alternative to Event with an if hasattr(os, "eventfd") guard?