selectors: Add urgent data to read event

[b5372eda-6675-4503-a2f8-d774d4d4f105]

BPO	30844
Nosy	@pitrou, @vstinner, @1st1, @pklanke
PRs	python/cpython#2562

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature', '3.7'] title = 'selectors: Add urgent data to read event' updated_at = user = 'https://github.com/pklanke' ``` bugs.python.org fields: ```python activity = actor = 'pklanke' assignee = 'none' closed = False closed_date = None closer = None components = [] creation = creator = 'pklanke' dependencies = [] files = [] hgrepos = [] issue_num = 30844 keywords = [] message_count = 22.0 messages = ['297629', '297668', '297669', '297670', '297677', '297707', '297720', '297721', '297738', '302739', '303026', '303029', '303031', '303032', '303033', '303034', '303035', '303102', '303109', '303111', '305434', '305478'] nosy_count = 5.0 nosy_names = ['pitrou', 'vstinner', 'neologix', 'yselivanov', 'pklanke'] pr_nums = ['2562'] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue30844' versions = ['Python 3.7'] ```

[b5372eda-6675-4503-a2f8-d774d4d4f105]

To be able to use GPIO Sysfs Interface on our embedded platforms we require exceptional event support.

[vstinner]

To be able to use GPIO Sysfs Interface on our embedded platforms we require exceptional event support.

Antoine Pitrou noticed that "exception" term can be confusion in Python, since exceptions are like "raise ValueError(...)".

The manual page mentions "out-of-band (OOB) data".

http://man7.org/linux/man-pages/man2/select_tut.2.html

   exceptfds
          This set is watched for "exceptional conditions".  In
          practice, only one such exceptional condition is common: the
          availability of out-of-band (OOB) data for reading from a TCP
          socket.  See recv(2), send(2), and tcp(7) for more details
          about OOB data.  (One other less common case where select(2)
          indicates an exceptional condition occurs with pseudoterminals
          in packet mode; see ioctl_tty(2).)  After select() has
          returned, exceptfds will be cleared of all file descriptors
          except for those for which an exceptional condition has
          occurred.

Is it what you need for a GPIO? GPIO is unrelated to TCP, right?

[vstinner]

http://man7.org/linux/man-pages/man7/tcp.7.html

Sockets API TCP provides limited support for out-of-band data, in the form of (a single byte of) urgent data. In Linux this means if the other end sends newer out-of-band data the older urgent data is inserted as normal data into the stream (even when SO_OOBINLINE is not set). This differs from BSD-based stacks.

   Linux uses the BSD compatible interpretation of the urgent pointer
   field by default.  This violates RFC 1122, but is required for
   interoperability with other stacks.  It can be changed via
   /proc/sys/net/ipv4/tcp_stdurg.

   It is possible to peek at out-of-band data using the recv(2) MSG_PEEK
   flag.

   Since version 2.4, Linux supports the use of MSG_TRUNC in the flags
   argument of recv(2) (and recvmsg(2)).  This flag causes the received
   bytes of data to be discarded, rather than passed back in a caller-
   supplied buffer.  Since Linux 2.4.4, MSG_TRUNC also has this effect
   when used in conjunction with MSG_OOB to receive out-of-band data.

[vstinner]

I don't know anything about OOB data, so I continue to add more pointers :-)

https://en.wikipedia.org/wiki/Out-of-band_data

"Out of band data is a logically independent transmission channel between a pair of connected stream sockets."

[pitrou]

About naming: it is true that the manpage for select() uses the wording "exceptional condition". But the Linux man page for poll() (and by extension epoll_wait()) does not use that wording, it says "There is urgent data to read". The POSIX man page for poll() says "High-priority data may be read without blocking".

I would not argue about this if this weren't for the fact that "exception" already has a different meaning in Python land. Re-using the term for something else is confusing. Since "urgent data" and "high-priority data" are already accepted elsewhere for the same concept, reusing one of them should be ok.

[vstinner]

"urgent data", "High-priority data"

How is an application supposed to handle these data? Read them before any other data?

The selectors API returns a list of (key, events) tuples. So an application has to iterate on this list twice? A first time to look for urgent data, and then iterate again to handle other events?

Pseudo-code:

ready = selector.select(timeout)
for key, events in ready:
   if events & selectors.EVENT_URGENT:
       process_urgent_event(key, events)
for key, events in ready:
   process_other_events(key, events)

I just want to make sure that I understand correctly how these events should be used.

Would it be worth it to provide an helper to group urgent event and other events? Maybe a new select_urgent() method which would return two lists?

A selector which creates the ready list already knows if an event is urgent or not, and so could directly group them in two separated lists.

"Urgent event" doens't mean that key.events would only contain EVENT_URGENT, it can contain other events. So maybe the grouping function should be something like:

---

ready_urgent = []
ready = []
for ...:
   key = ...
   events = ...
   if not key:
      continue
   if events == EVENT_URGENT:
      ready_urgent.append((key, key.events & events))
   elif events & EVENT_URGENT:
      ready_urgent.append((key, key.events & events))
      ready.append((key, key.events & events))
   else:
      ready.append((key, key.events & events))

---

I don't know if it makes sense :-) Maybe it's better to let applications handle that themself ;-)

[b5372eda-6675-4503-a2f8-d774d4d4f105]

"The selectors API returns a list of (key, events) tuples. So an application has to iterate on this list twice?"

No. "urgent data" means 'urgent' towards other events for thís key (key being the file object), not towards events for other file objects.

AFAIK the returned ready list contains a single tuple for each file object, containing all events for that file object. Most likely urgent data events should be handled before handling other events for a given file object, but IMO there is no need to handle urgent data events of all file objects, before handling other events.

[vstinner]

Pim Klanke: "(...) IMO there is no need to handle urgent data events of all file objects, before handling other events."

Hum, ok. So no need to extend the selectors API for that. I also understand that it's better to let applications decide how to prioritize these events :-) I'm totally fine if we only provide events and let the application decide how to handle them.

The question will be more important in the asyncio API, bpo-30847.

[b5372eda-6675-4503-a2f8-d774d4d4f105]

I'm confused about the wrapper method around winsock select and curious to why this is necessary. I have send an email to neologix to share some light on the subject.

[b5372eda-6675-4503-a2f8-d774d4d4f105]

In the selectors module, the winsock select method is wrapped. The second parameter (writefds) is passed to both writefds and exceptfds and the returned number of fds for write and except are summed and returned in the second element of the tuple.

Can anyone explain to me why the winsock select method is wrapped like this?

[vstinner]

I'm not confortable with the change because of following questions:

• It seems like your patch changes the SelectSelector behaviour on Windows. How is a selectors user supposed to upgrade his/her code to get the same behaviour on Python 3.6 and 3.7? Would it make sense to add a flag to SelectSelectors get the old behaviour?

• KqueueSelector doesn't support urgent event. How is a selectors user suppose to be aware of them? Use a blacklist of selectors which doesn't support urgent events? This list might evolve in the future, so maybe the selector should announce which events are supported?

• This change alone is going to change asyncio behaviour on Windows, no? Because of the SelectSelector behaviour change on Windows.

[vstinner]

Using Git history, I found the following commit which added "r, w, x = select(r, w, w, timeout)" for Windows in tulip (old name of the asyncio project, when it was developed outside CPython):

commit 84124f3d725c9931249d083e78f43fcda91c383a Author: Richard Oudkerk \shibturn@gmail.com\ Date: Fri Jan 18 13:03:09 2013 +0000

Minimal chages to make tests pass on Windows

https://github.com/python/asyncio/commit/84124f3d725c9931249d083e78f43fcda91c383a

[b5372eda-6675-4503-a2f8-d774d4d4f105]

On 26-09-17 11:51, STINNER Victor wrote:

STINNER Victor added the comment:

I'm not confortable with the change because of following questions:

• It seems like your patch changes the SelectSelector behaviour on Windows. How is a selectors user supposed to upgrade his/her code to get the same behaviour on Python 3.6 and 3.7? Would it make sense to add a flag to SelectSelectors get the old behaviour? The wrapper method around the winsock select method causes SelectSelector to behave differently on Windows, which is wrong IMHO. The behaviour should be the same. Why this wrapper exists is unclear to me and since it conflicts with the change I want to make, I asked questions about it 8 weeks ago. Now time has passed and no answers were given, I decided to remove it. Without further documentation I do not know how to modify it otherwise.

The other solution would be to add the new feature to all OS's but Windows, but again, since the wrapper method defies logic and is undocumented, I chose to remove it.

• KqueueSelector doesn't support urgent event. How is a selectors user suppose to be aware of them? Use a blacklist of selectors which doesn't support urgent events? This list might evolve in the future, so maybe the selector should announce which events are supported? KqueueSelector raises a ValueError on registering an invalid event. Accepted events defined in the base class are READ, WRITE and URGENT. Accepted events is overruled in KQueueSelector, allowing only READ and WRITE.

• This change alone is going to change asyncio behaviour on Windows, no? Because of the SelectSelector behaviour change on Windows. Reviewing the documentation of winsock select, I can think of no reason why this should change the runtime behaviour of SelectSelector, as well as asyncio.

----------

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue30844\

---

[b5372eda-6675-4503-a2f8-d774d4d4f105]

On 26-09-17 12:29, STINNER Victor wrote:

STINNER Victor added the comment:

Using Git history, I found the following commit which added "r, w, x = select(r, w, w, timeout)" for Windows in tulip (old name of the asyncio project, when it was developed outside CPython):

commit 84124f3d725c9931249d083e78f43fcda91c383a Author: Richard Oudkerk \shibturn@gmail.com\ Date: Fri Jan 18 13:03:09 2013 +0000

 Minimal chages to make tests pass on Windows

https://github.com/python/asyncio/commit/84124f3d725c9931249d083e78f43fcda91c383a Thanks!

Although this still does not teach us why the test failed on Windows and what is solved by adding the wrapper method, it does suggest that indeed the runtime behaviour will not be affected by removing the wrapper method.

The unit test might. To be honest, I did not run unit tests on Windows. Maybe by doing this, it might just tell us exactly what is accomplished with the wrapper method.

----------

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue30844\

---

[vstinner]

Anothe piece of history, the creation of the selectors module, bpo-16853, directly with the win32 "select.select(r, w, w, timeout)":

commit 243d8d85debaa319a2be0143003a9e881a0f5646 Author: Charles-François Natali \cf.natali@gmail.com\ Date: Wed Sep 4 19:02:49 2013 +0200

Issue bpo-16853: Add new selectors module.

[vstinner]

On Windows, exceptfds of select() is not only related to urgent ("out of band") data, it also notifies connect() failure:

exceptfds:

If processing a connect call (nonblocking), connection attempt failed.
OOB data is available for reading (only if SO_OOBINLINE is disabled).

https://msdn.microsoft.com/en-us/library/windows/desktop/ms740141(v=vs.85).aspx

I'm not sure that we can easily simplify the third parameter of select() as "urgent data". The exact semantics seems to not be portable at all.

poll() describes better the event types. Extract of my local Linux poll() manual:

POLLPRI: "There is urgent data to read (e.g., out-of-band data on TCP socket; pseudoterminal master in packet mode has seen state change in slave)."

POLLERR: "Error condition (only returned in revents; ignored in events)."

POLLHUP: "Hang up (only returned in revents; ignored in events). Note that when reading from a channel such as a pipe or a stream socket, this event merely indicates that the peer closed its end of the channel. Subsequent reads from the channel will return 0 (end of file) only after all outstanding data in the channel has been consumed."

etc.

[vstinner]

It would help to look how Twisted, eventlet, gevent and others handle "urgent data" and "exceptions". Check if they succeeded to formalize these events.

asyncore uses select() or poll().

asyncore.poll() uses select.select(). It adds the fd to exceptfds if the fd is in the readfds or writefds, then asyncore calls _exception():

        r = []; w = []; e = []
        for fd, obj in map.items():
            is_r = obj.readable()
            is_w = obj.writable()
            (...)
            if is_r or is_w:
                e.append(fd)
        (...)

    try:
        r, w, e = select.select(r, w, e, timeout)
    except select.error, err:
        (...)

    (...)

        for fd in e:
            obj = map.get(fd)
            if obj is None:
                continue
            _exception(obj)

asyncore.poll2() uses select.poll(). It only uses POLLPRI if the fd is readable but always checks for error condition (POLLERR) (if asyncio waits for read and/or write events):

        for fd, obj in map.items():
            flags = 0
            if obj.readable():
                flags |= select.POLLIN | select.POLLPRI
            # accepting sockets should not be writable
            if obj.writable() and not obj.accepting:
                flags |= select.POLLOUT
            if flags:
                # Only check for exceptions if object was either readable
                # or writable.
                flags |= select.POLLERR | select.POLLHUP | select.POLLNVAL
                pollster.register(fd, flags)

[b5372eda-6675-4503-a2f8-d774d4d4f105]

On 26-09-17 14:10, STINNER Victor wrote:

STINNER Victor added the comment:

On Windows, exceptfds of select() is not only related to urgent ("out of band") data, it also notifies connect() failure:

exceptfds:

 If processing a connect call (nonblocking), connection attempt failed.
 OOB data is available for reading (only if SO_OOBINLINE is disabled).

https://msdn.microsoft.com/en-us/library/windows/desktop/ms740141(v=vs.85).aspx Ok. So what is achieved with the wrapper method is to catch a connection attempt failure error on the writefds, by signaling a WRITE_EVENT on the writefd.

Application will then most likely try to write to the fd and an error will occur, telling us the connection attempt failed.

I will try to change the wrapper method so that it still contains this hack (because that is what it is IMO), but also allows other fds to be monitored for exceptional events.

I'm not sure that we can easily simplify the third parameter of select() as "urgent data". The exact semantics seems to not be portable at all. The goal of the patch is not to simplify the third parameter of select, but simply to extend the selectors module with the ability to subscribe for the POLLPRI ('urgent data to read') event, so selectors module can be used with sysfs gpio kernel driver.

To support this in SelectSelector the third parameter of select needs to be used, causing other exceptional events like POLLERR and POLLHUP to trigger select as well. For PollLikeSelector, this has always been so, because one does not need to register for POLLERR and POLLHUP (and POLLNVAL).

The goal of the selectors module is to offer a platform-independent abstraction layer on top of I/O monitoring functions in select. Our goal is to extend this with the ability to handle exceptional conditions.

Because some people get confused using the word 'exceptional' when programming in Python, I was asked to use one of the accepted synonyms. Since POLLPRI is the only exceptional condition event we explicitly need to register for, this is the description we choose. IMHO, this description makes what we try to accomplish harder to understand, than it was confusing.

poll() describes better the event types. Extract of my local Linux poll() manual:

POLLPRI: "There is urgent data to read (e.g., out-of-band data on TCP socket; pseudoterminal master in packet mode has seen state change in slave)."

POLLERR: "Error condition (only returned in revents; ignored in events)."

POLLHUP: "Hang up (only returned in revents; ignored in events). Note that when reading from a channel such as a pipe or a stream socket, this event merely indicates that the peer closed its end of the channel. Subsequent reads from the channel will return 0 (end of file) only after all outstanding data in the channel has been consumed."

etc.

----------

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue30844\

---

[b5372eda-6675-4503-a2f8-d774d4d4f105]

On 26-09-17 14:01, STINNER Victor wrote:

STINNER Victor added the comment:

Anothe piece of history, the creation of the selectors module, bpo-16853, directly with the win32 "select.select(r, w, w, timeout)":

commit 243d8d85debaa319a2be0143003a9e881a0f5646 Author: Charles-François Natali \cf.natali@gmail.com\ Date: Wed Sep 4 19:02:49 2013 +0200

 Issue bpo-16853: Add new selectors module.

---------- 8 or so weeks ago I've send an email to Charles-François Natali (a.k.a. neologix). I asked him if he could explain the wrapper method. Neologix is in the nosy list as well, but has not answered any questions up till now

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue30844\

---

[b5372eda-6675-4503-a2f8-d774d4d4f105]

On 26-09-17 14:13, STINNER Victor wrote:

STINNER Victor added the comment:

It would help to look how Twisted, eventlet, gevent and others handle "urgent data" and "exceptions". First of all, there are no exceptions, only "exceptional conditions". Second of all, There is no "urgent data" AND "exceptional conditions"; "urgent data" is one of possible exceptional conditions, but not all exceptional conditions mean there is urgent data to read.

Check if they succeeded to formalize these events. IMO, they have made it more confusing by using the word 'exception'. In this case I would have stuck with 'exceptional'. But at least it is better than having to replace it with 'urgent data'

asyncore uses select() or poll().

asyncore.poll() uses select.select(). It adds the fd to exceptfds if the fd is in the readfds or writefds, then asyncore calls _exception():

     r = []; w = []; e = []
     for fd, obj in map.items():
         is_r = obj.readable()
         is_w = obj.writable()
         (...)
         if is_r or is_w:
             e.append(fd)
     (...)

     try:
         r, w, e = select.select(r, w, e, timeout)
     except select.error, err:
         (...)

     (...)

     for fd in e:
         obj = map.get(fd)
         if obj is None:
             continue
         \_exception(obj)

asyncore.poll2() uses select.poll(). It only uses POLLPRI if the fd is readable but always checks for error condition (POLLERR) (if asyncio waits for read and/or write events): It registers for POLLERR, POLLHUP and POLLNVAL, events one does need not to subscribe for. These events are output events only and should not be added to mask.

     for fd, obj in map.items():
         flags = 0
         if obj.readable():
             flags |= select.POLLIN | select.POLLPRI
         # accepting sockets should not be writable
         if obj.writable() and not obj.accepting:
             flags |= select.POLLOUT
         if flags:
             # Only check for exceptions if object was either readable
             # or writable.
             flags |= select.POLLERR | select.POLLHUP | select.POLLNVAL
             pollster.register(fd, flags)

Adding POLLPRI to the flags if fd is readable doesn't cut it when using it with sysfs gpio kernel driver. The fd for a sysfs input is ALWAYS readable. We want to ONLY wake on POLLPRI event. ----------

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue30844\

---

[vstinner]

It seems like kqueue supports urgent data:

"EV_OOBAND: Read filter on socket may set this flag to indicate the presence of out of band data on the descriptor."

Example: https://github.com/daurnimator/cqueues/commit/52baaf1c25bc7e6f7cb4685cb05f4ed47a3f404a

Be careful, I also found:

// Older versions of Mac OS X may not define EV_OOBAND.
#if !defined(EV_OOBAND)
# define EV_OOBAND EV_FLAG1
#endif // !defined(EV_OOBAND)

[b5372eda-6675-4503-a2f8-d774d4d4f105]

On 02-11-17 16:54, STINNER Victor wrote:

STINNER Victor \victor.stinner@gmail.com\ added the comment:

It seems like kqueue supports urgent data:

"EV_OOBAND: Read filter on socket may set this flag to indicate the presence of out of band data on the descriptor."

Example: https://github.com/daurnimator/cqueues/commit/52baaf1c25bc7e6f7cb4685cb05f4ed47a3f404a Looks promising. I will have to look into this. Be careful, I also found:

// Older versions of Mac OS X may not define EV_OOBAND.

if !defined(EV_OOBAND)

define EV_OOBAND EV_FLAG1

endif // !defined(EV_OOBAND)

----------

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue30844\

---

[foresto]

Adding POLLPRI to the flags if fd is readable doesn't cut it when using it with sysfs gpio kernel driver. The fd for a sysfs input is ALWAYS readable. We want to ONLY wake on POLLPRI event.

Indeed, and the gpio driver is not alone in this respect. Here are a few more sysfs nodes that explicitly support polling, and I expect we could find plenty of others by looking for sysfs_notify in the kernel and other drivers.

Experimentation shows that these are useful only if the POLLPRI or POLLERR states can be checked separately from readable state, because these sysfs nodes are always readable.