os.writev() does not accept generators (as buffers argument)

[01e27b45-90f2-4c74-9e5e-7e7e54c3d78e]

BPO	27020
Nosy	@socketpair, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', 'library'] title = 'os.writev() does not accept generators (as buffers argument)' updated_at = user = 'https://github.com/socketpair' ``` bugs.python.org fields: ```python activity = actor = 'socketpair' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'socketpair' dependencies = [] files = [] hgrepos = [] issue_num = 27020 keywords = [] message_count = 7.0 messages = ['265536', '265542', '265578', '265579', '265667', '265668', '265669'] nosy_count = 2.0 nosy_names = ['socketpair', 'serhiy.storchaka'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue27020' versions = ['Python 3.3', 'Python 3.4', 'Python 3.5', 'Python 3.6'] ```

[01e27b45-90f2-4c74-9e5e-7e7e54c3d78e]

Unlike socket.sendmsg(), os.writev() does not support generators.

Proof:

In [4]: os.writev(1, [b'aa', b'bb', b'\n']) aabb Out[4]: 5

In [5]: os.writev(1, (i for i in [b'aa', b'bb', b'\n'])) ... TypeError: writev() arg 2 must be a sequence

[serhiy-storchaka]

This makes sense. The purpose of writev() is to write all data in single atomic operation. In any case a sequence should be built.

This looks rather as intentional behavior than as a bug to me.

[01e27b45-90f2-4c74-9e5e-7e7e54c3d78e]

Well, I'm improving asyncio performance.

Instead of using bytearray as a big buffer and appending to it, I use deque of buffers. This eliminates memory copying.

When writing is possible, I call writev()/sendmsg().

Unfortunatelly, OS sets limit on count of buffers passed to writev()/sendmsg(). So I should slice deque to cut first N members. This is not possible too, deque()[:N] does not work.

So, I decide to use itertools.islice(deque(), 0, N) for that. This work great with sendmsg(). But fail with writev().

Also, writev guarantee atimicity with ther writes in kernel, we must not limit way of obtainig buffers before passing to kernel.

If Python have emulation of writev() on OS where it is not supported, it should be thrown out, because atomicity requirements will not be accomplished. And writes to non-seekable fd from another process may be intermixed. Say, for example, files opened in append-only mode.

[01e27b45-90f2-4c74-9e5e-7e7e54c3d78e]

Pull request to asyncio:

https://github.com/python/asyncio/pull/339

[01e27b45-90f2-4c74-9e5e-7e7e54c3d78e]

How sequence is iterated in writev():

https://github.com/python/cpython/blob/e4945dbd854befd04fe2d5ad99ab52c8d13dc162/Modules/posixmodule.c#L8086

How IOV is filled in semdmsg(): https://github.com/python/cpython/blob/fa5d369fc83f695d19eb57048efa7ec07ca3c1d7/Modules/socketmodule.c#L3817

Also, IMHO these parts of code shoud be merged.

[01e27b45-90f2-4c74-9e5e-7e7e54c3d78e]

Proof for sendmsg:

In [1]: import socket In [2]: (r, w) = socket.socketpair() In [3]: w.sendmsg((i for i in [b'1', b'2'])) Out[3]: 2

[01e27b45-90f2-4c74-9e5e-7e7e54c3d78e]

Moreover, sendmsg() flattens iterable to a list -- this is not very good for performance and memory.

Why not to fill IOV one-by-one, doubling it's size (via realloc) when need ?