os.dup2(fd, fd, inheritable=False) behaves inconsistently

[9b2dd46a-d3b9-42c4-96ed-ebdd39741ebb]

BPO	32862
Nosy	@benjaminp, @eryksun, @izbyshev
PRs	python/cpython#5713

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', '3.8', '3.9', '3.10', 'extension-modules', 'library'] title = 'os.dup2(fd, fd, inheritable=False) behaves inconsistently' updated_at = user = 'https://github.com/izbyshev' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = False closed_date = None closer = None components = ['Extension Modules', 'Library (Lib)'] creation = creator = 'izbyshev' dependencies = [] files = [] hgrepos = [] issue_num = 32862 keywords = ['patch'] message_count = 3.0 messages = ['312266', '312295', '312298'] nosy_count = 3.0 nosy_names = ['benjamin.peterson', 'eryksun', 'izbyshev'] pr_nums = ['5713'] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue32862' versions = ['Python 3.8', 'Python 3.9', 'Python 3.10'] ```

Linked PRs

• gh-102148

[9b2dd46a-d3b9-42c4-96ed-ebdd39741ebb]

os.dup2(fd, fd, inheritable=False) may fail or change fd inheritability in following ways:

1) POSIX without F_DUP2FD_CLOEXEC 1.1) dup3() is available (a common case for Linux): OSError (EINVAL, dup3() doesn't allow equal descriptors) 1.2) dup3() is not available: fd made non-inheritable

2) POSIX with F_DUP2FD_CLOEXEC (FreeBSD): inheritability is not changed

3) Windows: fd made non-inheritable

In contrast, os.dup2(fd, fd, inheritable=True) never changes fd inheritability (same as before PEP-446 landed). I suggest to make os.dup2(fd, fd, inheritable=False) behave the same.

[eryksun]

In Windows the CRT file descriptor is actually still inheritable. This only makes the underlying OS handle non-inheritable. I don't think there's a way to make an existing FD non-inheritable using public CRT functions. See bpo-32865.

[9b2dd46a-d3b9-42c4-96ed-ebdd39741ebb]

@eryksun: Thank you for the note! I've commented on bpo-32865. This adds even more inconsistency to this corner case.

[izbyshev]

I've revisited this issue, and here is corrected and expanded list of how os.dup2(fd, fd, inheritable=False) behaves across platforms:

• On Windows, Linux before 2.6.27, and macOS dup2() is attempted (validating the fd), and on success the fd is then separately made non-inheritable.

• On platforms where F_DUP2FD_CLOEXEC fcntl command is used:

  • The FreeBSD man page doesn't explicitly document behavior for the case of equal fds. The implementation does set FD_CLOEXEC in this case (I've also confirmed it in a VM with FreeBSD 12.2).
  • The Solaris man page says this about F_DUP2FD_CLOEXEC: "Similar to F_DUP2FD, except that the FD_CLOEXEC flag associated with the new file descriptor is set, unless fildes is equal to arg, in which case the flags are unchanged." I have no way to test this.
  • The Illumos man page says this about F_DUP2FD_CLOEXEC: "...If filedes equals arg, the call will fail setting errno to EINVAL." (essentially as dup3() on Linux). The source code confirms this.

• On Linux 2.6.27+ dup3() is used, which always fails with EINVAL (even if the fd is invalid).

So overall we have the full range of possible behavior across platforms: inheritability is changed, not changed, or OSError is raised.

One way to fix this inconsistency would be to make os.dup2(fd, fd, False) behave as os.dup2(fd, fd): just validate the fd and don't change its inheritability. This is what I originally attempted in PR #5713. But now, given that on all platforms except Solaris os.dup2(fd, fd, False) either fails or makes the fd non-inheritable, I think this change could be viewed as a regression: we probably don't want to introduce new cases where inheritable fds can unexpectedly appear, and the caller explicitly asks for inheritable=False here.

Likewise, changing os.dup2(fd, fd, False) to always fail (like dup3()) would add a new error case on several platforms and can be viewed as a regression.

If we were designing os.dup2() from scratch, the right thing to do would probably be to make inheritable always affect the inheritability (and make it False by default). But now we can't change how os.dup2() behaves by default (even PEP-446 didn't touch it).

I'm not sure whether it's worth to separate the default (inheritable is not passed explicitly) and the explicit inheritable=True case and make the latter always make the returned fd inheritable. But I think that something needs to be done to at least fix the inheritable=False mess. So I've implemented PR #102148 to always make the fd non-inheritable in this case.

Cc: @vstinner

[vstinner]

As an user, I expect that os.dup2(fd, fd, inheritable=False) raises an exception if fd is invalid, or makes the file descriptor non-inheritable.

In Python in general, we attempt to have a consistent behavior on all platforms, adding platform-specific code for that if needed. For example, there is a lot of (platform-specific) code to have a portable behavior with Not-a-Number (NaN) in math functions.

For the special case fd2 == fd, we should avoid dup3(). Maybe just have the same portable code on all platforms for this special case.

[izbyshev]

As an user, I expect that os.dup2(fd, fd, inheritable=False) raises an exception if fd is invalid, or makes the file descriptor non-inheritable.

For the special case fd2 == fd, we should avoid dup3()

PR #102148 is in line with this.

Maybe just have the same portable code on all platforms for this special case.

I'm not aware of a portable way of validating an fd that is better than simply delegating to dup2() (like in PR #102148). The obvious way to validate an fd (fcntl) doesn't exist on Windows. Moreover, fd validation is not as easy as it might sound (see is_valid_fd), and an fd might be valid for some operations, but not for others. I don't think CPython should invent its own semantics of the fd validity for os.dup2() , risking it be different from what the actual dup2() does for the sake of providing a consistent behavior. The os module is low-level and has always been tied to the underlying libc, and users came to expect that. This situation is different from math functions.