python / cpython

The Python programming language
https://www.python.org
Other
63.28k stars 30.29k forks source link

subprocess.Popen() is sometimes slower in python3 under illumos #81971

Closed a4db0af7-fd9c-456a-9b4c-3abc97097202 closed 4 years ago

a4db0af7-fd9c-456a-9b4c-3abc97097202 commented 5 years ago
BPO 37790
Nosy @pitrou, @vstinner, @MojoVampire, @nickhendo
Files
  • 1.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = None closed_at = created_at = labels = ['library'] title = 'subprocess.Popen() is sometimes slower in python3 under illumos' updated_at = user = 'https://bugs.python.org/AlexanderPyhalov' ``` bugs.python.org fields: ```python activity = actor = 'movement' assignee = 'none' closed = True closed_date = closer = 'vstinner' components = ['Library (Lib)'] creation = creator = 'Alexander.Pyhalov' dependencies = [] files = ['48549'] hgrepos = [] issue_num = 37790 keywords = [] message_count = 15.0 messages = ['349197', '349207', '349211', '349857', '349893', '349894', '349897', '350064', '351343', '351352', '367162', '367181', '367182', '367185', '369358'] nosy_count = 6.0 nosy_names = ['pitrou', 'vstinner', 'movement', 'Alexander.Pyhalov', 'josh.r', 'nickhendo'] pr_nums = [] priority = 'normal' resolution = 'out of date' stage = 'resolved' status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue37790' versions = ['Python 3.5'] ```

    a4db0af7-fd9c-456a-9b4c-3abc97097202 commented 5 years ago

    We've moved illumos-gate wsdiff python tool from Python 2 to Python 3. The tool does the following - for each file from old and new proto area compares file attributes to find differences in binary otput (spawning elfdump, dump and other utilities). Performance has degraded in two times when commands.getstatusoutput() was replaced by subprocess.getstatusoutput(). os.popen() now is Popen() wrapper, so it also has poor performance. Even naive popen() implementation using os.system() and os.mkstemp() behaved much more efficiently (about two times faster e.g. 20 minuts vs 40 minutes for single tool pass).

    99ffcaa5-b43b-4e8e-a35e-9c890007b9cd commented 5 years ago

    Have you tried switching to using Popen itself (or run, which keeps it to one layer of convenience wrapping)?

    subprocess.getstatusoutput is three layers of wrapping (it calls check_output, which in turn calls run, which in turn calls Popen), and (unlike Python 2) has to decode all the output. run would avoid two layers of wrapping, can optionally receive the raw bytes instead of decoding to str, and avoids needing to wrap the whole thing in a shell (which system, older popen, and getstatusoutput all do).

    Beyond that, it looks like when 3.8 releases, Popen should get *much* faster if the call meets certain conditions, see https://docs.python.org/3.8/whatsnew/3.8.html#optimizations for details. If you can make your use cases conform to those requirements (e.g. by using shutil.which to determine the absolute paths to your utilities instead of relying on PATH lookup), the speed up should eliminate (or more than eliminate) the performance regression you're seeing (per bpo-35537, on macOS, which got the least benefit, it was still a 1.7x improvement; on other OSes, the multiplier hits 61x or 106x).

    a4db0af7-fd9c-456a-9b4c-3abc97097202 commented 5 years ago

    I've tried to rewrite subporcess.getstatusoutput() usage with subprocess.Popen() and switch to shell=False, it didn't help, so I doubti it getstatusoutput() overhead, it's Popen() issue.

    pitrou commented 5 years ago

    Can you post a simple script showcasing the performance degradation?

    a4db0af7-fd9c-456a-9b4c-3abc97097202 commented 5 years ago

    I couldn't reproduce entire test, as wsdiff script is rather large, but here's the simplified version. If Popen is used more often, difference is much more significant.

    # Using workaround $ python3.5 \~/tmp/1.py 1 10.780487537384033 # Without workaround $ python3.5 \~/tmp/1.py 13.347045660018921 # Using python2.7 $ python2.7 \~/tmp/1.py 9.83385586739

    a4db0af7-fd9c-456a-9b4c-3abc97097202 commented 5 years ago

    Even if I use import subprocess process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines=True) return process.stdout.read() difference in times are the same.

    pitrou commented 5 years ago

    I don't seen any significant difference here (Ubuntu 18.04):

    $ time python2 sp.py 
    10.3433089256

    real 0m10,362s user 0m6,565s sys 0m4,372s

    $ time python3 sp.py 
    11.746907234191895

    real 0m11,781s user 0m7,356s sys 0m5,239s

    vstinner commented 5 years ago

    This issue lacks a lot of information:

    Attached 1.py uses 2 commands, appararently both use a shell:

    "objdump '%s'" is unsafe and can lead to shell injection: try to avoid the usage of a shell. Use subprocess.Popen directly, or an helper which doesn't use shell=True.

    48db9b7f-4b2d-4438-aab5-1b07c167d4a4 commented 5 years ago

    We have recently bumped into a similar problem. Using FreeBSD, subprocess calls were taking more than 10 times the usual time to execute after migrating to python3.6. After some digging, the default for 'close_fds' was changed to 'True'. On linux, this actually made things faster, but for unix, much slower. Passing 'close_fds=False' solved this for us.

    vstinner commented 5 years ago

    I create bpo-38061 "FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()".

    vstinner commented 4 years ago

    We've moved illumos-gate wsdiff python tool from Python 2 to Python 3.

    I guess that you're talking about https://github.com/illumos/illumos-gate/blob/master/usr/src/tools/scripts/wsdiff.py

    This project is part of illumos: "An open-source Unix operating system". It's a fork of Solaris.

    Python does not officially support Solaris not Illumos. If someone provides a patch, we can review it. But I'm not aware of anyone working actively on supporting Illumos.

    In the meanwhile, there are two workarounds:

    On Linux and FreeBSD (if fdescfs kernel module is loaded and mounted), Python 3 iterates on /proc/self/fd/. On FreeBSD, the closefrom() syscall can be used: PR 19697. But I'm not aware of any more efficient solution for Illumos.

    Since the latest comment of the original poster was last August, I close the issue.

    3b1c5871-4fff-449a-b2cf-9991c772fecb commented 4 years ago

    closefrom() is on both Solaris and illumos too - and might even have originated there as an API - so if that's the issue, it should be trivially fixable

    pitrou commented 4 years ago

    I think we would accept a PR since it would probably be trivial, but someone has to submit it :-)

    vstinner commented 4 years ago

    I merged FreeBSD changes:

    New changeset 162c567d164b5742c0d1f3f8bd8c8bab9c117cd0 by Victor Stinner in branch 'master': bpo-38061: os.closerange() uses closefrom() on FreeBSD (GH-19696) https://github.com/python/cpython/commit/162c567d164b5742c0d1f3f8bd8c8bab9c117cd0

    New changeset e6f8abd500751a834b6fff4f107ecbd29f2184fe by Victor Stinner in branch 'master': bpo-38061: subprocess uses closefrom() on FreeBSD (GH-19697) https://github.com/python/cpython/commit/e6f8abd500751a834b6fff4f107ecbd29f2184fe

    Free feel to propose a more generic way to detect when closefrom() function is available ;-) Or just a patch to add "|| defined(sun)".

    3b1c5871-4fff-449a-b2cf-9991c772fecb commented 4 years ago

    I checked, and the supposition this is due to lack of closefrom() doesn't seem to be correct. Running the test case and looking at 'truss' output, there is no large number of close() that one would expect if this was the issue.

    I don't see Alexander's 2-time speedup however:

    root@piano:/export/src/cpython# /export/python3/bin/python3 ./1.py 11.679689645767212 root@piano:/export/src/cpython# vi 1.p^C root@piano:/export/src/cpython# /export/python3/bin/python3 ./1.py foo 10.402687549591064 root@piano:/export/src/cpython# python2.7 ./1.py 10.0434100628

    Any difference doesn't seem to be distinguishable from noise on my system.

    If I run this:

    from __future__ import print_function
    import os
    import sys
    import time
    
    def getoutput(cmd):
            # Hand-crafted variant
            if len(sys.argv) >1:
                    import shlex, tempfile
                    f, fpath = tempfile.mkstemp()
                    status = os.system("{ " + cmd + "; } >" +
                            shlex.quote(fpath) + " 2>&1")
                    with os.fdopen(f, "r") as tfile:
                            output = tfile.read()
                    os.unlink(fpath)
                    if output[-1:] == '\n':
                            output = output[:-1]
                    return output
            else:
                    import subprocess
                    p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=None, close_fds=True)
                    return p.communicate()[0]
    
    t = time.time()
    for file in getoutput("find /usr/bin -type f 2>/dev/null").decode().split('\n'):
            diff = getoutput("/usr/bin/objdump '%s' 2>/dev/null" % file)

    then I get more variation than can be measured by changing close_fds.

    Running something similar (no decode()) under python 2.7 is *slower* than python3.

    So, something other than closefrom() is going on here.