name conflicts between deprecated or removed stdlib modules and those in the cwd break standard_library.install_aliases()

[Russell-Jones]

Symptom: if one runs

#!/usr/bin/env python

from future import standard_library
standard_library.install_aliases()

where there's a directory called commands containing the file __init__.py , this traceback occurs

Traceback (most recent call last):
  File "mytest.py", line 9, in <module>
    standard_library.install_aliases()
  File "/usr/lib/python2.7/site-packages/future/standard_library/__init__.py", line 460, in install_aliases
    obj = getattr(oldmod, oldobjname)

It doesn't happen if one commesnts out the lines

         ('subprocess', 'getoutput', 'commands', 'getoutput'),
         ('subprocess', 'getstatusoutput', 'commands', 'getstatusoutput'),

in the file mentioned in the traceback.

This is reasonable for 2.6 and less where subprocess doesn't exist, but could the testing be tightened so it doesn't happen on 2.7?

[edschofield]

Thanks for the heads-up! Yes, I agree this would be preferable.

Would you like to submit a patch? I'd be grateful for the help! ...

On 3 Oct 2015, at 00:48, Russell-Jones notifications@github.com wrote:

Symptom: if one runs

!/usr/bin/env python

from future import standard_library standard_library.install_aliases() where there's a directory called commands containing the file init.py , this traceback occurs

Traceback (most recent call last): File "mytest.py", line 9, in standard_library.install_aliases() File "/usr/lib/python2.7/site-packages/future/standard_library/init.py", line 460, in install_aliases obj = getattr(oldmod, oldobjname) It doesn't happen if one commesnts out the lines

     ('subprocess', 'getoutput', 'commands', 'getoutput'),
     ('subprocess', 'getstatusoutput', 'commands', 'getstatusoutput'),

in the file mentioned in the traceback.

This is reasonable for 2.6 and less where subprocess doesn't exist, but could the testing be tightened so it doesn't happen on 2.7?

— Reply to this email directly or view it on GitHub.

[Russell-Jones]

Thanks for the quick response. Would this be OK, or is there a better way to do it? I don't really know much about futurize or 2to3 internals.

Update: Looking at this again, 3 has getoutput and getstatusoutput, but they are described as legacy. Subprocess was added in 2.4, but the check_output() function was added in 2.7, giving a cleaner 2/3 option.:

standard_library/__init__.py

MOVES = [('collections', 'UserList', 'UserList', 'UserList'),
         ('collections', 'UserDict', 'UserDict', 'UserDict'),
         ('collections', 'UserString','UserString', 'UserString'),
         ('itertools', 'filterfalse','itertools', 'ifilterfalse'),
         ('itertools', 'zip_longest','itertools', 'izip_longest'),
         ('sys', 'intern','__builtin__', 'intern'),
         # The re module has no ASCII flag in Py2, but this is the default.
         # Set re.ASCII to a zero constant. stat.ST_MODE just happens to be one
         # (and it exists on Py2.6+).
         ('re', 'ASCII','stat', 'ST_MODE'),
         ('base64', 'encodebytes','base64', 'encodestring'),
         ('base64', 'decodebytes','base64', 'decodestring'),
]
if PY2 and not PY27:
    MOVES +=[ ('subprocess', 'getoutput', 'commands', 'getoutput'),
              ('subprocess', 'getstatusoutput', 'commands', 'getstatusoutput'),]

MOVES += [ ('subprocess', 'check_output', 'future.backports.misc', 'check_output'),
         ('math', 'ceil', 'future.backports.misc', 'ceil'),
         ('collections', 'OrderedDict', 'future.backports.misc', 'OrderedDict'),
         ('collections', 'Counter', 'future.backports.misc', 'Counter'),
         ('itertools', 'count', 'future.backports.misc', 'count'),
         ('reprlib', 'recursive_repr', 'future.backports.misc', 'recursive_repr'),

moves/subprocess

if PY2 and not PY27:
    __future_module__ = True
    from commands import getoutput, getstatusoutput

I'll set up a PR if it looks OK.

[Russell-Jones]

On further consideration, I don't think this is a good approach. I'm going to look at solving it more generally. One approach would be manipulating the import mechanism in standard_library/__init__.py to include only the stdlib in the import path. I'll see if I can work out a way to do that.

[Russell-Jones]

I'm looking at using something like the following instead of __import__(), possibly reusing/refactoring the code in RenameImport._find_and_load_module() https://docs.python.org/2/library/imp.html#imp.find_module

I need to write some tests to check I have it right, but I think this is the right direction, at least. The tests would in python 2 for the keys in MOVES create (name).py and (name)/__init__.py each raising an ImportError, then checking they don't happen. I think others are necessary, but I'm out of time for now.

Edit: I think one'd be to pull an import of each type (like imp.PKG_DIRECTORY) from stdlib that's not in MOVES.

path = [x for x in sys.path if x.startswith('/') and x != os.getcwd()]
module_info = imp.find_module(newmodname)
if os.path.dirname(os.path.abspath(module_info[1])) == os.getcwd():
# i.e. an import from the current directory
    module_info = imp.find_module(newmodname, path)
imp.load_module(newmodname, *module_info)

[rooftopcellist]

@Russell-Jones @edschofield I think that either of the proposed solutions would work. I am also running in to this problem in our project AWX though one of our dependencies, django-radius. Is there a plan to go through with either of this?

Our issue: https://github.com/ansible/awx/issues/1979

[lboudard]

@edschofield I'm encountering same issue https://stackoverflow.com/questions/52061056/python-future-install-aliases-module-object-has-no-attribute-getoutput

is it possible to bypass it waiting for a fix? Thanks!

[potasiak]

Please check whether you don't have a command.py (or command.pyc) in your current working directory, when running Python scripts / applications where this issue occurs.

In our case, we've been running Ansible provisioning but did not set a CWD when running Python scrips with shell/command modules, so user's home directory was used as the CWD.

In the user's directory, there are some Python scripts that we use for various semi-automatic operations, including one named "commands.py". When we run any scripts with user's home directory as CWD, Python resolved the commands import to this file instead of the Python's built-in commands module.

In summary, my recommendation would be to:

• Check current working directory for the scripts where the issue occurs.
• Check Python imports search paths (sys.path).
• Debug script to check where the module causing this issue is imported from.