Python3

[rjloura]

This PR merges @deserat python3 feature branch with master plus some additional fixes for py3 issues I found along the way. Fixes #8

[rjloura]

$ make testall
python test/testall.py
-- test with Python 2.7 (/usr/bin/python)
auth/privatekey/sign_and_verify_binary_ecdsa_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_384_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_384_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_521_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_521_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_sha256 ... ok
auth/privatekey/sign_and_verify_rsa_md5 ... ok
auth/privatekey/sign_and_verify_rsa_sha265 ... ok
mantaclient/cleantestarea/clean ... ok
mantaclient/dir/listheaddel ... ok
mantaclient/dir/put ... ok
mantaclient/link/put ... ok
mantaclient/manyfiles/count [slow] ... ok
mantaclient/misc/imports ... ok
mantaclient/misc/version ... ok
mantaclient/object/putgetdel ... ok
mantash/find/empty ... ok
mantash/find/type ... ok
mantash/ls/bare ... ok
mantash/ls/dir ... ok
mantash/ls/dirstar ... ok
mantash/options/help ... ok
mantash/options/version ... ok

----------------------------------------------------------------------
Ran 24 tests in 60.906s

OK
-- test with Python 3.6 (/home/rui/.virtualenvs/pymanta3/bin/python)
auth/privatekey/sign_and_verify_binary_ecdsa_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_384_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_384_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_521_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_521_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_sha256 ... ok
auth/privatekey/sign_and_verify_rsa_md5 ... ok
auth/privatekey/sign_and_verify_rsa_sha265 ... ok
mantaclient/cleantestarea/clean ... ok
mantaclient/dir/listheaddel ... ok
mantaclient/dir/put ... ok
mantaclient/link/put ... ok
mantaclient/manyfiles/count [slow] ... ok
mantaclient/misc/imports ... ok
mantaclient/misc/version ... ok
mantaclient/object/putgetdel ... ok
mantash/find/empty ... ok
mantash/find/type ... ok
mantash/ls/bare ... ok
mantash/ls/dir ... ok
mantash/ls/dirstar ... ok
mantash/options/help ... ok
mantash/options/version ... ok

----------------------------------------------------------------------
Ran 24 tests in 63.433s

OK

[cburroughs]

Thanks for running with this! Our python readiness as it related to being on supported releases will rapidly surpass nodejs!

I don't have a lot of experience with python3, but do have some with code bases that maintains compatibility with both. GitHub tells me +1,247 −645, but I see mostly formatting changes. It's hard to tell which are the relevant parts to look at. I don't want to make work to unwind all that and would be fine with applying pycodestyle after this is in. In a similar vein we should probably just use tox to test multiple environments instead of trying to maintain our own thing, but that doesn't need to be part of this. The change that did jump out at me around things like the exception syntax looked reasonable.

I was also able to run the integration tests against both python2.7 and 3.6. I dusted off a few automated tools. I am confused by the suggestion of print((some_expression)) over print(some_expression), but otherwise I believe what they are saying is mostly correct. For example, if I fire up python3 on the terminal it does die on func.func_name

$ pylint --py3k manta/
************* Module manta
manta/__init__.py:5:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module manta.auth
manta/auth.py:5:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module manta.appdirs
manta/appdirs.py:18:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
manta/appdirs.py:251:4: W1648: Module moved in Python 3 (bad-python3-import)
************* Module manta.cmdln
manta/cmdln.py:42:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
manta/cmdln.py:60:12: W1609: raw_input built-in referenced (raw_input-builtin)
manta/cmdln.py:1701:23: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
manta/cmdln.py:1703:24: W1656: dict.values referenced when not iterating (dict-values-not-iterating)
manta/cmdln.py:1705:23: W1655: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module manta.errors
manta/errors.py:6:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module manta.client
manta/client.py:4:0: W1618: import missing `from __future__ import absolute_import` (no-absolute-import)
manta/client.py:57:23: W1602: basestring built-in referenced (basestring-builtin)
manta/client.py:801:26: W1619: division w/o __future__ statement (old-division)

$ python-modernize --no-six manta/
 Loading the following fixers:
    lib2to3.fixes.fix_apply  (apply)
    lib2to3.fixes.fix_except  (except)
    lib2to3.fixes.fix_exec  (exec)
    lib2to3.fixes.fix_execfile  (execfile)
    lib2to3.fixes.fix_exitfunc  (exitfunc)
    lib2to3.fixes.fix_funcattrs  (funcattrs)
    lib2to3.fixes.fix_has_key  (has_key)
    lib2to3.fixes.fix_idioms  (idioms)
    lib2to3.fixes.fix_long  (long)
    lib2to3.fixes.fix_methodattrs  (methodattrs)
    lib2to3.fixes.fix_ne  (ne)
    lib2to3.fixes.fix_numliterals  (numliterals)
    lib2to3.fixes.fix_operator  (operator)
    lib2to3.fixes.fix_paren  (paren)
    lib2to3.fixes.fix_reduce  (reduce)
    lib2to3.fixes.fix_renames  (renames)
    lib2to3.fixes.fix_repr  (repr)
    lib2to3.fixes.fix_set_literal  (set_literal)
    lib2to3.fixes.fix_standarderror  (standarderror)
    lib2to3.fixes.fix_sys_exc  (sys_exc)
    lib2to3.fixes.fix_throw  (throw)
    lib2to3.fixes.fix_tuple_params  (tuple_params)
    lib2to3.fixes.fix_types  (types)
    lib2to3.fixes.fix_ws_comma  (ws_comma)
    lib2to3.fixes.fix_xreadlines  (xreadlines)
    libmodernize.fixes.fix_file  (file)
    libmodernize.fixes.fix_import  (import)
    libmodernize.fixes.fix_next  (next)
    libmodernize.fixes.fix_print  (print)
    libmodernize.fixes.fix_raise  (raise)
 Applying the following explicit transformations:
    (None)

RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored manta/appdirs.py
--- manta/appdirs.py    (original)
+++ manta/appdirs.py    (refactored)
@@ -11,6 +11,8 @@
 # - Mac OS X: http://developer.apple.com/documentation/MacOSX/Conceptual/BPFileSystem/index.html
 # - XDG spec for Un*x: http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

+from __future__ import absolute_import
+from __future__ import print_function
 __version_info__ = (1, 2, 0)
 __version__ = '.'.join(map(str, __version_info__))

@@ -338,9 +340,9 @@
     print("-- app dirs (without optional 'version')")
     dirs = AppDirs(appname, appauthor, version="1.0")
     for prop in props:
-        print("%s: %s" % (prop, getattr(dirs, prop)))
+        print(("%s: %s" % (prop, getattr(dirs, prop))))

     print("\n-- app dirs (with optional 'version')")
     dirs = AppDirs(appname, appauthor)
     for prop in props:
-        print("%s: %s" % (prop, getattr(dirs, prop)))
+        print(("%s: %s" % (prop, getattr(dirs, prop))))
RefactoringTool: Refactored manta/auth.py
--- manta/auth.py   (original)
+++ manta/auth.py   (refactored)
@@ -2,6 +2,7 @@

 """Manta client auth."""

+from __future__ import absolute_import
 import binascii
 import sys
 import io
RefactoringTool: Refactored manta/client.py
--- manta/client.py (original)
+++ manta/client.py (refactored)
@@ -1,6 +1,7 @@
 # Copyright (c) 2016 Joyent, Inc.  All rights reserved.
 """The Manta client."""

+from __future__ import absolute_import
 import sys
 import logging
 import io
RefactoringTool: Refactored manta/cmdln.py
--- manta/cmdln.py  (original)
+++ manta/cmdln.py  (refactored)
@@ -36,6 +36,7 @@

 from __future__ import print_function

+from __future__ import absolute_import
 __version_info__ = (2, 0, 0)
 __version__ = '.'.join(map(str, __version_info__))

@@ -796,7 +797,7 @@
                 # *perhaps* consider: arg=value arg2=value2 ...
                 warnings.warn("argument '**%s' on '%s.%s' command "
                               "handler will never get values" %
-                              (name, self.__class__.__name__, func.func_name))
+                              (name, self.__class__.__name__, func.__name__))
             if co_flags & CO_FLAGS_ARGS:
                 name = argnames.pop(-1)
                 tail = "[%s...]" % name.upper()
RefactoringTool: Refactored manta/errors.py
--- manta/errors.py (original)
+++ manta/errors.py (refactored)
@@ -1,6 +1,7 @@
 # Copyright 2012 Joyent, Inc.  All rights reserved.
 """python-manta errors"""

+from __future__ import absolute_import
 __all__ = ["MantaError", "MantaResourceNotFoundError", "MantaAPIError"]

 import logging
RefactoringTool: Refactored manta/version.py
--- manta/version.py    (original)
+++ manta/version.py    (refactored)
@@ -1,5 +1,6 @@
 # Copyright (c) 2018, Joyent, Inc.

+from __future__ import print_function
 __version_info__ = (2, 7, 0)
 __version__ = ".".join(map(str, __version_info__))

RefactoringTool: Files that need to be modified:
RefactoringTool: manta/appdirs.py
RefactoringTool: manta/auth.py
RefactoringTool: manta/client.py
RefactoringTool: manta/cmdln.py
RefactoringTool: manta/errors.py
RefactoringTool: manta/version.py

[rjloura]

Thanks @cburroughs. Yes a lot of @deserat's patch was formatting changes. I decided to bring everything along instead of cherry picking just the python3 relevant stuff.

Good call running pylint --py3k. Here are the results I have now:

(pymanta3) rui@hayek:~/git/python-manta$ pylint --py3k manta/
No config file found, using default configuration
************* Module manta.appdirs
W:254, 4: Module moved in Python 3 (bad-python3-import)
************* Module manta.cmdln
W: 61,12: raw_input built-in referenced (raw_input-builtin)
W:1702,23: dict.keys referenced when not iterating (dict-keys-not-iterating)
W:1704,24: dict.values referenced when not iterating (dict-values-not-iterating)
W:1706,23: dict.keys referenced when not iterating (dict-keys-not-iterating)

------------------------------------------------------------------
Your code has been rated at 9.97/10 (previous run: 9.97/10, +0.00)

The manta.appdirs warning is about the _winreg module which no longer exists in python3. I am inclined to punt on this for now and open an issue for ensuring windows support for 2 reasons: 1) I don't have a way to test windows :), and 2) in the README.md we note that windows isn't officially supported. I opened (#59) for this.

The manta.cmdln warnings should probably be fixed in "upstream" https://github.com/trentm/cmdln (see #57).

$ pylint --py3k test/*
No config file found, using default configuration

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

$ pylint --py3k bin/*
No config file found, using default configuration

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

[cburroughs]

Glad the tool helped. I added #61 & #62 for the followup I'd like to do.

[trentm]

[10:22:09 trentm@bluesteel:~/joy/python-manta (pr-58)]
$ python3 bin/mantash
b'[trent.mick@us-east.manta.joyent.com /trent.mick/stor]$ '
b'[trent.mick@us-east.manta.joyent.com /trent.mick/stor]$ 'ls
.joyent
bin
books

Note the b'...' quoting of the PS1.

[rjloura]

The b' ' quoting is due to the Mantash._update_prompt() method, which binary encodes the PS1:

        allkeys = ''.join(codes.keys()).replace('\\', '\\\\')
        uprompt = re.sub(r'\\([%s])' % allkeys, repl, self.ps1)
        self._prompt_str = uprompt.encode(sys.stdout.encoding or 'UTF-8',
                                          'replace') 

Later when this is printed with Cmdln.stdout.write() in Cmdln.cmdloop() we pass it to Cmdln._str() first, but that tries to either encode it (even though it is already encoded) or get its str() representation.

In py2 str(b'some bytes') returns the byte object, but in py3 str(b'some bytes') returns b'some bytes'

We should not use str() in py3 per (https://docs.python.org/3/howto/pyporting.html#text-versus-binary-data).

A simple solution here would be to not encode the prompt_str in Mantash._update_prompt(). But we may want to revisit Cmdln's handling of strings / bytes for output.

[rjloura]

Ran tests on Windows using cygwin:

$ python test/test.py
auth/privatekey/sign_and_verify_binary_ecdsa_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_384_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_384_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_521_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_521_sha256 ... ok
auth/privatekey/sign_and_verify_ecdsa_md5 ... ok
auth/privatekey/sign_and_verify_ecdsa_sha256 ... ok
auth/privatekey/sign_and_verify_rsa_md5 ... ok
auth/privatekey/sign_and_verify_rsa_sha265 ... ok
mantaclient/cleantestarea/clean ... ok
mantaclient/dir/listheaddel ... ok
mantaclient/dir/put ... ok
mantaclient/link/put ... ok
mantaclient/manyfiles/count [slow] ... ok
mantaclient/misc/imports ... ok
mantaclient/misc/version ... ok
mantaclient/object/putgetdel ... ok
mantash/find/empty ... ok
mantash/find/type ... ok
mantash/ls/bare ... ok
mantash/ls/dir ... ok
mantash/ls/dirstar ... ok
mantash/options/help ... ok
mantash/options/version ... ok

----------------------------------------------------------------------
Ran 24 tests in 65.005s

OK

Also tested some basic gets and puts as well as mantash command line.