"webdocs" make target fails under Python 3

[Ichimonji10]

This bug was originally filed against version 2.0.0-0.7.rc2. However, this bug is also reproductible on the "devel" branch, and probably other branches too. Keep that in mind when reading the remainder of this issue.

Issue Type

Bug Report

Ansible Version

I'm running Ansible 2.0.0-0.7.rc2:

$ ansible --version
ansible 2.0.0 (detached HEAD cc98528ecb) last updated 2015/12/07 14:44:25 (GMT -400)
  lib/ansible/modules/core:  not found - use git submodule update --init lib/ansible/modules/core
  lib/ansible/modules/extras:  not found - use git submodule update --init lib/ansible/modules/extras
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

More precisely:

$ git status
HEAD detached at v2.0.0-0.7.rc2
nothing to commit, working directory clean

Ansible Configuration

I have changed nothing. I am running directly from the git repository.

Environment

I'm running an up-to-date version of Arch Linux. However, that shouldn't matter much, given that Ansible itself is running in a virtualenv. The virtualenv setup procedure is as follows:

# create a virtualenv
N=2
virtualenv --python python$N ~/.virtualenvs/ansible$N
source ~/.virtualenvs/ansible$N/bin/activate

# install ansible from source
git clone git@github.com:ansible/ansible.git
cd ansible
git checkout v2.0.0-0.7.rc2
pip install -e .

I do the same for Python 3 by setting N=3 and repeating the procedure. The Python versions as reported by python --version are Python 2.7.10 and Python 3.5.0.

Summary

The webdocs make target fails for Ansible v2.0.0-0.7.rc7 under Python 3. It fails due to a variety of syntax errors, such as the use of except Foo, e and print 'foo'.

Steps To Reproduce

Create a virtualenv and install ansible from source, as outlined in the "Environment" section. Execute make webdocs.

Expected Results

make webdocs completes and produces usable documentation.

Actual Results

make webdocs completes under Python 2, but not Python 3.

Proposed Fix

According to Community Information & Contributing → Ansible Users → Contributing Code (Features or Bugfixes), "code developed for Ansible needs to support Python 2.6+". According to that standard, it should be OK to do things like replace a except Foo, e with except Foo as e.

In my environment (Python 2.7.10 and 3.5.0), the following change works allows me to execute make webdocs under Python 2 and 3. However, I'm not sure how to properly test this change to ensure it does not break anything.

diff --git a/docsite/build-site.py b/docsite/build-site.py
index 587a189..24f9fc9 100755
--- a/docsite/build-site.py
+++ b/docsite/build-site.py
@@ -15,6 +15,7 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+from __future__ import print_function

 __docformat__ = 'restructuredtext'

@@ -24,9 +25,9 @@ import traceback
 try:
     from sphinx.application import Sphinx
 except ImportError:
-    print "#################################"
-    print "Dependency missing: Python Sphinx"
-    print "#################################"
+    print("#################################")
+    print("Dependency missing: Python Sphinx")
+    print("#################################")
     sys.exit(1)
 import os

@@ -40,7 +41,7 @@ class SphinxBuilder(object):
         """
         Run the DocCommand.
         """
-        print "Creating html documentation ..."
+        print("Creating html documentation ...")

         try:
             buildername = 'html'
@@ -69,10 +70,10 @@ class SphinxBuilder(object):

             app.builder.build_all()

-        except ImportError, ie:
+        except ImportError:
             traceback.print_exc()
-        except Exception, ex:
-            print >> sys.stderr, "FAIL! exiting ... (%s)" % ex
+        except Exception as ex:
+            print("FAIL! exiting ... (%s)" % ex, file=sys.stderr)

     def build_docs(self):
         self.app.builder.build_all()
@@ -83,9 +84,9 @@ def build_rst_docs():

 if __name__ == '__main__':
     if '-h' in sys.argv or '--help' in sys.argv:
-        print "This script builds the html documentation from rst/asciidoc sources.\n"
-        print "    Run 'make docs' to build everything."
-        print "    Run 'make viewdocs' to build and then preview in a web browser."
+        print("This script builds the html documentation from rst/asciidoc sources.\n")
+        print("    Run 'make docs' to build everything.")
+        print("    Run 'make viewdocs' to build and then preview in a web browser.")
         sys.exit(0)

     build_rst_docs()
@@ -93,4 +94,4 @@ if __name__ == '__main__':
     if "view" in sys.argv:
         import webbrowser
         if not webbrowser.open('htmlout/index.html'):
-            print >> sys.stderr, "Could not open on your webbrowser."
+            print("Could not open on your webbrowser.", file=sys.stderr)
diff --git a/hacking/module_formatter.py b/hacking/module_formatter.py
index f4ab5d7..4c94ca3 100755
--- a/hacking/module_formatter.py
+++ b/hacking/module_formatter.py
@@ -140,7 +140,7 @@ def list_modules(module_dir, depth=0):
             if os.path.isdir(d):

                 res = list_modules(d, depth + 1)
-                for key in res.keys():
+                for key in list(res.keys()):
                     if key in categories:
                         categories[key] = merge_hash(categories[key], res[key])
                         res.pop(key, None)
@@ -451,7 +451,7 @@ def main():

     categories = list_modules(options.module_dir)
     last_category = None
-    category_names = categories.keys()
+    category_names = list(categories.keys())
     category_names.sort()

     category_list_path = os.path.join(options.output_dir, "modules_by_category.rst")

[bcoca]

ansible itself does not work with python3, we have made much progress in adding support for it, but are not there yet.

we have not even started looking at things like the build utilities for documentation or anything else.

[Ichimonji10]

Good time as any to start? ¯\_(ツ)_/¯

[bcoca]

submit as a PR, then we can test and merge if possible.

[Ichimonji10]

Will do!

[Ichimonji10]

This issue affects at least the "devel" and "stable-2.0" branches. Given that "Patches should always be made against the ‘devel’ branch.", I'm updating this bug report and will submit the PR against the "devel" branch.

[Ichimonji10]

See #13470. Note that that PR is made against the "devel" branch, not the "stable-2.0" branch.

[bcoca]

@Ichimonji10 that is fine, PRs should always go against devel, we will backport if needed (since 2.0 is in feature freeze, this will probably be in 2.1)

[Ichimonji10]

OK. Good to know I'm on the right track.

[jimi-c]

Closed as the above was merged in, thanks again!

[Ichimonji10]

Thank you both.