[1.1.0] Compatibility with AutoPkg 2.x (Python 3)

homebysix commented 4 years ago

Summary

A major update to the AutoPkg command line tool is just around the corner. AutoPkg 2.0 is due in early February 2020, and it will run in Python 3 only. In order for JSSImporter to work with AutoPkg 2.0, a few moderate tweaks will be necessary to both JSSImporter and python-jss. I'm opening this issue as a place to track those tweaks and link to pull requests that might resolve them.

Proposed changes

Here's the high-level list of the changes I think that will need to be made. I already have development branches that incorporate many of these changes, but it would be good to have a second or third pair of eyes.

For reference, the pull requests that contain these changes are here:

Imports

Most imports work fine in both Python 2 and Python 3, but python-jss's import uapiobjects as uapi fails. This should be addressed by https://github.com/homebysix/python-jss/commit/980b62799a8b34f67ee7da1b5af818953898aaa5.

Print statements

Simple matter of changing print foo to print(foo) in most cases. Should be addressed by https://github.com/homebysix/python-jss/commit/5a88ff17d7bacd5eb33547212f7023ce081b50ff.

xml.etree.ElementTree

This is a bit out of my wheelhouse, but as I understand it there are two implementations of ElementTree, and we prefer the Python implmentation. A regular from xml.etree import ElementTree results in the C implementation, which Shea was working around using Element = ElementTree._Element_Py. That no longer seems to work the same in Python 3.7.x, but if we pop the standard xml.etree.ElementTree, we can then use ElementTree = importlib.import_module('xml.etree.ElementTree') to load the Python module. I've made this adjustment to python-jss in https://github.com/homebysix/python-jss/commit/2fdc4b7432df73b519c7cb69cf693ded082e7aa9 and to JSSImporter in https://github.com/homebysix/JSSImporter/commit/fc6bfbb7edf2104633b43436b243630f81eaff67. I'm not sure of the underlying mechanics here, but it seems to work in Python 3.7.x while retaining compatibility with Python 2.7.x.

String types

JSSImporter and python-jss include a few references to unicode and basestring, which are only used in Python 2. While it's possible to use six.string_types to resolve this, I've chosen instead to map unicode and basestring to Python 3's str, which includes both plain text and unicode. This is similar to the approach taken by Munki. I've made the change for JSSImporter in https://github.com/homebysix/JSSImporter/commit/ec8077d6a73cdab6d645bd8d512a12497232421b and for python-jss in https://github.com/homebysix/python-jss/commit/8904c92f0a5a2bb06e8cd60e0e0a3a31bf028e16 and https://github.com/homebysix/python-jss/commit/b3e2c163289160f9a2d071cd044a8c42846c9e5c. I've also changed unicode() calls to str.encode("utf-8") in https://github.com/homebysix/python-jss/commit/0d0452fd0bd8bd0441bb6bec19e72d68c818c46b.

dict.iteritems()

Calls to dict.iteritems() should be replaced by the equivalent use of dict.items() or dict.keys(). I've adjusted this in https://github.com/homebysix/python-jss/commit/daf7b4d0c6d527a07864916cf1421eaf169d4ce8 for python-jss and https://github.com/homebysix/JSSImporter/commit/5240ab7383e675416d66952a14a9be868f841089 and https://github.com/homebysix/JSSImporter/commit/58606669f4210884147b36f4d916a1b89af742a0 for JSSImporter.

Encoding and decoding

In Python 3, some functions produce bytes output instead of strings. Specifically, calls to requests.response.content and subprocess.check_output() that worked in Python 2 need to be explicitly decoded or cast as strings before they'll work in Python 3. I've done this in https://github.com/homebysix/python-jss/commit/1f70b24aef7f8c2ffae279ed8b127e8db86b1716 and https://github.com/homebysix/python-jss/commit/f0c359d8dbb9e5b0eea0b6cf20331ebfbe953def.

FoundationPlist and GURL

These Munki modules have both been updated upstream, including some adjustments for Python 3 compatibility. I've brought in those changes in https://github.com/homebysix/python-jss/commit/18d1093ded1922cea53cca5921d0af67f23b54f9 and https://github.com/homebysix/python-jss/commit/c68eb813a60e9bff93f821c67b25d4ccafb759f5.

Testing

I would love for least one or two others to validate that my branches allow them to successfully run their JSS recipes/overrides on both AutoPkg 1.4.1 and AutoPkg 2.0 RC1.

I've found the simplest way to put the test code in place is to overwrite the folder at /Library/Application Support/JSSImporter/jss and the file at /Library/AutoPkg/autopkglib/JSSImporter.py with the corresponding paths from my branches of the python-jss and JSSImporter projects. You'll need to update your override trust info as well.

Next steps

If the testing goes well, feel free to merge the changes. It may be wise to post minimum viable releases for Python 3 compatibility soon so that Mac admins can be ready for the early February AutoPkg 2.0 release.

There are a few non-urgent issues raised by pylint that can be addressed in the future, and I also have ideas for how to incorporate pre-commit, but those items don't need to hold up the process at this point.

Thanks for your consideration!

grahampugh commented 4 years ago

Hi @homebysix - this is amazing work, thank you so much for your efforts.

In my initial tests, I have found that my usual build method (using the makefile which utilises munkipkg) to create the installation package is no longer incorporating requests and boto into the package. To make JSSImporter work, I therefore have to manually install them afterwards using pip3 install requests boto --user. Any ideas what might have changed to cause this?

This is my normal quick build-and-install method while testing:

make clean; make; sudo installer -pkg pkg/jssimporter/build/jssimporter-1.1.0.pkg -tgt /

grahampugh commented 4 years ago

I believe I have now solved this - I don't think the bundled-in requests were ever really being used. I will pull your branches into the jssimporter organisation so that I can merge in the subsequent changes.

jssimporter / JSSImporter