Open homebysix opened 4 years ago
Hi @homebysix - this is amazing work, thank you so much for your efforts.
In my initial tests, I have found that my usual build method (using the makefile
which utilises munkipkg
) to create the installation package is no longer incorporating requests
and boto
into the package. To make JSSImporter work, I therefore have to manually install them afterwards using pip3 install requests boto --user
. Any ideas what might have changed to cause this?
This is my normal quick build-and-install method while testing:
make clean; make; sudo installer -pkg pkg/jssimporter/build/jssimporter-1.1.0.pkg -tgt /
I believe I have now solved this - I don't think the bundled-in requests
were ever really being used. I will pull your branches into the jssimporter organisation so that I can merge in the subsequent changes.
Summary
A major update to the AutoPkg command line tool is just around the corner. AutoPkg 2.0 is due in early February 2020, and it will run in Python 3 only. In order for JSSImporter to work with AutoPkg 2.0, a few moderate tweaks will be necessary to both JSSImporter and python-jss. I'm opening this issue as a place to track those tweaks and link to pull requests that might resolve them.
Proposed changes
Here's the high-level list of the changes I think that will need to be made. I already have development branches that incorporate many of these changes, but it would be good to have a second or third pair of eyes.
For reference, the pull requests that contain these changes are here:
Imports
Most imports work fine in both Python 2 and Python 3, but python-jss's
import uapiobjects as uapi
fails. This should be addressed by https://github.com/homebysix/python-jss/commit/980b62799a8b34f67ee7da1b5af818953898aaa5.Print statements
Simple matter of changing
print foo
toprint(foo)
in most cases. Should be addressed by https://github.com/homebysix/python-jss/commit/5a88ff17d7bacd5eb33547212f7023ce081b50ff.xml.etree.ElementTree
This is a bit out of my wheelhouse, but as I understand it there are two implementations of ElementTree, and we prefer the Python implmentation. A regular
from xml.etree import ElementTree
results in the C implementation, which Shea was working around usingElement = ElementTree._Element_Py
. That no longer seems to work the same in Python 3.7.x, but if wepop
the standardxml.etree.ElementTree
, we can then useElementTree = importlib.import_module('xml.etree.ElementTree')
to load the Python module. I've made this adjustment to python-jss in https://github.com/homebysix/python-jss/commit/2fdc4b7432df73b519c7cb69cf693ded082e7aa9 and to JSSImporter in https://github.com/homebysix/JSSImporter/commit/fc6bfbb7edf2104633b43436b243630f81eaff67. I'm not sure of the underlying mechanics here, but it seems to work in Python 3.7.x while retaining compatibility with Python 2.7.x.String types
JSSImporter and python-jss include a few references to
unicode
andbasestring
, which are only used in Python 2. While it's possible to usesix.string_types
to resolve this, I've chosen instead to mapunicode
andbasestring
to Python 3'sstr
, which includes both plain text and unicode. This is similar to the approach taken by Munki. I've made the change for JSSImporter in https://github.com/homebysix/JSSImporter/commit/ec8077d6a73cdab6d645bd8d512a12497232421b and for python-jss in https://github.com/homebysix/python-jss/commit/8904c92f0a5a2bb06e8cd60e0e0a3a31bf028e16 and https://github.com/homebysix/python-jss/commit/b3e2c163289160f9a2d071cd044a8c42846c9e5c. I've also changedunicode()
calls tostr.encode("utf-8")
in https://github.com/homebysix/python-jss/commit/0d0452fd0bd8bd0441bb6bec19e72d68c818c46b.dict.iteritems()
Calls to
dict.iteritems()
should be replaced by the equivalent use ofdict.items()
ordict.keys()
. I've adjusted this in https://github.com/homebysix/python-jss/commit/daf7b4d0c6d527a07864916cf1421eaf169d4ce8 for python-jss and https://github.com/homebysix/JSSImporter/commit/5240ab7383e675416d66952a14a9be868f841089 and https://github.com/homebysix/JSSImporter/commit/58606669f4210884147b36f4d916a1b89af742a0 for JSSImporter.Encoding and decoding
In Python 3, some functions produce bytes output instead of strings. Specifically, calls to
requests.response.content
andsubprocess.check_output()
that worked in Python 2 need to be explicitly decoded or cast as strings before they'll work in Python 3. I've done this in https://github.com/homebysix/python-jss/commit/1f70b24aef7f8c2ffae279ed8b127e8db86b1716 and https://github.com/homebysix/python-jss/commit/f0c359d8dbb9e5b0eea0b6cf20331ebfbe953def.FoundationPlist and GURL
These Munki modules have both been updated upstream, including some adjustments for Python 3 compatibility. I've brought in those changes in https://github.com/homebysix/python-jss/commit/18d1093ded1922cea53cca5921d0af67f23b54f9 and https://github.com/homebysix/python-jss/commit/c68eb813a60e9bff93f821c67b25d4ccafb759f5.
Testing
I would love for least one or two others to validate that my branches allow them to successfully run their JSS recipes/overrides on both AutoPkg 1.4.1 and AutoPkg 2.0 RC1.
I've found the simplest way to put the test code in place is to overwrite the folder at /Library/Application Support/JSSImporter/jss and the file at /Library/AutoPkg/autopkglib/JSSImporter.py with the corresponding paths from my branches of the python-jss and JSSImporter projects. You'll need to update your override trust info as well.
Next steps
If the testing goes well, feel free to merge the changes. It may be wise to post minimum viable releases for Python 3 compatibility soon so that Mac admins can be ready for the early February AutoPkg 2.0 release.
There are a few non-urgent issues raised by
pylint
that can be addressed in the future, and I also have ideas for how to incorporate pre-commit, but those items don't need to hold up the process at this point.Thanks for your consideration!