Python 3 Support

[ryan-lane]

Salt is currently not compatible with python 3. Ubuntu trusty and above default to using python 3, so it would be really nice to be able to run salt in the default python version.

[s0undt3ch]

I believe that the only show stopper towards Py3 is m2crypto which is still a hard dependency for Salt.

On April 15, 2014 12:29:00 AM GMT+01:00, ryan-lane notifications@github.com wrote:

Salt is currently not compatible with python 3. Ubuntu trusty and above default to using python 3, so it would be really nice to be able to run salt in the default python version.

---

Reply to this email directly or view it on GitHub: https://github.com/saltstack/salt/issues/11995

Pedro Algarvio @ Phone

[ryan-lane]

Any milestone associated with fixing this?

[gtmanfred]

Looking through pycrypto on RHEL6, what is keeping us from just using pycrypto and dropping m2crypto?

It looks like someone was trying to include python3 support in the SWIG stuff for m2crypto, but abandoned it about a year ago when he failed to gain any traction upstream.

[s0undt3ch]

This is best answered by @thatch45

[cachedout]

Related to #7413

[s0undt3ch]

That's the one I couldn't remember(and the phone did not help). Thanks @cachedout!

[thatch45]

We just merged some code that deprecates the need for m2crypto. So we can start on full py3 support

[tebanep]

Any progress regarding this Issue? Python 3.5 was released yesterday and it will be the default in Ubuntu 16.04 LTS.

[s0undt3ch]

Several progresses, including dropping m2crypto, however, there's still work to be done. This is a massive task.

[kamikaze]

Please make Salt working with Python 3.5. Python 3 came out in 2008! C'moooon

[s0undt3ch]

...And only a couple of months ago we were able to drop m2crypto...

Believe me, all of us want Salt fully running under Py3, but have you looked at the current size of Salt's source?

... Its an ongoing daunting task, but its not forgotten...

One just needs to keep believing and submitting PRs...

[blarghmatey]

@s0undt3ch Is there any list of tasks related to Python 3 support? I think that this effort would benefit from having an easy to find set of issues/updates that would move the progress forward toward a 3.x supported release.

[s0undt3ch]

Full internal Unicode support, without breaking py2 compat. And I'm sure quite a few modules would need some syntax updates, ie, .iteritems() usage, for example.

But honestly, try to start a master in py3 and fix as you go along... Without breaking py2 compact....

Allowing the master to run in py3 will give us some milieage to this task. Then the minion, etc...

[deepakhj]

+1. Ubuntu 16.04 LTS installs Python 3 by default.

[damon-atkins]

+1 python library cryptography #30287 OpenStack have developed a library, anyone "platform" support openstack will need to use this crypto library I suspect. e.g. It's standard with Solaris 11, when Solaris provided support for OpenStack & Solaris Zones.

It would be nice to see Jenkins do a Python 3 Lint as a starting place

[thatch45]

Just as an update here, we have made great progress, but this is complicated. Once we can get tests passing on #32366 then we will be able to run remote execution, pillar and states on python 3, this will signify enough to start trying to run the full test suite on python3

As for lint, our linting currently looks for python3 issues as well as python 2 issues and we test that salt will install on python3, hence verifying that there are not syntax errors on python3.

[thatch45]

Ok, as an update here, I got the tests RUNNING on python3, we have a little over 300 tests out of well over 5000 that will need to be fixed based on our initial test runs, once #32929 is merged than we will post where the py3 jenkins tests are running

[DanyC97]

@thatch45 were are you going to post py3 jenkins tests ?

[thatch45]

I will post them here as well as well as on salt-users

[thatch45]

Sorry this took so long, but there was a lot that needed to get updated in the test suite to be able to handle it. The tests are ALMOST running, but we still have a few hicpus. You can watch as we get them closer and closer here: http://166.78.178.63:8080/job/branch_tests/job/develop/job/salt-linode-py3-arch/ http://166.78.178.63:8080/job/branch_tests/job/develop/job/salt-linode-py3-cent7/

We THINK they should be running later today, as we are down to what is just adding some deps to the test environment

[scoates]

I noticed that this IP no longer responds on this port:

~$ curl http://166.78.178.63:8080/
curl: (7) Failed to connect to 166.78.178.63 port 8080: Connection refused

Is this still running? Perhaps elsewhere?

[s0undt3ch]

Use https://jenkins.saltstack.com/, although, that job is probably long gone.

[scoates]

Thanks! Was hoping to determine the status of this myself, but it'd be much easier if someone from the core team could give us a current update (and possibly future projection) of Salt running on Python 3. Please? (-: I'd also like to find time to help if possible.

[cachedout]

Hi Sean,

We're making pretty good progress on this actually. We have about 75% of the tests passing right now on Python3.

If you'd like to help, we'd love it of course!

It's as easy as running the test suite in Python3 and fixing whatever doesn't pass. Let us know how we can help. Thanks!

-mp

On Mon, Aug 15, 2016 at 8:42 AM, Sean Coates notifications@github.com wrote:

Thanks! Was hoping to determine the status of this myself, but it'd be much easier if someone from the core team could give us a current update (and possibly future projection) of Salt running on Python 3. Please? (-: I'd also like to find time to help if possible.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/11995#issuecomment-239705861, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG0AELgVvZPHk-sIRAbiR0WKHDyBI9Cks5qf6gCgaJpZM4Byen- .

[scoates]

Could I trouble someone to write full instructions on how to go about this, now?

The instructions are out of date, it seems.

I hacked together what I thought might work, and it had been running since Sunday (with a lot of fails) and it kept getting hung up on I-can't-tell-what until I pressed ^C a few times. It finally finished with this:

================================================================================
FAILED (total=5976, skipped=1438, passed=4272, failures=119, errors=147)
============================  Overall Tests Report  ============================
Traceback (most recent call last):
  File "/home/vagrant/salt/venv/lib/python3.4/shutil.py", line 395, in _rmtree_safe_fd
    dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'top_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tests/runtests.py", line 603, in <module>
    main()
  File "tests/runtests.py", line 595, in main
    parser.finalize(1)
  File "/home/vagrant/salt/venv/lib/python3.4/site-packages/salttesting/parser/cover.py", line 234, in finalize
    SaltTestingParser.finalize(self, exit_code)
  File "/home/vagrant/salt/venv/lib/python3.4/site-packages/salttesting/parser/__init__.py", line 644, in finalize
    self.post_execution_cleanup()
  File "tests/runtests.py", line 368, in post_execution_cleanup
    TestDaemon.clean()
  File "/home/vagrant/salt/salt/tests/integration/__init__.py", line 1337, in clean
    shutil.rmtree(dirname, onerror=remove_readonly)
  File "/home/vagrant/salt/venv/lib/python3.4/shutil.py", line 463, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/vagrant/salt/venv/lib/python3.4/shutil.py", line 397, in _rmtree_safe_fd
    onerror(os.open, fullname, sys.exc_info())
  File "/home/vagrant/salt/salt/tests/integration/__init__.py", line 1333, in remove_readonly
    func(path)
TypeError: Required argument 'flags' (pos 2) not found

I suspect that is not expected. (-:

[thatch45]

What version of salt were you running against? On develop right now the entire test suite does fail on python3 in a way similar to what you are describing. While the test suite is not going to run clean for the upcoming release salt will install and run on python3. Cleaning up all of the test suite will likely take another release cycle, and the goal for the upcoming release is that people can install on python3 and help fix things. Right now, you can run subsections of the test suite on python3, just not the entire test suite quite yet.

[scoates]

Yep. I was/am on develop. Is there a better branch? Sorry for the potentially dumb questions, but I haven't found a doc that seems to work and would still like to help.

[thatch45]

Yes, develop is the right place, just keep in mind that it is a moving target, you are doing it right. Start by running sections of the test suite, thats how we are working on it right now. So you can run python3 tests/runtests.py -c to just run client tests, all of the sections are listed under python3 tests/runtests.py --help

[thatch45]

Ok, for the Carbon release we can run the master and minion on python3 and MOST of it works, the test suite runs but not completely. The goal for the Nitrogen releases after Carbon is that the test suite will run without failures on python3

[thatch45]

But in the upcoming release we will openly encourage users to try salt out on Python3 and help us get it over the line, at this point since it runs on Python3 it should be much easier for those seeking to help us finish Python3 support to assist.

[damon-atkins]

How about windows moving to python 3? It's 100% controlled buy Salt Stack it would seem to be the easiest to get over the line.

[thatch45]

We have been considering it. Much of the windows code is shared, but much is also isolated. We have been able to run salt on windows under python3. The thing though about us controlling everything with windows also lowers the need to run it on python 3, since we bundle everything. But yes, shipping the salt windows packages to use python3 is an goal.

[damon-atkins]

Python 3 has much better windows support due to the native Unicode support, or default position of strings being Unicode. An example is the registry binary API in python is string based talking to a Unicode registry. In PY3 the binary API is newer and its all Unicode. In order to fix a lot of the issues I had to convert strings back and forth for the old binary API in PY2.

[mswart]

@thatch45 Are there plans for the Carbon/Nitrogen releases to provide also packages with salt running Python 3? As salt is currently the only reason I still have Python 2, I would love to try it with Python 3. But it is a burden to package/install salt manually and would reduce the amount of testing I (and probably others) would/could do.

[thatch45]

@mswart we don't plan on python3 packages right away, but we are working toward having them for the follow on release of Salt. What platform are you on? We are debating on what platforms to start with. Also, if you are running RHEL/CentOS what python version are you using to get python3? Since that is the only major platform left that does not ship python3

[mswart]

@thatch45 I am using only Ubuntu and Debian. Salt with Python 3 is primary relevant on Ubuntu 16.04 LTS (with Python 3.5) and secondary on the upcoming Debian Stretch (probably with Python 3.6). Especially for new installations it is sad that salt newly installs Python 2 and a bunch of Python 2 packages - the default installations have no Python 2 anymore. Older versions of Ubuntu and Debian will first be upgraded before the switch to Python 3.

[thatch45]

Ubuntu is easier here, Ubuntu, Debian, SLES, all the LTS linux distros except RHEL have stock python3. I do think that we will work towards a Ubuntu package early on. Thanks for the info :)

[damon-atkins]

from __future__ import unicode_literals or from __future__ import absolute_import, unicode_literals I have been slowing adding this to windows, since much of Windows is Unicode It needs to be added to all *.py files to help with porting. Can it be added post Carbon Release RC1 to develop? FYI https://docs.djangoproject.com/en/1.10/topics/python3/

[rehmanzile]

Any idea on when python3 will be supported? All of my libraries are developed in python3 and salt is a show stopped at the moment. Thanks.

[meggiebot]

@rehmanzile we made some good progress supporting Python 3 for Carbon (releasing soon), but Python 3 isn't officially supported yet. We plan to make more progress for Nitrogen.

[damon-atkins]

@meggiebot Can SaltStack have a chat about from __future__ import unicode_literals see my posts. We need PY2 and PY3 both working in unicode. At the end of the day the Master could be running PY3 and the Windows Minions running PY3 and old Linux systems running PY2 minions, and new Linux systems running PY3 etc. They all need to treat text the same way.

[chnliyong]

salt-ssh and salt cp.push have problem using io function, as type str difference between python2 and python3. Python Version: 3.5.2 Salt Version: 2016.11.0rc1

python3.5/site-packages/salt/master.py", line 1306, in _file_recv fp_.write(load['data']) TypeError: a bytes-like object is required, not 'str'

[cachedout]

All tests now pass on Python 3. We'll be announcing support in Nitrogen in a few weeks.

[rehmanzile]

Awesome! 🥇

[thatch45]

[tonybaloney]

@cachedout did we get a coverage report for all the state, module and proxy packages? Also, did a static analyser get run over the code base?

I was looking at running a coverage tool yesterday to see where tests could be added to validate py3

[cdunklau]

Fantastic news!

I'm not really sure where to post this, so I'm using this issue since it likely has good visibility. Any decision to use the unicode_literals future import must not be considered lightly. I suspect the Salt dev team is aware of these issues since I only see it used in a few files, but just in case:

The from __future__ import unicode_literals method of making it "easy" to write polyglot 2/3 code is fraught with subtle dangers. PythonCharmers/python-future#22 discusses this at length, with lots of references to specific problems that it can cause.

[abcfy2]

I find the 2017.7.0 release notes mention that salt has supported python 3. But I don't find any docs about how to use python 3. And the official doc still says the dependency of python is Python 2.6 >= 2.6 <3.0.

[damon-atkins]

Python 2.7 is the minimum for 2017.7 I think Python 3.6 is the minimum for 2017.7

I would think not all modules would work with Python 3.6 yet, esp. the ones which are rarely used or do not have tests developed for them.

[twangboy]

Salt needs to support Python 3.4 for now. Windows ships with 3.5.