Upgrade to Python 3.3 or newer

[steinn]

Mailpile is currently written in Python 2.7, which is no longer being actively developed and improved by the Python community. In order to stay current with the state of the art and reap the benefits of ongoing developments, Mailpile should upgrade to Python 3.3 (or newer).

(Rephrased by BRE for voting. Original issue text follows.)

I'm curious what Python version(s) Mailpile is targeting. It would be interesting to see Mailpile supporting Python 2.7 and 3.3+

The current code runs on Python 2.7 and it can probably run on Python 2.6 with minor changes (if any). By only supporting 2.7 and 3.3+ it should hopefully be fairly easy to use the same code base.

Python 2.6 end of life is in October, PEP 361, and therefor I think a good reason to not supporting it. Python 2.7 end of life is in May 2015, PEP 373, which isn't far away!

Ubuntu is also planing on moving to Python 3 in the 14.04 Release, Ubuntu Python 3, which should be released according to schedule in April 2014. That alone is quite interesting and will hopefully get more people using Python 3.

An issue with updating the current code base to Python 3.3+ support is the Python GnuPGInterface library. It does not support Python 3.3 and probably never will (at least by upstream) since it has not been updated since 2002. Haven't looked into alternatives but there must be something.

If there is interest for his I'm willing to help.

[Ivoz]

Regarding the Python GnuPGInterface library, it seems there are newer versions (0.3.5) on both pypi and googlecode.

https://pypi.python.org/pypi/python-gnupg/ https://code.google.com/p/python-gnupg/

The newer PyPI tags claim compatibility all the way up to python 3.3, so it would be interesting to hear why the older sourceforge source is being used and not pypi.

[danx0r]

we may want to look at http://pythonhosted.org/six/

seems to facilitate writing 2.7-compatible code that is as future-proof against py3 changes as possible

[illume]

For best compatibility, single source is the way forward for python projects. Since the stated reason for staying with the legacy python 2.x was for best compatibility, it seems time to change the stance of the project to move to a single source.

It is almost April 2014 so more distros are moving to python 3. The python community is also moving in a big way to python 3.3/3.4. As previously said, python 2.7 is nearing end of life, meaning no security fixes, and no updates for platforms (new windows/osx etc). Writing a single source code base that works on 3.3+ and 2.6/2.7 isn't too much different, and lots of projects have documented their experiences with this method now.

As well as six, there is http://python-future.org/ and some of the projects with small compatibility modules include twisted, numpy, pygame, coverage, etc.

Here is an old blog post from 2009 showing how it is done: http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html

The jinja2 package has a _compat module which is about 100 lines long: https://github.com/mitsuhiko/jinja2/blob/master/jinja2/_compat.py

Since you are already using jinja2, I don't see why you couldn't vendorise its compatibility module :)

[BjarniRunar]

I agree that this is becoming more compelling all the time. We have already started to hit what at first glance looks like limitations in the python base libraries (e.g. #558 ) and it's unlikely that those bugs will ever get fixed on the 2.x branch.

I also did some very basic tests which implied that Python 3 is much better at releasing unused memory back to the operating system, and considering that RAM usage is one of our main bottlenecks that is a strong argument in favour of making the switch.

The core team (myself, @smari and @brennannovak) is discussing this.

[nifgraup]

Python 2.7 EOL has been moved 5 years into the future, see http://hg.python.org/peps/file/76d43e52d978/pep-0373.txt

[Ivoz]

Mailpile is still a relatively new application, so it shouldn't be aiming to stay behind.

[steinn]

@nifgraup yes but after 2015 there isn't a guarantee of regular releases, only if the core devs have interest in it. I think they are hoping that vendors (ubuntu, red hat, etc.) will handle releasing patches (this is my understanding from the python dev mailing list).

[Ivoz]

The releases will essentially just be bugfix releases, when python devs feel one is warranted for whatever comes up in the future. The promise is essentially saying they'll keep on officially paying attention to 2.7 and releasing bugfixes as needed until 2020.

[nidico]

This is also relevant due to STARTTLS support in imaplib starting in Python 3.2 (see #868).

[fawkesley]

+1 for Python3 support, especially as you're a "greenfield" project.

I'm trying to run on Python 3 - blocker so far is the python spambayes library.

[Ivoz]

For python 3 libraries, there is a port of PyDNS called Py3DNS. I'm unsure about its cross compatibility with Python 2. PGPDump is 2/3 compatible, same with MarkupSafe. Spambayes hasn't been updated since 2011 and I assume is python 2 compatible only, so you'd need to work out what to do about that. Is it the best python library for filtering spam?

[fawkesley]

I looked into porting spambayes a few weeks back but fell at the first hurdle - sourceforge!! Might be worth forking and bringing over to github, wonder how the code looks... :S

[konklone]

Note that since this ticket was filed, the communty is up to Python 3.4 (3.4.3, as of this comment).

[nklever]

One more argument to upgrade to Python 3.x is due to STARTTLS support in pop3lib starting in Python 3.4 (see POP3lib Doc)

[joar]

I'd put a vote on this, however I do this.

I don't have any experience making single-source Python 2/3 applications beyond from __future__ import print_function, but as far as I know, six is the way to do it, and it's what Djjango uses.

I don't know how much 2to3 will do for you, but it might be an idea to run it against mailpile and see how much you get.

[Ivoz]

@joar the biggest hurdle is usually getting your dependencies 2/3 compatible first.

[lukebranch]

I recommend keeping Python 2.7 compatibility. There are still a huge number of projects and libraries that will stay with 2.7 for the foreseeable future.

[FirefighterBlu3]

I'm a huge user of py3, my entire set of servers in numerous datacenters is nearly Python 3.4 pure. It's pretty easy to support py2.7 with native py3 written code which is probably the best design since it only means a few tweak to support "old" python while the rest is designed for modern python.

I've just started in on mailpile and I'm happy to help with a bit of time here and there.

[BjarniRunar]

Just wanted to chime in again; thanks for all the comments guys. This remains on our long term roadmap, but won't be addressed in the near term because our priority at the moment is to just ship something that works. :-)

[da2x]

The Nikola project went through a Python version re-prioritization recently. Long story short, the project will drop Python 2.7 because it’s a hassle to maintain and most users run Python 3 or had it setup.

An environment with only Python 2.7 is not receiving updates and should be considered insecure. It’s five and a half year old and only receiving a trickle of maintenance updates. Given the Mailpile project’s focus is on security, it should only rely on recent software that receives updates. I see the major roadbloacks for running on Python 3 now are dependencies on libraries like Spambayes that aren’t receiving updates either.

Mailpile is now one of only five software on my machine that runs Py2 and not Py3. Please bring that list down to four.

[aviau]

:+1:

AFAIK @BjarniRunar does not have the time for that now. It would be great if one of you guys could start looking into dependencies like Spambayes, maybe contribute some Python3 support patches to them! I'm sure Python3 support patches wouldn't be refused in Mailpile either, but we have to find a way to do it incrementally and without affecting the upcoming release.

It could be a great way to start contirbuting to Mailpile:

• write tests for python2
• gradually enable tests for python3, as they work
• this way we detect regressions and we can move incrementally

If there is interest in helping the Mailpile Dev Team (the one and only @BjarniRunar) with this, I would definitely put some time in it.

Its relatively easy to support both Python3 and Python2. I think that this is what we should do for now.

[BjarniRunar]

@aviau That sounds like a very reasonable plan. Thank you for the suggestion!

[tilboerner]

What's the current state of this issue? I think it might be helpful to have a roadmap or checklist for the migration to Python 3. It would make it easy to tackle the project one step at a time, make progress visible, and newcomers might find it easier to find a starting point.

Which dependencies are there, which have completed the move, which ones haven't? Assuming the application will remain Python2 compatible for a while, which parts of the application can be migrated independently? Which are the necessary steps for each part? Is 3.3.* still the minimum target, or can it be moved to 3.4.*?

I'm thinking of something like the following (feel free to edit it out of this comment):

Dependencies

• [x] pyfoomatic (v3.1.4)
• [ ] dingbats (not as of v0.9)

Mailpile

• [ ] package
  • [ ] package.subpackage Here is a relevant note.
    • [x] test coverage
    • [ ] migrated code

[BjarniRunar]

Hi @tilboerner!

Current state: this is a "community only" issue at the moment, and I'm not aware of anyone from the community actively working on it. I'm not willing to spend any effort on it myself because I am already overworked and my efforts are more needed elsewhere (making a usable release).

So if you're interested, just dive in... whoever does the work becomes the boss. If you want to create a TODO list, that sounds super reasonable. If you want to just start randomly patching files to make them 3.x compatible, that's also perfectly fine. :-)

However, to prevent you from wasting your time: please keep pull requests very small in scope so I can easily review them. Many small pull requests will get merged much more quickly than a few large ones.

[koleesch1]

Hello, here is my requirements-dev.txt for pip to run virtualenv...

SpamBayes doesn't exists for python 3... Pydns moved to Py3dns

I get an error during start of mailpile "from jinja2 import Environment, BaseLoader, TemplateNotFound ImportError: No module named jinja2" jinja2 is installed...

requirements_dev.txt

appdirs setuptools>=11.3 cryptography>=1.3.4 lxml>=2.3.2 Jinja2

spambayes>=1.1b1

selenium>=2.40.0 markupsafe nose mock>=1.0.1 pycrypto py3dns pgpdump pillow pbr pycrypto

[jonathan-s]

@BjarniRunar It seems like spambayes is the main blocker for python3, would something like https://github.com/dinever/antispam be acceptable?

[BjarniRunar]

@jonathan-s: Mailpile's antispam and statistical classifiers are written in a modular way, so dropping in a replacement for spambayes should be relatively straightforward. So yes, if it works well then that's a roadblock cleared! The bar isn't low - spambayes works really well in my experience - but replacing it with something slightly worse but actively being developed/improved is still an option, especially if it lets us move towards Python 3.x down the line.

Looking at the antispam package you linked, the API it currently offers appears too limited (in particular, I saw no way to ensure the classifier database was saved encrypted), but if it's under development it could certainly be fixed.

[jonathan-s]

@BjarniRunar I've been trying to find other packages related to bayesian spam. Sadly it seems like there are not many packages out there. And for the linked package it looks fairly simple and I'm unsure whether it'll add many more features (unless someone from mailpile community decides that these features are important).

For mailpile what features are important for spam classification? You mentioned that the classifier database needs to be saved encrypted.

[BjarniRunar]

@jonathan-s, in practice load/save to/from an in-memory buffer is required for the encryption. But if there is a classifier class that cooperates with Python's pickle, then we get this free - so on second thought there may not be a need for an explicit API.

Our spambayes code uses a lower-level spambayes API to bypass the spambayes built-in tokenizer and instead use the Mailpile search-engine tokenizer - the idea being that consistency between those two subsystems would be useful for exploring/explaining spam classifier decisions at some point, but also to ensure that metadata like key IDs and message structure were used during classification. Basically, if it's useful to be able to search for an attribute, it should be a useful signal for classification.

Our entire interface with Spambayes is tiny: https://github.com/mailpile/Mailpile/blob/master/mailpile/plugins/autotag_sb.py

[jonathan-s]

Given the small footprint and the low-level implementation you're using it looks like it'll be pretty easy to copy the relevant files to use in mail pile internally.

On Sat 27. Jan 2018 at 15:13, Bjarni Rúnar Einarsson < notifications@github.com> wrote:

@jonathan-s https://github.com/jonathan-s, in practice load/save to/from an in-memory buffer is required for the encryption. But if there is a classifier class that cooperates with Python's pickle, then we get this free - so on second thought there may not be a need for an explicit API.

Our spambayes code uses a lower-level spambayes API to bypass the spambayes built-in tokenizer and instead use the Mailpile search-engine tokenizer - the idea being that consistency between those two subsystems would be useful for exploring/explaining spam classifier decisions at some point, but also to ensure that metadata like key IDs and message structure were used during classification. Basically, if it's useful to be able to search for an attribute, it should be a useful signal for classification.

Our entire interface with Spambayes is tiny: https://github.com/mailpile/Mailpile/blob/master/mailpile/plugins/autotag_sb.py

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mailpile/Mailpile/issues/160#issuecomment-360987389, or mute the thread https://github.com/notifications/unsubscribe-auth/ACBsEn7hd8O2V5TCta0XYd-4iXwyykZQks5tOy7qgaJpZM4A-GJ6 .

-- Sent from Gmail Mobile

[BjarniRunar]

@jonathan-s - I haven't looked into it, but that may also be a feasible way to resolve this. Part of me feels it would be better for the wider community if we made an effort to take over spambayes and port it to Python 3, but I have no idea how difficult that will be. And perfect is of course the enemy of good...

[jonathan-s]

Taking over spambayes would be a daunting task. And the gains would be neglible. The downloads of the library seems to have shrunk by ~90% looking at the sourceforge statistics.

On Sat 27. Jan 2018 at 16:04, Bjarni Rúnar Einarsson < notifications@github.com> wrote:

@jonathan-s https://github.com/jonathan-s - I haven't looked into it, but that may also be a feasible way to resolve this. Part of me feels it would be better for the wider community if we made an effort to take over spambayes and port it to Python 3, but I have no idea how difficult that will be. And perfect is of course the enemy of good...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mailpile/Mailpile/issues/160#issuecomment-360990538, or mute the thread https://github.com/notifications/unsubscribe-auth/ACBsEmt7c0l_38A_OolS8GI69m8zVm2Lks5tOzrqgaJpZM4A-GJ6 .

-- Sent from Gmail Mobile

[intika]

:skull: RIP Python 2 ^^

[darkuranium]

Any news on this? Python2's been EOL for nearly a year by now.

[bjmgeek]

It works for me on PyPy 3 Python 3.6.9 (7.3.2+dfsg-2, Sep 26 2020, 23:01:23) [PyPy 7.3.2 with GCC 10.2.0]

The only thing was that requirements.txt included pydns but for python3 it needs to be py3dns

update. I thought it worked, but I didn't realize that it was using python2.7 explicitly, instead of just /usr/bin/env python. When my distro removed python2, it stopped working.

[BjarniRunar]

Thank you for reporting that, @bjmgeek ! We had done a bunch of prep work, but actually testing it never happened. It's really heartening to hear this, makes the whole thing seem much less daunting.

[bjmgeek]

It doesn't pass unit tests though. pydns being one sticking point.

[raboof]

It works for me on PyPy 3

on master? I tried running master with python3 but that immediately produced errors - I tried to stumble through fixing some of those in https://github.com/mailpile/Mailpile/compare/master...raboof:python3 (but be aware I don't actually know python ;) ) - is there some other trick I'm missing?

[reyman]

I'm also interested by an answer on this @BjarniRunar @bjmgeek ? There is only py3dns that blocks ?

[alerque]

Every time I setup a new server with webmail I come around to see if Mailpile has ditched Python 2 yet. I so want to love this project, but it needs a refresh.

[alerque]

I got excited there for a minute, what it seems like "closed as completed" actually means "won't fix in this project, but someday the multi-year stealth project might pick up where this left off". Color me not impressed yet since there doesn't seem to be a release cut or even on the near horizon, this was just a bulk closure of all issues.

[BjarniRunar]

Sorry to disappoint! The bulk closer wouldn't let me add a reason. Or I couldn't figure out how.

Thanks for taking the time to complain though, it's great to have a reminder as to why I'm mostly staying out of the public eye this time around! :+1:

[alerque]

You have a project people (including myself) really want to be excited about, but you don't give them anything to hand onto, not even development status reports? Maybe if the dev work and even alpha releases / dev milestones towards a product were in view you might get some excitement and help building it.