Add universal wheel config

[aragilar]

This adds support for universal wheels, and updates the upload instructions.

[jgm]

Sorry, I have no ideal what universal wheels are. Can you explain the motivation for this change? I'm not really a Python programmer.

[aragilar]

Wheels are "compiled" python packages. If a package needs to have a c-extension, in a wheel, it has been compiled. If a package uses 2to3, 2to3 has already been run.

Universal wheels are for packages where the same code runs unmodified on both python 2 and 3.

Wheels have the advantage that they can install faster, and the metadata is (currently) more reliable than in sdists.

There's more information about where python packaging is at the moment at https://packaging.python.org/ (including why to avoid python setup.py upload).

[ickc]

Another suggestion is to setup Travis to deploy to PyPI automatically. This way, the notes in PACKAGING can be coded in Travis. (and the new dependency for uploading to PyPI can be installed in Travis.) Would @jgm want travis on this?

A downside is Travis is needed to setup before this. But since the examples/ has a very predictable structure, it can be used to write a simple form of test automatically, albeit not testing the pandocfilters.py directly. I could take a look into this if you want.

@aragilar, do you mind pointing me to examples on how to uses 2to3 with wheels? I want to improve the way I setup panflute's Travis deployment so that the pasteurize step is done only to the Python2 output. I think the setup would be very similar to the application of 2to3 to Python3 output. Thanks.

[aragilar]

The problem with getting Travis to deploy is Travis needs your PyPI details, which allows them to take any action on PyPI in your name. Whilst Travis is unlikely to actually do anything malicious, there's the potential that a security breach of Travis could result in someone using your PyPI account to upload malware, especially if the package is popular enough. Warehouse (the new PyPI codebase) had some planned improvements for the use case of uploading from a CI, but I don't think that's been implemented yet.

@ickc As far as I know, 2to3 is done automatically through setuptools (and maybe distutils?), so that's probably not of any help. The reason I know about 2to3 being useful with wheels is docutils uses it (as it still supports python 2.4), and it does make a noticeable difference to not call 2to3 on docutils when installing.

[ickc]

@aragilar, thanks for the info on 2to3. I did quite a bit of research into the opposite direction of 2to3, but always hit a deadend (I currently use pasteurize in panflute, which generate 2&3 compatible code from 3 only code. But then the Python3 users will also received a pasteurized version....). It seems ironic, because the adoption of Python3 actually depends more on the support of writing in Python3.

Regarding the security issue, do you have some refrences? Is there any weakness in the communication between Travis and PyPI's server? If it is just Travis in general, it seems quite a common practice to use Travis to deploy for you.

One potential issue with Travis that I know of is the way the password is encrypted. If one takes the easy route to setup through the web interface, any collaborators could make a pull request and tell travis to email the password to them. But this problem can be solved by using the travis cli, AFAIK.

[aragilar]

@ickc https://github.com/pypa/warehouse/issues/994 is the tracking issue for warehouse, there's probably other discussions of this. The main problem is unlike many other systems where there's a API key (which is usually per app, making revocation easy), PyPI uses a username/password with no 2FA to make sure you are you (which means if there's a problem it becomes more challenging to fix). This isn't Travis specific, any public CI system which could expose the details (e.g. run pull requests) is potentially unsafe. Basically, I don't see the advantage not having to type two commands (which you could put in a shell script/makefile/other stuff which you could automate the release with) vs problems with PyPI due to exposed password. Personally, I've found using devpi (I upload the new sdist to devpi on my laptop, test that it builds fine via pip install, pip wheel proj, upload wheel to devpi, push to PyPI) to make the entire thing painless, and makes releasing to PyPI no different than deciding to use a specific checkout of my project.

[ickc]

@aragilar, thanks for the great info.

Regarding the workflow to deploy to PyPI, letting a CI to deploy can simplifies things a lot. I suppose one can do CI locally before each deploy from a local machine. But if one setup a Travis CI, it means one would want to deploy only after the Travis CI build successfully (which potentially build across py versions or even difference OSes).

That said, back to the case of pandocfilters, setting up Travis just for the deployment will be overkill, especially if there's some potential pitfall. So it depends if we need the other advantages Travis can offer too (which is not related to this pull request per se). For example, a matrix can be set to see exactly which combinations of python version or pandoc version might make some of the examples fail.

[aragilar]

Was there anything needed for this to be merged?

[ickc]

I think this change is trivial, as long as @jgm don't mine using twine. If not, wheel can still be used by the previous upload method.

[jgm]

I haven't had time to explore all these packaging options, and since I may deprecate this package in favor of panflute anyway, I don't want to spend the time.

[ickc]

In this case I think if PACKAGING is changed to not using twine, the barrier of this pull request is lesser and may be easier to be accepted.

Regarding deprecation:

I see that some filters written in pandocfilters are kind of "abandonware". Deprecating pandocfilters might resulted in ending the life of all those filters (some are quite interesting and sophisticated meaning that it is difficult to port).

On the other hand, given that pandoc 2.0 probably would receives some major change in the AST (table colspan/rowspan, potentially also attributes granted for other elements, etc.), maintaining pandocfilters might be an extra burden. I have considered volunteering myself to maintain this, but am not sure about if I'll have enough time to define the new ASTs properly.

May be we should start asking around in pandoc discuss and see if there's enough interest to keep pandocfilters around? If not much people object to it, may be the time needed to maintain pandocfilters can be devoted instead to assisting people to migrate from pandocfilters to panflute. I think @sergiocorreia would be willing to write tutorials on porting it. And I'm also willing to help people writing filters in panflute.