PyPI Top 200 != WoS Top 200

[cclauss]

PyPI Top 200 != WoS Top 200

$ python
>>> import xmlrpclib
>>> xmlrpclib.ServerProxy('https://pypi.python.org/pypi').top_packages(200)

Or see: Moved to http://python3wos.mybluemix.net/ https://github.com/cclauss/python3wos_asyncio

The following 6 Python 2 modules are in the PyPI Top 200 but not on the WoS...

Module name                   Latest       Python 3?
===========                   ======       =========
bcdoc                         0.16.0       False  Miss
ipaddress                     1.0.16       False  Miss
moznetwork                    0.27         False  Miss
mozfile                       1.2          False  Miss
mozcrash                      0.17         False  Miss
functools32                   3.2.3-2      False  Miss

[ztane]

ipaddress and functools32 are backports of python 3 stdlib modules. Bcdoc seems to be an abandoned botocore-related tool. I wonder if all these are really genuine installs or is some CI inflating the numbers.

[ztane]

I am also thinking that all these moz* ones are useless, they're just mozilla build tools, no one really cares whether they use python 2 or 3 now.

[cclauss]

My sense is that:

1. The Top 200 is the Top 200... It does not matter if a CI tool or humans put a package there.
2. We can and should put equivalents (like simplejson has) in place for ipaddress and functools32.
3. We should let the Wall of Shame aspect of the tool do its work. If moz, Google, and other maintainers (like bcdoc) have tools in red, they will be more proactive about fixing them or removing them.
4. Given that we are at 87.5% Python 3 compliance, we should consider if we should move to Top 500 or not.

[ztane]

My sense is that the python3wos should be about shaming the actual things that actually need to be ported. Furthermore, the politics of PyPI are at fault. There are quite a many friendly forks there but people do not want to let others take over their dormant projects, or are not contactable, or something. That's why we have suds-jurko and others with other names.

[ztane]

btw, I looked at bcdoc and it has six among its requirements, I believe it supports python 3, only it doesn't have any troves at all.

[cclauss]

We can solve the suds --> suds-jurko issue with EQUIVALENTS (once they are working properly) https://github.com/ubershmekel/python3wos/blob/master/pypi_cron.py#L24

@ztane See https://github.com/boto/bcdoc/issues/39

[cclauss]

With EQUIVALENTS, we are now at 92% Python 3 compliant in the PyPI Top 200!! Three screenshots:

Removing all the green packages highlights just the laggards and automatically changes the page title...

This one shows a package which is not Python 3 compatible but has a fork that is. Also, two packages (futures and ssl) in the middle of the page that have been placed into the Python 3 standard library.

These screenshots were not done with the code that is currently in this repo but were instead done on a version that strips out the GAE, memcache, and database stuff and just focuses on building info the Top x PyPI packages on the fly. Using Python 3 asyncio, this code builds index.html form PyPI in about 45 seconds on my MacBook Pro and about 20 seconds on Bluemix!!

I would be happy to check these code changes here or I can create a fork.

[ubershmekel]

The top_packages rpc method was introduced 2 years after Python3Wos. I had to walk the entire package list at the time to get the top 200.

https://bitbucket.org/pypa/pypi/commits/738b70c967ecdf15470841a5af627de1ef33cfb1

Google App Engine supplying the hosting for free was great, but it also makes a lot of simple work error prone. Looking at what top_packages does - I think we can delete most of the database code in this project and just use that rpc call.

[cclauss]

Agreed. top_packages is a huge help. Also, Python 3 asyncio greatly speeds the individual package data gathering. I used Jinja2 as a plug replacement for GAE.template and it worked like a charm.

I am a bit perplexed what is the best way to proceed...

• Should we stay on Python 2 or shift to Python 3?
• Should we stay on GAE or shift to Bluemix, AWS, etc?
• Should we create a python3wos_async directory in the project and perfect things there, or is could this be done with a branch of the main project?

[ubershmekel]

We're at python version 3.5, with 178/200 packages supporting python 3. This project is nearing its end of life. I wouldn't try anything too bold. Just making it reliable in the least amount of work. I might have time for that in the next few weeks. There's going to be some migration from the DB because some of the equivalents aren't marked in the equivalents list and are just marked in the DB.

[cclauss]

@ubershmekel Please check out https://github.com/cclauss/python3wos_asyncio and let me know if you are interested to put some or all of these ideas into your version/repo. I think the removal of the database and memcache combined with the use of top_packages and asyncio makes the app much easier to deal with. Looking at the top 200 packages gives a 92% compliance rate but looking at the top 2000 packages drops the ratio close to the shame threshold. I do not think the project is done... I think WOS can continue help the Python community to focus effort on shifting popular packages to support Python 3 and on having proper trove classifiers on PyPI.

[cclauss]

The difference between https://python3wos.appspot.com and http://python3wos.mybluemix.net is now 6 packages: 178 vs. 184. The appspot version has not run in the past 10 days but it indicates that 89% of the top 200 packages (or their forks) are Python 3 compatible while the Bluemix version says that 92% are.

[ztane]

Yeah, mysql and suds alternatives are still missing from appspot.

[ubershmekel]

I believe I fixed these issues. Let me know if you find any more. Thank you for your help.