PyPI search broken

[powellchristoph]

Last night, it appears that PyPi might have made a change to their search functionality. Several packages that have been installed 8,700 times or more are suddenly breaking and it is because they are no longer being returned in the _search_package function.

It appears to me that packages that include uppercase characters are being downcased by pip. Pyshop cannot make a match because the package names are stored as camel-case and so pyshop searches PyPi for a match and then uses that to find the package. But since PyPi's search is broken, its not returning any results which mean pyshop returns package not found.

Case in point, Babel and MarkupSafe.

If you search for babel on PyPi, it returns no results on their webpage or in pyshop's search_results. But the package is listed in /simple and you can navigate to them.

I added some debug lines.

015-05-19 20:17:30,964 INFO  [pyshop.security][Dummy-2] [162.209.100.94] GET http://wheel.cldbckp.com/prod/simple/babel/
2015-05-19 20:17:31,140 INFO  [pyshop.views.base][Dummy-2] dispatch view Show
2015-05-19 20:17:31,143 INFO  [pyshop.views.simple][Dummy-2] refresh package babel
2015-05-19 20:17:31,262 DEBUG [pyshop.views.simple][Dummy-2] Found 17 packages
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Type: <type 'list'>
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.3.5', 'name': 'babel-cli', 'summary': 'A command line interface for sending anything anywhere.'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.7.1', 'name': 'babel-obviel', 'summary': 'Obviel Template message extractor for Babel'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.1', 'name': 'babelchart', 'summary': 'Pluggable timeseries data converter'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.5.4', 'name': 'babelfish', 'summary': 'A module to work with countries and languages'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.4.0', 'name': 'django-babel', 'summary': 'Utilities for using Babel in Django'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.3.0', 'name': 'django-babel-underscore', 'summary': 'Implements a underscore extractor for django-babel.'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.0', 'name': 'flask-babel-utclocal-utils', 'summary': 'UTC to local (and vice versa) datetime conversion utilities for use with Flask-Babel.'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.2.2', 'name': 'frasco-babel', 'summary': 'I18n and L10n support for Frasco'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1', 'name': 'gears-babel', 'summary': 'Gears compiler for Babel (previously 6to5) http://babeljs.io/'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.0', 'name': 'genibabel', 'summary': 'Genetic Imaging Babel (GenIBabel)'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '1.1.1', 'name': 'gpsbabel', 'summary': 'Python wrapper for GPSBabel project'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '2.0.0', 'name': 'nibabel', 'summary': 'Access a multitude of neuroimaging data formats'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '1.8.2', 'name': 'openbabel', 'summary': 'Python interface to the Open Babel chemistry library'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.3', 'name': 'pyifbabel', 'summary': 'A pure-Python implementation of the Treaty of Babel'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.0', 'name': 'radiobabel', 'summary': 'Interact with a number of online music services using a single unified API.'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.43', 'name': 'unbabel-py', 'summary': 'Python Wrapper around Unbabel HTTP API'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.3', 'name': 'webassets-babel', 'summary': 'Babel filter for Webassets'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found 17, matched 0
2015-05-19 20:17:31,264 INFO  [pyshop.views.simple][Dummy-2] package babel has no versions

Search for markupsafe, PyPi shows a match but Pyshop doesnt list any.

2015-05-19 20:17:51,616 INFO  [pyshop.security][Dummy-1] [162.209.100.94] GET http://wheel.cldbckp.com/prod/simple/markupsafe/
2015-05-19 20:17:51,794 INFO  [pyshop.views.base][Dummy-1] dispatch view Show
2015-05-19 20:17:51,798 INFO  [pyshop.views.simple][Dummy-1] refresh package markupsafe
2015-05-19 20:17:52,006 DEBUG [pyshop.views.simple][Dummy-1] Found 0 packages
2015-05-19 20:17:52,006 DEBUG [pyshop.views.simple][Dummy-1] Type: <type 'list'>
2015-05-19 20:17:52,006 INFO  [pyshop.views.simple][Dummy-1] package markupsafe has no versions

Here is the output for Babel yesterday. It finds the package and lists the versions.

934918 2015-05-18 22:47:07,730 INFO  [pyshop.security][Dummy-4] [162.209.108.184] GET http://wheel.cldbckp.com/prod/simple/babel/
934919 2015-05-18 22:47:07,907 INFO  [pyshop.views.base][Dummy-4] dispatch view Show
934920 2015-05-18 22:47:07,910 INFO  [pyshop.views.simple][Dummy-4] refresh package babel
934921 2015-05-18 22:47:08,336 DEBUG [pyshop.views.simple][Dummy-4] Found 29, matched 1
934922 2015-05-18 22:47:08,350 INFO  [pyshop.views.simple][Dummy-4] mirror package Babel now
934923 2015-05-18 22:47:08,368 INFO  [pyshop.views.simple][Dummy-4] Looking for non local user cmlenz
934924 2015-05-18 22:47:08,381 INFO  [pyshop.views.simple][Dummy-4] Looking for non local user asmodai
934925 2015-05-18 22:47:08,398 INFO  [pyshop.views.simple][Dummy-4] Looking for non local user asmodai
934926 2015-05-18 22:47:08,400 INFO  [pyshop.views.simple][Dummy-4] Looking for non local user mitsuhiko
934927 2015-05-18 22:47:08,408 DEBUG [pyshop.views.simple][Dummy-4] refreshing Babel package
934928 2015-05-18 22:47:08,411 INFO  [pyshop.views.simple][Dummy-4] No new version to mirror
934929 2015-05-18 22:47:08,411 DEBUG [pyshop.views.simple][Dummy-4] pypi versions: [u'1.3', u'1.2', u'1.1', u'1.0', u'0.9.6', u'0.9.5', u'0.9.4', u'0.9.3', u'0.9.2', u'0.9.1', u'0.9', u'0.8.1', u'0.8']
934930 2015-05-18 22:47:08,412 DEBUG [pyshop.views.simple][Dummy-4] mirrored versions: [u'1.3', u'1.2', u'1.1', u'1.0', u'0.9.6', u'0.9.5', u'0.9.4', u'0.9.3', u'0.9.2', u'0.9.1', u'0.9', u'0.8.1', u'0.       8']
934931 2015-05-18 22:47:08,412 INFO  [pyshop.views.simple][Dummy-4] package Babel mirrored
934932 2015-05-18 22:47:08,609 INFO  [pyshop.security][Dummy-3] [162.209.108.184] GET http://wheel.cldbckp.com/prod/repository/261/Babel-1.3.tar.gz

I would be happy to work on this with you as I need a fix asap. My CICD pipeline is down and therefore my shop until this is resolved.

[powellchristoph]

I will be opening a bug with PyPi also. Should package names be normalized in the database? Eg, downcased and '-' converted to '_' ?

[powellchristoph]

FYI, I found the commits to PyPi that broke the RPC search function. https://bitbucket.org/pypa/pypi/commits/43471d1062ac47bc57809c0280316503a6d2a00b

It appears that they pushed out using Eleasticsearch for their searching. https://bitbucket.org/pypa/pypi/commits/all

[powellchristoph]

https://bitbucket.org/pypa/pypi/issue/286/search-functionality-broken

[brmzkw]

should be fixed by https://github.com/mardiros/pyshop/pull/56

[powellchristoph]

Wow, thanks for the quick response!

[mardiros]

@powellchristoph

I have tested before applying the patch of @brmzkw but I don't see any problem (using postgresql)

Did you test the patch ? does it works for you ? Does PyPI has fixed the problem on their side ?

@brmzkw

I have applyied the patch a bit quick but don't see the point. What is the relation between the _search_package method and by_name ?

[powellchristoph]

PyPI has applied a fix. https://bitbucket.org/pypa/pypi/issue/286/search-functionality-broken

I will test the patch shortly.

[mardiros]

OK thanks for the feedback

[brmzkw]

What is the relation between the _search_package method and by_name

I thought it was related. My bad, my fix fixed something else :)