KA Lite fails to install on Raspbian since early Oct - piwheels URL 404s - why is "rm /etc/pip.conf" necessary?

[holta]

Everyone's KA Lite (0.17.3) are failing to install onto (updated) Raspbian Stretch / RPi3 suddenly since almost 4 weeks ago (it worked prior to early October). This is the 404 error that occurs:

https://www.piwheels.hostedpi.com/simple/importlib/

As a result of newly revised Raspbian package python-pip that installs the following /etc/pip.conf :

[global]
extra-index-url=https://www.piwheels.hostedpi.com/simple

We've lost hope that Raspbian will resolve this as this "python-pip" package showed it was in testing for most of a month, according to apt -a list python-pip :

python-pip/testing,now 9.0.1-2+rpt1 all [installed]

But now apt -a list python-pip shows it's "stable" — and yet we still have the failure to install KA Lite every day:

python-pip/stable,now 9.0.1-2+rpt1 all [installed]

So after almost a month of KA Lite failing to install as part of Internet-in-a-Box, we can no longer keep praying that Raspbian Stretch 2017-09-07's subsequent updates will magically fix this.

Is there another strategy we can works towards, in order to get KA Lite & piwheels to play well together?

Are the following 2 possibly relevant (or similar) towards KA Lite perhaps helping everyone to fix this properly?

https://www.piwheels.hostedpi.com/simple/importlib2/
https://www.piwheels.hostedpi.com/simple/backport_importlib/

Original bug description & full error message here: https://github.com/iiab/iiab/issues/392

Further context @ https://www.raspberrypi.org/forums/viewtopic.php?t=194886#p1227134 including Raspberry Pi community manager & piwheels creator Ben Nuttall (@bennuttall) who suggests we escalate to KA Lite towards fixing this properly? Or does it make sense for KA Lite to speak with him directly?

Far better than the current sloppy/destructive workaround for sure!! The claim that Aug 31's piwheels bug might be related...

Packages with . in their names are searched with -.

...is interesting but doesn't inspire confidence. We need forward progress getting KA Lite to install — given what appears to be a very simple 404 failure due to a bad URL — but yes for completeness Ben Nuttall asks us all to read https://github.com/bennuttall/piwheels/issues/20#issuecomment-340108171 FWIW.

Thanks @benjaoming & Thanks All for pointing us in the right direction!

[bennuttall]

It seems this bug was fixed in https://github.com/bennuttall/piwheels/issues/20#issuecomment-340108171 - it's not a bug in Raspbian, but a misconfiguration in the web server running piwheels, which has now been fixed.

When we soft-launched piwheels, we soon realised that there was an issue with pip replacing dots with hyphens when searching for dependencies, which messed up some installations, and we configured the web server to account for this, and the issue is now resolved. Now we're more confident piwheels works well, we've pushed ahead and built wheels for Python 3.4 and 3.5 (almost complete now).

Background: piwheels is a project to build wheels (pre-compiled packages) of everything on PyPI. Most packages are pure Python, but those requiring compilation take a long time to build on the Pi and no ARM platform wheels are available on PyPI. Now that they've all been built on a Pi 3, they're made available to Raspbian users via the new default pip config, speeding up installations.

pip in Raspbian Stretch is now configured to use piwheels as an additional index alongside PyPI. 404s in piwheels (i.e. a package we haven't successfully built) makes no difference to just installing from PyPI, because that's the fallback.

Run the command:

pip install importlib -v

for verbose output. You'll see that pip discovers two indexes to search:

2 location(s) to search for versions of importlib:
  * https://pypi.python.org/simple/importlib/
  * https://www.piwheels.hostedpi.com/simple/importlib/

then it searches both indexes for available files, and finds several from PyPI and none from piwheels. It then determines the latest version from all files available, and downloads importlib-1.0.4.zip. When running in a Python 3 environment, the installation fails. Since piwheels is Python 3 only at present, we do not have a build of importlib, which is Python 2 only.

Similarly, installing importlib2 (which is Python 3 compatible):

pip install importlib2 -v

It finds several files (source distributions) from PyPI and several files (bdist wheels) from piwheels. It chooses the latest wheel (importlib2-3.5.0.2-py3-none-any.whl) from piwheels as preference to sdist from PyPI. It installs no problem.

So... regarding this issue. As far as I can tell there is no issue, and no further work required. There's no bug in Raspbian nor in piwheels nor in any instructions. If someone can verify this, please do and post information in a reply.

Hope that helps - sorry for the confusion.

[holta]

@bennuttall thanks for the long explanation!

To be crystal clear, can you just clarify A or B here for everyone involved:

(A) https://www.piwheels.hostedpi.com/simple/importlib/ will work in future, and if so approximately when is service expected to be restored?

...or...

(B) This 404 URL (that still doesn't work today) is erroneous and/or deprecated and should NOT be relied upon?

(Thanks Much as we're all trying to understand what's at the bottom of this very serious ongoing failure!)

[bennuttall]

piwheels currently only builds for Python 3, so importlib does not exist in our index (because we don't have a successful build of it). The 404 is the same as if a package merely didn't exist on PyPI, e.g. https://pypi.python.org/pypi/blahblahblah

The fact piwheels is missing this package should not make the slightest difference to anyone trying to install importlib. Either they're trying to install it for Python 3 and it will fall back to PyPI and fail, or they're trying to install it for Python 2, it will not be found on piwheels, but found on PyPI from where it will be downloaded and installed no problem.

Either piwheels has a wheel, and the user gets an install faster, or it doesn't, and the user gets the sdist from PyPI, as before.

So really, it's (A). I can't give a date we'll add Python 2.7 support, as it's not a priority.

[benjaoming]

@holta some options:

a) pip install ka-lite-static instead, it bundles dependencies

b) Since you are running Raspbian, use the PPA/Debian pkgs instead of whls from piwheels/PyPi: https://launchpad.net/~learningequality/+archive/ubuntu/ka-lite

[benjaoming]

re: talk of Python 3 - KA Lite is not Python 3 compatible and there are no intentions of making it so, just to clear that up :) (but Kolibri is!)

[holta]

a) pip install ka-lite-static instead, it bundles dependencies

b) Since you are running Raspbian, use the PPA/Debian pkgs instead of whls from piwheels/PyPi: https://launchpad.net/~learningequality/+archive/ubuntu/ka-lite

c) "rm /etc/pip.conf" immediately after "apt install python-pip" is not looking so bad after all ?

Thanks @benjaoming and thanks @bennuttall for providing options: we have our Internet-in-a-Box team call today and this helps a ton working towards a longer-term fix!

(Even if yeah it remains a mystery why KA Lite 0.17.3 installed perfectly onto Raspbian Stretch prior to Oct ~10, when Raspbian imposed the new python-pip package, Ah Well :-)

[bennuttall]

re: talk of Python 3 - KA Lite is not Python 3 compatible and there are no intentions of making it so, just to clear that up :)

https://pythonclock.org/

[bennuttall]

(Even if yeah it remains a mystery why KA Lite 0.17.3 installed perfectly onto Raspbian Stretch prior to Oct ~10, when Raspbian imposed the new python-pip package, Ah Well :-)

I'm getting a MemoryError when trying to install it, even if I remove the pip config (so piwheels is out of the equation). Is something in pip using too much memory for the Pi? I had a similar issue recently trying to build something hefty (maybe matplotlib?) - I could only get it to work on a Raspbian Lite image (no desktop). Maybe something in pip 9 (which came with Stretch) introduced this?

[bennuttall]

Update: I had Chromium open when I tried the first time, and after closing it, pip install ka-lite-static worked fine.

This really is nothing to do with piwheels :)

[bennuttall]

Further update: pip install ka-lite also works fine. I think it was probably just a MemoryError. @holta can you try again, making sure you have nothing intensive running at the time? If you get an error, please paste it here.

[benjaoming]

https://pythonclock.org/

Ahahaha yes, I think in 2 years and 5 months, we have turned KA Lite into the sort of stable/conservative software that spacecrafts use, also spacecrafts might not be online. Should maybe put IIAB in a time capsule and send it to Mars to celebrate when our bugs are all fixed :D

Is something in pip using too much memory for the Pi?

Yes, this can happen. In this case maybe since Pip auto-creates wheels from tar.gz sdists during installation, and I think most of KA Lite's dependencies are bundled like this. I think you should still consider switching to ka-lite-static, because all these dependencies are bundled.

[holta]

Clarif: Internet-in-a-Box uses pip install ka-lite-static and always has.

Still, KA Lite's installation has not worked since Raspbian's Oct 10 introduction of a new python-pip, unless we first delete /etc/pip.conf

We still don't understand what a sane way forward is, to obviate this pattern of nightmares, accidentally inflicted on Africans far more than Martians.

We'll keep monitoring the situation. If it helps our Ansible code is as follows today:

- name: Install ka-lite with pip
  pip: name=ka-lite-static
       version={{ kalite_version }}
       virtualenv={{ kalite_venv }}
       virtualenv_site_packages=no
       extra_args="--no-cache-dir"
  when: internet_available and is_debuntu

Background on the virtual environment sandbox we use to avoid Python nightmares is:

kalite_version: "0.17.3"
kalite_venv: "/usr/local/kalite/venv"
kalite_program: "{{ kalite_venv }}/bin/kalite"

[bennuttall]

@holta please can you post the output from pip install ka-lite-static, or at least tell me what the error is? Because I can't reproduce it. Also tell me what kind of Pi you're using. Pi 1/2/3/Zero? Which model, etc?

[bennuttall]

Also I should add that when piwheels adds Python 2 wheels, the installation for this will be much faster.

[holta]

Most all installs occur on RPi 3. Output from pip install ka-lite-static is a bit mangled by Ansible 2.4.x but it is here: https://github.com/iiab/iiab/issues/392#issue-264554203

Also I should add that when piwheels adds Python 2 wheels, the installation for this will be much faster.

Great News: we've very conscious of that given that we live in an RPi 3 world (for grassroots fabricators who build these digital libraries) and RPi Zero W (for the masses, who then use these handheld digital libraries around the planet at schools, medical clinics and in their own homes...)

We'll have another meeting with our Internet-in-a-Box team Thursday or Monday to try to sort out the most sane+sustainable way forward. If there is confirmed blowback from our precautionary use of Python virtual environment (sandbox) we'll dig into that if it turns out we must~

Thanks again @bennuttall for all your help!

[benjaoming]

@holta

Clarif: Internet-in-a-Box uses pip install ka-lite-static and always has.

ka-lite-static has 0 (zero) dependencies apart from Python 2's stdlibs. For instance, importlib is bundled in kalite/packages/dist -- the problems encountered during installation seems to be because dependencies that aren't supposed to be installed in the virtualenv are installed directly through pip-packages/kalite.txt ?

[bennuttall]

Ok we discovered that there's a known bug in pip which breaks when additional indexes 404. The bug is still open in pypa/pip, and we can't expect a fix to find its way into Debian any time soon.

However, we (piwheels) have mitigated this by creating empty directories for packages we haven't successfully built, which means pip gets an empty list of files rather than a 404 (which is an enhandled exception causing the fatal error).

Hopefully that means that this is no longer an issue here.

[holta]

Thank you to @bennuttall & his piwheels team for creating stub site https://www.piwheels.hostedpi.com/simple/importlib/ which indeed solved the problem!