remove pip cache before installing packages

[matthiasdiener]

The background to this PR is that @MTCam and me were debugging an mpi4py issue with Wyatt (which manifested itself in that mpi4py didn't load due to a missing .so) on Lassen. It turned out that there was an old wheel cache file of mpi4py in the cache in his home directory, which led to the impression that install.sh (and even a manual pip install mpi4py) was working well, while installing an old mpi4py package that was built incorrectly. We don't know where that cache file came from.

This seems to be a relatively common issue, I think I hit that as well some time ago and was struggling to fix it. It is aggravated by the facts that (1) the installation seems to go well, and (2) the not very useful error message (no amount of tinkering with the library path or removing/reinstalling emirge/mpi4py will fix this).

To fix this, we have a couple of options:

• purge the cache (like in this PR):
  • pro: fixes the issue for all packages, even potentially other packages than mpi4py
  • con: somewhat destructive, but only causes slightly higher build times on next installation
• purge only the mpi4py cache:
  • pro: less destructive
  • con: other packages might have a similar issue (pyopencl? islpy?)
• use --no-cache-dir argument to pip when running install.sh:
  • pro: non-destructive
  • con: still leaves a potentially non-working package on the system, guaranteeing future excitement ;-)

[inducer]

Thanks for explaining. Fair enough IMO. Could you

• add an interactive prompt along the lines of "hey, we recommend zapping your pip cache. We're about to do it. You OK with that? [y]"
• an option to preapprove that
• add a link to this PR in a few spots

? I'd be OK with this PR in that form.

[matthiasdiener]

A potentially less destructive way to solve this:

• before installing any non-dev (i.e., without --editable) pip package, remove the corresponding package from the cache

In this case, we probably wouldn't even need to prompt?

[inducer]

I'm OK with that (and with not prompting). But do make sure there is (noisy!) console output that states what's happening.

[matthiasdiener]

A potentially less destructive way to solve this:

• before installing any non-dev (i.e., without --editable) pip package, remove the corresponding package from the cache

In this case, we probably wouldn't even need to prompt?

I implemented this.