Add new option --rebuild (or --never-binary) to the install command

paugier commented 1 week ago

What's the problem this feature will solve?

I try to split issue https://github.com/pypa/pip/issues/12954 in smaller and more tractable issues.

As discussed in detailed in https://github.com/pypa/pip/issues/12954, the option --no-binary was used in particular for HPC to rebuild a package from source with different build arguments or different libraries (for example hdf5 sequential/parallel, or openMPI/MPICH).

However, since 23.1 (https://pip.pypa.io/en/stable/news/#v23-1), "--no-binary does not disable the cache of locally built wheels anymore. It only means “don’t download wheels”. (https://github.com/pypa/pip/issues/11453)".

As pointed out here, web documentation and supercomputer tutorials still use --no-binary for this purpose because of the old behavior (before 23.1) and the misleading name of this option (--no-binary only means “don’t download wheels”).

For this purpose ("reinstall a package from source"), one now needs to run things like:

pip cache remove h5py; pip install h5py --no-binary h5py --force
pip cache remove pyFFTW; pip install pyfftw --no-binary pyfftw --force
pip cache remove fluidfft_fftwmpi*; pip install fluidfft-fftwmpi --no-binary fluidfft-fftwmpi --force

Such commands are quite ugly, long and error prone. Note the "pyFFTW" and "fluidfft_fftwmpi", whereas the packages are named pyfftw and fluidfft-fftwmpi (https://github.com/pypa/pip/issues/13086).

Describe the solution you'd like

I propose to add a --rebuild option so that

pip install pyfftw --rebuild

(re)installs pyfftw from source (no wheel at all).

It seems to me that it would be convenient to be able to use --rebuild alone (like above, equivalent to pip install pyfftw --rebuild pyfftw) or as

pip install fluidsim[mpi] --rebuild mpi4py --rebuild pyfftw --rebuild h5py

like for --no-binary.

I think

pip install pyfftw mpi4py h5py --rebuild

should be equivalent to

pip install pyfftw mpi4py h5py --rebuild pyfftw --rebuild mpi4py --rebuild h5py

Alternative Solutions

Other name like --never-binary could be used.

Additional context

https://github.com/pypa/pip/issues/12954

Code of Conduct

[X] I agree to follow the PSF Code of Conduct.

sbidoul commented 5 days ago

Since the issue seems very much related to caching, I wonder if options to provide finer grained control of the cache of locally built wheels would address the use case too.

pfmoore commented 5 days ago

I agree with @sbidoul here. I would prefer to see this named in a way that makes it clear that it's about using the cache. So something like --no-build-cache PKG (with the usual :all: keyword to disable all build cache use) would work better for me.

The invocation would then be:

pip install fluidsim[mpi] --no-build-cache mpi4py --no-build-cache pyfftw --no-build-cache h5py

For me, --rebuild sounds too much like it would also rebuild an already-installed copy of the package.

pypa / pip