cannot install via wheels on nvidia/cuda

[bertsky]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
• TensorFlow installed from (source or binary): binary (wheel via nvidia-pyindex)
• TensorFlow version: nvidia-tensorflow~=1.15.5+nv23.*
• Python version: Python 3.8
• Installed using: pip
• CUDA/cuDNN version: CUDA 11.3 and 12.0, cudnn8
• GPU model and memory: A40 46 GB

Describe the problem

In trying to find a suitable Docker base image supporting nvidia-tensorflow 1.15, I tried two versions of nvidia/cuda:

• nvidia/cuda:11.3.1-cudnn8-runtime-ubuntu20.04
• nvidia/cuda:12.0.1-cudnn8-runtime-ubuntu20.04

(I cannot just use nvidia/tensorflow images, because I also need to ship Pytorch and others in my image.)

They both complain:

ERROR: Could not find a version that satisfies the requirement tensorrt~=8.5 (from nvidia-tensorflow) (from versions: 0.0.1.dev5, 0.0.1)
ERROR: No matching distribution found for tensorrt~=8.5 (from nvidia-tensorflow)

Provide the exact sequence of commands / steps that you executed before running into the problem

docker run --gpus all --rm -it nvcr.io/nvidia/cuda:12.0.1-cudnn8-runtime-ubuntu20.04 bash # or nvidia/cuda:11.3.1-cudnn8-runtime-ubuntu20.04
apt-get update
apt-get install python3-pip
pip install nvidia-pyindex
pip install nvidia-tensorflow

Any other info / logs

I also tried to satisfy the tensorrt requirement via apt:

apt-get install tensorrt-dev

But because libcudnn8 and libnccl2 are marked as held packages, that will complain

The following packages have unmet dependencies:
 tensorrt-dev : Depends: libnvinfer-dev (= 8.6.0.12-1+cuda12.0) but it is not going to be installed
                Depends: libnvinfer-plugin-dev (= 8.6.0.12-1+cuda12.0) but it is not going to be installed
                Depends: libnvparsers-dev (= 8.6.0.12-1+cuda12.0) but it is not going to be installed
                Depends: libnvonnxparsers-dev (= 8.6.0.12-1+cuda12.0) but it is not going to be installed

[nluehr]

Try updating pip.

python3 -mpip install --upgrade pip

[bertsky]

python3 -m pip install --upgrade pip

wow, that did help!

I usually test this first, but in this case I turns out I also confused the system provided /usr/lib/python3/dist-packages/pip (v20) with the new one under /usr/local/lib/python3.8/dist-packages/pip (v23). After a hash -r, it did use the right version.

Much obliged!