search

pytorch / builder

Continuous builder and binary build scripts for pytorch
BSD 2-Clause "Simplified" License
325 stars 219 forks source link

Check if CUDA GPG keys needs to be updated #1025

Open atalman opened 2 years ago

atalman commented 2 years ago

Nvidia is Updating the CUDA Linux GPG Repository Key Ref: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by the apt, dnf/yum, and zypper package managers beginning April 27, 2022.

If you don’t update your repository signing keys, expect package management errors when attempting to access or install packages from CUDA repositories.

TODO: Check if we use these keys anywhere within Pytorch/Builder and Domain Libraries packages and update the keys based on this document: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

sberryman commented 2 years ago

@atalman - I ran into this issue using pytorch/pytorch:1.11.0-cuda11.3-cudnn8-devel as my base image.

Error:

Sending build context to Docker daemon  451.6kB
Step 1/7 : FROM pytorch/pytorch:1.11.0-cuda11.3-cudnn8-devel
 ---> 730572d0c0dd
Step 2/7 : RUN apt-get update &&     apt-get install -y         libgl1-mesa-dev         libgtk2.0-dev         cmake
 ---> Running in e531e1cb9d82
Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease [1581 B]
Ign:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Get:3 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]
Get:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]
Get:5 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Get:6 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages [73.8 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [932 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2765 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1503 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [22.8 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [966 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2277 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3199 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [12.2 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [12.9 kB]
Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease' is not signed.

Here is the post from nvidia: https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772

Working with Containers CUDA applications built using older NGC base containers may contain outdated repository keys. If you build Docker containers using these images as a base and update the package manager or install additional NVIDIA packages as part of your Dockerfile, these commands may fail as they would on a non-container system. To work around this, integrate the above commands into the Dockerfile you use to build the container.

Existing containers in which the package manager is not used to install updates will not be affected by this key rotation.

atalman commented 2 years ago

HI @sberryman Thank you I could reproduce this with our 1.11 docker image . This issue can be mitigated by following these steps: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key

I am looking forward on fixing it for our 1.12 release

sberryman commented 2 years ago

Sure

FROM pytorch/pytorch:1.11.0-cuda11.3-cudnn8-devel

RUN apt-get update

Build: docker build -t testimage:latest -f Dockerfile .

Output:

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM pytorch/pytorch:1.11.0-cuda11.3-cudnn8-devel
 ---> 730572d0c0dd
Step 2/2 : RUN apt-get update
 ---> Running in e179c6764abf
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease [1581 B]
Ign:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Get:5 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]
Get:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]
Err:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages [73.8 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2798 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1512 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [957 kB]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [22.8 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [991 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2286 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3231 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [12.9 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [12.2 kB]
Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease' is not signed.
The command '/bin/sh -c apt-get update' returned a non-zero code: 100
sberryman commented 2 years ago

This is the way I've gotten around the build issue:

FROM pytorch/pytorch:1.11.0-cuda11.3-cudnn8-devel

# clean up old repo lists
RUN rm -f /etc/apt/sources.list.d/nvidia-ml.list /etc/apt/sources.list.d/cuda.list

# replace the keys
COPY cuda-keyring_1.0-1_all.deb /tmp/cuda-keyring_1.0-1_all.deb
RUN apt-key del 7fa2af80 && \
    apt-key del A4B46996 && \
    dpkg -i /tmp/cuda-keyring_1.0-1_all.deb

RUN apt-get update