search

NVIDIA / nvidia-container-toolkit

Build and run containers leveraging NVIDIA GPUs
Apache License 2.0
2.53k stars 275 forks source link

GPG error "public key is not available" in Ubuntu 20.04 CUDA 11.4.0 image while building #257

Open wilkesreid opened 2 years ago

wilkesreid commented 2 years ago

1. Issue or feature description

The following Dockerfile does not build today (April 28, 2022), even though it built successfully yesterday:

FROM    nvidia/cuda:11.4.0-runtime-ubuntu20.04
RUN     apt-get update

The error is the following:

GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is not signed.

2. Steps to reproduce the issue

Create the above dockerfile and attempt to docker build it.

3. Information to attach (optional if deemed irrelevant)

I am running Docker version 20.10.14, build a224086 on WSL 2 Ubuntu 20.04 on Windows 10 Pro, Version 21H2, OS Build 19044.1645

nvidia-smi on my host machine:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.65       Driver Version: 471.96       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:0A:00.0  On |                  N/A |
|  0%   55C    P0    61W / 250W |   1515MiB /  8192MiB |    ERR!      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
wilkesreid commented 2 years ago

Possible duplicate of https://github.com/NVIDIA/nvidia-container-toolkit/issues/258

klueska commented 2 years ago

https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212772

wilkesreid commented 2 years ago

The instructions in that notice do not work in the docker image.

FROM  nvidia/cuda:11.4.0-runtime-ubuntu20.04
...
RUN   apt-key del 7fa2af80
ADD   https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb .
RUN   dpkg -i cuda-keyring_1.0-1_all.deb
...
RUN   apt-get update

Results in

Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
memray commented 2 years ago

The instructions in that notice do not work in the docker image.

FROM  nvidia/cuda:11.4.0-runtime-ubuntu20.04
...
RUN   apt-key del 7fa2af80
ADD   https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb .
RUN   dpkg -i cuda-keyring_1.0-1_all.deb
...
RUN   apt-get update

Results in

Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=

Tried following the methods in announcement and didn't work either. Well done Nvidia.

memray commented 2 years ago

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

jtran1999 commented 2 years ago

I noticed different keys available to day for the repo.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda

zestrells commented 2 years ago

This is what I used in the dockerfile to fix this issue.

    && apt-key del 7fa2af80 \
    && curl -L -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb \
    && dpkg -i cuda-keyring_1.0-1_all.deb \
mirekphd commented 2 years ago

None of these hacks above are sufficiently reliable yet, as NVIDIA is still working on the changes. Some latest CUDA and Ubuntu versions are already working (images such as CUDA 11.6 for Ubuntu 20.04 can be rebuild from their code at Gitlab), but others (older CUDA/Ubuntu versions such as CUDA 11.2) may still fail.

So given that CUDA 11.1 .. 11.6 toolkits are compatible with the same drivers version (>=450.80.02) it should be possible to adapt NVIDIA's Dockerfile with the latest CUDA version (11.6) and your OS of choice: https://gitlab.com/nvidia/container-images/cuda/-/tree/master/dist/11.6.2

More info: https://gitlab.com/nvidia/container-images/cuda/-/issues/158

372046933 commented 2 years ago

nvidia/cuda:11.2.1-base-ubuntu20.04 is updated this afternoon. I pull the updated image and the pub key problem disappeared

kinue00 commented 2 years ago

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

seems working for me (kaldi's official dockerfile ubuntu1804)

VictorZuanazzi commented 2 years ago

We are having the same problem with the image nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04 and 11.3. Any overview of which images are fixed?

zjuPeco commented 2 years ago 
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

This works for me

Co0perator commented 2 years ago

This may add nothing productive to the conversation but I thought it worth mentioning Most people seem to be getting this error in Ubuntu 20.04 Docker containers, I'm getting it on Ubuntu 20.04 Desktop.

ghost commented 2 years ago 
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

This works for me

This worked for me as well.

eijynagai commented 2 years ago 
RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

This works for me

This didn't work for me. Using Ubuntu server 20.04 and docker image.

I got this error W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease' is not signed.

hammer-wang commented 2 years ago

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

Thanks for the magic!

zabir-nabil commented 2 years ago

RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list

worked like a charm

ramosmarco commented 2 years ago

I noticed different keys available to day for the repo.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" sudo apt-get update sudo apt-get -y install cuda

You're right, the nvidia instrucction (https://developer.nvidia.com/cuda-11.2.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=2004&target_type=debnetwork) set to add this key sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub however that doesn't match with the repository key, but using the one you mentioned (sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub), that it match, and allow to update apt-get.

Thanks

riponcse10 commented 2 years ago

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

Thanks! Works like a charm!

LordNex commented 2 years ago

having the same issue on a 4gb Jetson Nano

zanedurante commented 2 years ago

I found this issue searching up my problem on google (working with a GCP instance with nvidia GPU, no docker). Eventually following the steps here worked for me: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

aycaecemgul commented 2 years ago

I found this issue searching up my problem on google (working with a GCP instance with nvidia GPU, no docker). Eventually following the steps here worked for me: https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/

This is the only solution that worked for ubuntu 20.04 desktop.

StatsGary commented 1 year ago

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

This worked for me.

pxorig commented 1 year ago

try this: 

FROM nvidia/cuda:10.1-cudnn8-devel-ubuntu18.04
ENV LANG C.UTF-8
RUN  rm -rf /var/lib/apt/lists/* \
         /etc/apt/sources.list.d/cuda.list \
          /etc/apt/sources.list.d/nvidia-ml.list 
RUN apt-get update
ri-cao commented 1 year ago

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

This trick works for me. Thank you! My system is Ubuntu 22.04. The base singularity image was built on Ubuntu 18.04

PolarisLight commented 1 year ago

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

Thanks, this works for me.

karray commented 1 year ago

A workaround seems to do the trick for me. Add those lines before apt-get update

RUN rm /etc/apt/sources.list.d/cuda.list
RUN rm /etc/apt/sources.list.d/nvidia-ml.list

Also check out discussions here

I encountered a similar issue with the nvidia/cuda Docker container, but the specific files mentioned in the workaround were not present in my container. Instead, the relevant file in my case was /etc/apt/sources.list.d/cuda-ubuntu2204-x86_64.list.

Therefore, it is advisable to check the destination of the apt source list before attempting to delete them.

lehcode commented 8 months ago

Working Docker solution is as follows, also with persistence across services being built;

ADD "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin" "/tmp/cuda-ubuntu2004.pin"
ADD "https://developer.download.nvidia.com/compute/cuda/11.3.1/local_installers/cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.deb" "/tmp/cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.deb"
ADD "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub" "/etc/apt/7fa2af80.pub"

RUN cat /etc/apt/7fa2af80.pub | apt-key add - && \
    dpkg -i /tmp/cuda-repo-ubuntu2004-11-3-local_11.3.1-465.19.01-1_amd64.deb && \
    apt update && apt -y install cuda && \