snapshots.ros.org GPG key expired

patrickwasp commented 11 months ago

FROM ros:foxy

RUN apt-get update

building results in:

2.558 W: GPG error: http://snapshots.ros.org/foxy/final/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG AD19BAB3CBF125EA ROS Spanshot builder <rosbuild@ros.org>
2.558 E: The repository 'http://snapshots.ros.org/foxy/final/ubuntu focal InRelease' is not signed.
------
failed to solve: process "/bin/sh -c apt-get update" did not complete successfully: exit code: 100

Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy Docker version 24.0.7, build afdd53b Docker Compose version v2.21.0

mikaelarguedas commented 11 months ago

Thanks for reporting, I have the same issue. This means that all docker images for all EOL ROS Distro are impacted.

@nuclearsandwich @tfoote are you aware of this key expiration ?

gavanderhoorn commented 11 months ago

This means that all docker images for all EOL ROS Distro are impacted.

indeed.

Came to report the same thing, but for ROS 1 docker images.

From one of my CI runs fi (for Melodic):

$ gpg --no-options --trust-model always --no-default-keyring --keyring /usr/share/keyrings/ros-archive-keyring.gpg --fingerprint
  /usr/share/keyrings/ros-archive-keyring.gpg
  -------------------------------------------
  pub   rsa3072 2018-11-19 [SC] [expired: 2023-11-19]
        4B63 CF8F DE49 746E 98FA  01DD AD19 BAB3 CBF1 25EA
  uid           [ expired] ROS Snapshot builder <rosbuild@ros.org>

looks like the snapshot key has expired?

jobafr commented 11 months ago

This also seems to impact noetic snapshots.

http://snapshots.ros.org/noetic/2022-07-27/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG AD19BAB3CBF125EA ROS Snapshot builder <rosbuild@ros.org>

The normal noetic repo for ubuntu does not seem to be affected.

mikaelarguedas commented 11 months ago

Indeed :+1: the key is used to sign everything in the snapshots repository so every snapshot will have the problem

BrianOpteran commented 11 months ago

:+1: This is also afffecting Galactic.

j-rivero commented 11 months ago

@nuclearsandwich @tfoote are you aware of this key expiration ?

We are aware, confirmed. We hope to fix it during the following hours. Will ping you here when ready.

130s commented 11 months ago

I do get the same error in 20.04-foxy image.

No idea why, but on ros:humble-ros-base-jammy image that I just updated on my host, I only get the following warning (and it seems to me apt is able to continue getting packages from ros.org, although I haven't checked whether those pkgs are the latest).

Log

``` # apt update : Reading state information... Done 15 packages can be upgraded. Run 'apt list --upgradable' to see them. W: http://packages.ros.org/ros2/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. Reading package lists... Done Building dependency tree... Done Reading state information... Done ```

BrianOpteran commented 11 months ago

@130s That's because the expired gpg key is for snapshots.ros.org and not packages.ros.org - so this issue only affects EOL ROS distros.

moriarty commented 11 months ago

Just want to clarify that this doesn't only affect EOL distros

this issue only affects EOL ROS distros.

This issue affects all users using snapshots: not only EOL distros.

Indeed 👍 the key is used to sign everything in the snapshots repository so every snapshot will have the problem

Most non-EOL users are likely using packages.ros.org but if non-EOL users are using snapshots.ros.org they will be affected by this issue. Using a non-EOL distro from snapshots.ros.org allows a little bit more control over incoming changes in base packages, to control when you do additional QA and the ability to rollback to a previous snapshot without the overhead of building everything from source or setting up a private mirrors etc.

@j-rivero thanks for looking into this quickly. Hope the fix is up soon

dambrosio commented 11 months ago

Indeed 👍 the key is used to sign everything in the snapshots repository so every snapshot will have the problem

I am seeing behavior where different "date-stamp" sources either exhibit the expired key or not.

For example:

http://snapshots.ros.org/humble/2023-09-25/ fails with GPG key expired
http://snapshots.ros.org/humble/2023-10-11/ works

I am using the same key for both sources:

RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA \
    && echo "deb http://snapshots.ros.org/${ROS_DISTRO}/${ROS_SNAPSHOT_DATESTAMP}/ubuntu $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/ros2.list > /dev/null

j-rivero commented 11 months ago

@j-rivero thanks for looking into this quickly. Hope the fix is up soon

We have extended the life of the key to match the the ROS key updated in 2021, this means that should work until 2025 Jun. You can verify the new signature in (at least) the ubuntu keyserver.

For new ci-runs-without-cache/installations probably nothing different needs to be done. For systems that have installed the previous expired key, please update/re-fetch/re-download the same key. If you are using apt-key like the images in this repo, this should be enough:

root@be9be22498a9:~# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA

getting two new signatures:

Executing: /tmp/apt-key-gpghome.54Y6qAVNP2/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
gpg: key AD19BAB3CBF125EA: "ROS Spanshot builder <rosbuild@ros.org>" 2 new signatures
gpg: Total number processed: 1
gpg:         new signatures: 2

Let me know if something is not working.

romainreignier commented 11 months ago

Works for me, thanks. But what will happen in 2025? Is there a solution to re-sign the packages in an internal apt repo mirror ?

j-rivero commented 11 months ago

But what will happen in 2025?

Hopefully the world would be a better place and the infra team will be able to setup a nice key rotation model that prevents this kind of problem to happen again :)

gavanderhoorn commented 11 months ago

Didn't this exact thing happen with the other key(s) a couple of years ago?

The post-mortem is even linked in https://discourse.ros.org/t/again-snapshot-repo-gpg-key-expired/34733.

mikaelarguedas commented 11 months ago

Thanks Jose!

@ruffsl this is going to be a challenge. we have a couple approaches but all a bit unsatisfactory

TL;DR by modifying the templates to force cache break and some hacking by hand for all images that we dont want to generate from template we can get something good for user and potantially acceptable for docker librarians. It will not be an exact equivalent package wise as the previously built and retired images

https://github.com/osrf/docker_templates/pull/108 https://github.com/osrf/docker_images/pull/698 (doesnt update pre-indigo images for now)

The goal is to make sure users of EOL images have an easy path to keep using these images and install ROS packages.

The most transparent way would be to:

convince docker librarians to accept to built images based on base image not supported for a long time Challenges:
convince docker librarians to accept to built images based on base images not supported for a long time
invalidate the docker cache as the key is the same as before so even a rebuild of the images will not update the key in the resulting images
as the signature didnt change we don't have a nice way to invalidate the docker cache so even if they accept to rebuild the images the key will still not be the right one and resulting images will still have the issue
in some cases we may not be able to rebuild these images identically, example that comes to mind: EOL ros1-bridge images
- the ROS2 distro expired and we built an image at that time
- the ROS1 distro kept moving forward, if we build against the new package versions it will not be the same image, if we try to keep trhe old one we have to transition to snapshots repo for the ROS1 part and have the right plumbing to plug ourselves to the right snapshot (if it exists) as final may be too recent compared to when the ROS 2 distro was EOL'd

Gabrocecco commented 11 months ago

@j-rivero thanks for looking into this quickly. Hope the fix is up soon

We have extended the life of the key to match the the ROS key updated in 2021, this means that should work until 2025 Jun. You can verify the new signature in (at least) the ubuntu keyserver.

For new ci-runs-without-cache/installations probably nothing different needs to be done. For systems that have installed the previous expired key, please update/re-fetch/re-download the same key. If you are using apt-key like the images in this repo, this should be enough:
root@be9be22498a9:~# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
getting two new signatures:
Executing: /tmp/apt-key-gpghome.54Y6qAVNP2/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
gpg: key AD19BAB3CBF125EA: "ROS Spanshot builder <rosbuild@ros.org>" 2 new signatures
gpg: Total number processed: 1
gpg:         new signatures: 2
Let me know if something is not working.

I tryed this but nothing changed. (i'm using WSL) apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA

The keys here seems to be ok but I still have the same problem.

mikaelarguedas commented 11 months ago

@Gabrocecco It looks like you added the new key on your native system but are tryon to compile a docker image, and the docker image fails. For the key to be known to your docker image, you need to add it to your dockerfile. Can you try adding

RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA

before the step 4 of your dockerfile?

Gabrocecco commented 11 months ago

@Gabrocecco It looks like you added the new key on your native system but are tryon to compile a docker image, and the docker image fails. For the key to be known to your docker image, you need to add it to your dockerfile. Can you try adding
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
before the step 4 of your dockerfile?

It worked perfectley now! Thank you so much

EmmanuelMess commented 11 months ago

@j-rivero thanks for looking into this quickly. Hope the fix is up soon

We have extended the life of the key to match the the ROS key updated in 2021, this means that should work until 2025 Jun. You can verify the new signature in (at least) the ubuntu keyserver.

For new ci-runs-without-cache/installations probably nothing different needs to be done. For systems that have installed the previous expired key, please update/re-fetch/re-download the same key. If you are using apt-key like the images in this repo, this should be enough:
root@be9be22498a9:~# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
getting two new signatures:
Executing: /tmp/apt-key-gpghome.54Y6qAVNP2/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
gpg: key AD19BAB3CBF125EA: "ROS Spanshot builder <rosbuild@ros.org>" 2 new signatures
gpg: Total number processed: 1
gpg:         new signatures: 2
Let me know if something is not working.

For using it in docker I had to add inverted commas:

RUN apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA

mxshen commented 11 months ago

@Gabrocecco It looks like you added the new key on your native system but are tryon to compile a docker image, and the docker image fails. For the key to be known to your docker image, you need to add it to your dockerfile. Can you try adding
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
before the step 4 of your dockerfile?

Works for me too. Thx!!!

mikaelarguedas commented 10 months ago

Images on the official docker library and on the OSRF profile have been rebuilt.

The only images that couldnt be rebuilt are images based on unsupported debian base images:

kinetic / jessie
lunar / stretch
melodic / stretch
(on arm32 only) dashing-ros1-bridge

If you dont use those you should be able to use the images from dockerhub without workaround.

If you still face issues don't hesitate to comment here and we can reopen

super-zyw commented 10 months ago

Hi, I tried the methods mentioned above but none of them works.

My docker file looks like

It is from this link https://github.com/carla-simulator/carla-autoware/blob/master/Dockerfile

Can somebody help?

mikaelarguedas commented 10 months ago

The ROS part of this Dockerfile seems to run fine on my side. Can you try building again without cahce (using the --no-cache flag and post the entire output in text form ?

```dockerfile ARG AUTOWARE_VERSION=1.14.0-melodic-cuda FROM autoware/autoware:$AUTOWARE_VERSION WORKDIR /home/autoware # CARLA ROS Bridge # There is some kind of mismatch between the ROS debian packages installed in the Autoware image and # the latest ros-melodic-ackermann-msgs and ros-melodic-derived-objects-msgs packages. As a # workaround we use a snapshot of the ROS apt repository to install an older version of the required # packages. RUN sudo rm -f /etc/apt/sources.list.d/ros1-latest.list RUN sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA RUN sudo sh -c 'echo "deb http://snapshots.ros.org/melodic/2020-08-07/ubuntu $(lsb_release -sc) main" >> /etc/apt/sources.list.d/ros-snapshots.list' RUN sudo apt-get update ```

Output

``` > [6/6] RUN sudo apt-get update: 0.393 Get:1 http://snapshots.ros.org/melodic/2020-08-07/ubuntu bionic InRelease [13.8 kB] 0.402 Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 0.407 Get:3 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB] 0.454 Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease [1581 B] 0.528 Get:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 0.535 Get:6 http://snapshots.ros.org/melodic/2020-08-07/ubuntu bionic/main amd64 Packages [877 kB] 0.575 Get:7 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB] 0.638 Err:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease 0.638 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC 0.780 Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1637 kB] 0.839 Get:9 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB] 0.925 Get:10 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB] 0.929 Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB] 1.131 Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [23.8 kB] 1.132 Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [3373 kB] 1.202 Ign:14 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease 1.279 Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB] 1.281 Get:16 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release [564 B] 1.344 Get:17 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release.gpg [833 B] 1.462 Get:18 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Packages [73.8 kB] 1.775 Get:19 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1688 kB] 3.170 Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1728 kB] 3.323 Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2411 kB] 3.537 Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3785 kB] 3.905 Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [30.8 kB] 3.945 Get:24 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [20.6 kB] 3.987 Get:25 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [64.0 kB] 4.084 Reading package lists... 4.784 W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC 4.784 E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is not signed. ------ Dockerfile:15 -------------------- 13 | RUN sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA 14 | RUN sudo sh -c 'echo "deb http://snapshots.ros.org/melodic/2020-08-07/ubuntu $(lsb_release -sc) main" >> /etc/apt/sources.list.d/ros-snapshots.list' 15 | >>> RUN sudo apt-get update 16 | -------------------- ERROR: failed to solve: process "/bin/bash -o pipefail -c sudo apt-get update" did not complete successfully: exit code: 100 ```

I'm getting errors but they seem to come from the nvidia repositories

PS: please provide text and not images as they are not searchable or copyable PS2: most likely you'll be better off filing issues in the repopsitory where the Dockerfile is located as you'll encounter issues out of the scope of the official docker images when trying to build that image

osrf / docker_images

snapshots.ros.org GPG key expired #697