Closed patrickwasp closed 11 months ago
Thanks for reporting, I have the same issue. This means that all docker images for all EOL ROS Distro are impacted.
@nuclearsandwich @tfoote are you aware of this key expiration ?
This means that all docker images for all EOL ROS Distro are impacted.
indeed.
Came to report the same thing, but for ROS 1 docker images.
From one of my CI runs fi (for Melodic):
$ gpg --no-options --trust-model always --no-default-keyring --keyring /usr/share/keyrings/ros-archive-keyring.gpg --fingerprint
/usr/share/keyrings/ros-archive-keyring.gpg
-------------------------------------------
pub rsa3072 2018-11-19 [SC] [expired: 2023-11-19]
4B63 CF8F DE49 746E 98FA 01DD AD19 BAB3 CBF1 25EA
uid [ expired] ROS Snapshot builder <rosbuild@ros.org>
looks like the snapshot key has expired?
This also seems to impact noetic snapshots.
http://snapshots.ros.org/noetic/2022-07-27/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG AD19BAB3CBF125EA ROS Snapshot builder <rosbuild@ros.org>
The normal noetic repo for ubuntu does not seem to be affected.
Indeed :+1: the key is used to sign everything in the snapshots repository so every snapshot will have the problem
:+1: This is also afffecting Galactic.
@nuclearsandwich @tfoote are you aware of this key expiration ?
We are aware, confirmed. We hope to fix it during the following hours. Will ping you here when ready.
I do get the same error in 20.04-foxy image.
No idea why, but on ros:humble-ros-base-jammy
image that I just updated on my host, I only get the following warning (and it seems to me apt
is able to continue getting packages from ros.org, although I haven't checked whether those pkgs are the latest).
@130s That's because the expired gpg key is for snapshots.ros.org
and not packages.ros.org
- so this issue only affects EOL ROS distros.
Just want to clarify that this doesn't only affect EOL distros
this issue only affects EOL ROS distros.
This issue affects all users using snapshots: not only EOL distros.
Indeed 👍 the key is used to sign everything in the snapshots repository so every snapshot will have the problem
Most non-EOL users are likely using packages.ros.org but if non-EOL users are using snapshots.ros.org they will be affected by this issue. Using a non-EOL distro from snapshots.ros.org allows a little bit more control over incoming changes in base packages, to control when you do additional QA and the ability to rollback to a previous snapshot without the overhead of building everything from source or setting up a private mirrors etc.
@j-rivero thanks for looking into this quickly. Hope the fix is up soon
Indeed 👍 the key is used to sign everything in the snapshots repository so every snapshot will have the problem
I am seeing behavior where different "date-stamp" sources either exhibit the expired key or not.
For example:
I am using the same key for both sources:
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA \
&& echo "deb http://snapshots.ros.org/${ROS_DISTRO}/${ROS_SNAPSHOT_DATESTAMP}/ubuntu $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/ros2.list > /dev/null
@j-rivero thanks for looking into this quickly. Hope the fix is up soon
We have extended the life of the key to match the the ROS key updated in 2021, this means that should work until 2025 Jun. You can verify the new signature in (at least) the ubuntu keyserver.
For new ci-runs-without-cache/installations probably nothing different needs to be done. For systems that have installed the previous expired key, please update/re-fetch/re-download the same key. If you are using apt-key
like the images in this repo, this should be enough:
root@be9be22498a9:~# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
getting two new signatures:
Executing: /tmp/apt-key-gpghome.54Y6qAVNP2/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
gpg: key AD19BAB3CBF125EA: "ROS Spanshot builder <rosbuild@ros.org>" 2 new signatures
gpg: Total number processed: 1
gpg: new signatures: 2
Let me know if something is not working.
Works for me, thanks. But what will happen in 2025? Is there a solution to re-sign the packages in an internal apt repo mirror ?
But what will happen in 2025?
Hopefully the world would be a better place and the infra team will be able to setup a nice key rotation model that prevents this kind of problem to happen again :)
Didn't this exact thing happen with the other key(s) a couple of years ago?
The post-mortem is even linked in https://discourse.ros.org/t/again-snapshot-repo-gpg-key-expired/34733.
Thanks Jose!
@ruffsl this is going to be a challenge. we have a couple approaches but all a bit unsatisfactory
TL;DR by modifying the templates to force cache break and some hacking by hand for all images that we dont want to generate from template we can get something good for user and potantially acceptable for docker librarians. It will not be an exact equivalent package wise as the previously built and retired images
https://github.com/osrf/docker_templates/pull/108 https://github.com/osrf/docker_images/pull/698 (doesnt update pre-indigo images for now)
The goal is to make sure users of EOL images have an easy path to keep using these images and install ROS packages.
The most transparent way would be to:
@j-rivero thanks for looking into this quickly. Hope the fix is up soon
We have extended the life of the key to match the the ROS key updated in 2021, this means that should work until 2025 Jun. You can verify the new signature in (at least) the ubuntu keyserver.
For new ci-runs-without-cache/installations probably nothing different needs to be done. For systems that have installed the previous expired key, please update/re-fetch/re-download the same key. If you are using
apt-key
like the images in this repo, this should be enough:root@be9be22498a9:~# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
getting two new signatures:
Executing: /tmp/apt-key-gpghome.54Y6qAVNP2/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA gpg: key AD19BAB3CBF125EA: "ROS Spanshot builder <rosbuild@ros.org>" 2 new signatures gpg: Total number processed: 1 gpg: new signatures: 2
Let me know if something is not working.
I tryed this but nothing changed. (i'm using WSL) apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
The keys here seems to be ok but I still have the same problem.
@Gabrocecco It looks like you added the new key on your native system but are tryon to compile a docker image, and the docker image fails. For the key to be known to your docker image, you need to add it to your dockerfile. Can you try adding
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
before the step 4 of your dockerfile?
@Gabrocecco It looks like you added the new key on your native system but are tryon to compile a docker image, and the docker image fails. For the key to be known to your docker image, you need to add it to your dockerfile. Can you try adding
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
before the step 4 of your dockerfile?
It worked perfectley now! Thank you so much
@j-rivero thanks for looking into this quickly. Hope the fix is up soon
We have extended the life of the key to match the the ROS key updated in 2021, this means that should work until 2025 Jun. You can verify the new signature in (at least) the ubuntu keyserver.
For new ci-runs-without-cache/installations probably nothing different needs to be done. For systems that have installed the previous expired key, please update/re-fetch/re-download the same key. If you are using
apt-key
like the images in this repo, this should be enough:root@be9be22498a9:~# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
getting two new signatures:
Executing: /tmp/apt-key-gpghome.54Y6qAVNP2/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA gpg: key AD19BAB3CBF125EA: "ROS Spanshot builder <rosbuild@ros.org>" 2 new signatures gpg: Total number processed: 1 gpg: new signatures: 2
Let me know if something is not working.
For using it in docker I had to add inverted commas:
RUN apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
@Gabrocecco It looks like you added the new key on your native system but are tryon to compile a docker image, and the docker image fails. For the key to be known to your docker image, you need to add it to your dockerfile. Can you try adding
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA
before the step 4 of your dockerfile?
Works for me too. Thx!!!
Images on the official docker library and on the OSRF profile have been rebuilt.
The only images that couldnt be rebuilt are images based on unsupported debian base images:
If you dont use those you should be able to use the images from dockerhub without workaround.
If you still face issues don't hesitate to comment here and we can reopen
Hi, I tried the methods mentioned above but none of them works.
My docker file looks like
It is from this link https://github.com/carla-simulator/carla-autoware/blob/master/Dockerfile
Can somebody help?
The ROS part of this Dockerfile seems to run fine on my side.
Can you try building again without cahce (using the --no-cache
flag and post the entire output in text form ?
I'm getting errors but they seem to come from the nvidia repositories
PS: please provide text and not images as they are not searchable or copyable PS2: most likely you'll be better off filing issues in the repopsitory where the Dockerfile is located as you'll encounter issues out of the scope of the official docker images when trying to build that image
building results in:
Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy Docker version 24.0.7, build afdd53b Docker Compose version v2.21.0