Ubuntu 22.04 repository not signed properly

[David-Deming]

What is the issue?

When attempting to install tailscale on ubuntu server 22.04 I am getting an error that the repository is not signed due to there being no public key. I checked /usr/share/keyrings/tailscale-archive-keyring.gpg and the key is there after running the commands to install Tailscale provided at https://tailscale.com/download/linux/ubuntu-2204

I also checked to make sure that /etc/apt/sources.list.d/tailscale.list had the proper verbiage to sign the repo and it is there and properly formatted.

Steps to reproduce

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/jammy.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/jammy.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list

sudo apt update sudo apt install tailscale

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Ubuntu server 22.04

Tailscale version

Unable to install any

Other software

No response

Bug report

No response

[awly]

@David-Deming could you share the output of all 4 commands you ran?

[vinrepos]

For me, I ran this command to install tailscale on a new Ubuntu instance. curl -fsSL https://tailscale.com/install.sh | sh

But the error is: E: The repository 'https://pkgs.tailscale.com/stable/ubuntu jammy InRelease' is not signed.

I believe this is the root cause:

[David-Deming]

Install commands:

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/jammy.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null

No output

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/jammy.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list

'# Tailscale packages for ubuntu jammy deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu jammy main'

Commands checking after install commands

ls /usr/share/keyrings | grep tail*

tailscale-archive-keyring.gpg

cat /etc/apt/sources.list.d/tailscale.list

'# Tailscale packages for ubuntu jammy deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu jammy main'

Attempting to install

apt update

Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease Get:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] Get:4 https://pkgs.tailscale.com/stable/ubuntu jammy InRelease Get:5 https://esm.ubuntu.com/cis/ubuntu jammy InRelease [4,575 B] Err:4 https://pkgs.tailscale.com/stable/ubuntu jammy InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 458CA832957F5868 Get:6 https://esm.ubuntu.com/apps/ubuntu jammy-apps-security InRelease [7,553 B] Hit:7 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease Get:8 https://esm.ubuntu.com/apps/ubuntu jammy-apps-updates InRelease [7,456 B] Get:9 https://esm.ubuntu.com/infra/ubuntu jammy-infra-security InRelease [7,450 B] Get:10 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] Get:11 https://esm.ubuntu.com/infra/ubuntu jammy-infra-updates InRelease [7,449 B] Get:12 http://us.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] Reading package lists... W: GPG error: https://pkgs.tailscale.com/stable/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 458CA832957F5868 E: The repository 'https://pkgs.tailscale.com/stable/ubuntu jammy InRelease' is not signed.

[awly]

@David-Deming interesting, I cannot reproduce this in an ubuntu:jammy container locally. What does this command output: gpg /usr/share/keyrings/tailscale-archive-keyring.gpg? You may need to apt install gpg first.

[awly]

@vinrepos that specific endpoint doesn't need to return anything for the repo to work. IIUC, apt update will fetch things like https://pkgs.tailscale.com/stable/ubuntu/dists/jammy/InRelease

[vinrepos]

Thanks @awly I had realised that but didn't get a chance to update my comment. Thanks for your comment.

In regards to the issue with keys, I saw that the official Ubuntu repos were also failing. After a bit of troubleshooting, I found out the shell path could be a problem. So I ensured /bin/sh is symlinked to /bin/bash (it was pointing to zsh before). I think I also had messed up with my keys so I re-imported it using commands in this answer. Afterwards, I logged into a new shell, did an apt clean and apt update and it then worked fine.

This is my lab environment so I didn't think about consequences. I wouldn't be doing this in prod without ensuring it works fine in a similar test environment first.

[David-Deming]

Sorry for the delay, I use tailscale for SSH so without it I have to physically access the server. The output is:

pub   rsa4096 2020-02-25 [SC]
      2596A99EAAB33821893C0A79458CA832957F5868
uid           Tailscale Inc. (Package repository signing key) <info@tailscale.com>
sub   rsa4096 2020-02-25 [E]

[awly]

@David-Deming that is very strange. I cannot reproduce this in neither an ubuntu:jammy container nor in a VM.

awly@ubuntu-jammy:~$ curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/jammy.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
[sudo] password for awly:

awly@ubuntu-jammy:~$ curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/jammy.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list
# Tailscale packages for ubuntu jammy
deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/ubuntu jammy main

awly@ubuntu-jammy:~$ sudo apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Get:5 https://pkgs.tailscale.com/stable/ubuntu jammy InRelease
Get:6 https://pkgs.tailscale.com/stable/ubuntu jammy/main amd64 Packages [10.6 kB]
Get:7 https://pkgs.tailscale.com/stable/ubuntu jammy/main all Packages [354 B]
Fetched 17.5 kB in 1s (22.5 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
19 packages can be upgraded. Run 'apt list --upgradable' to see them.

awly@ubuntu-jammy:~$ sudo apt install tailscale
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  tailscale-archive-keyring
The following NEW packages will be installed:
  tailscale tailscale-archive-keyring
0 upgraded, 2 newly installed, 0 to remove and 19 not upgraded.
Need to get 27.3 MB of archives.
After this operation, 50.6 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:2 https://pkgs.tailscale.com/stable/ubuntu jammy/main all tailscale-archive-keyring all 1.35.181 [3,082 B]
Get:1 https://pkgs.tailscale.com/stable/ubuntu jammy/main amd64 tailscale amd64 1.64.0 [27.3 MB]
Fetched 27.3 MB in 2s (12.0 MB/s)
Selecting previously unselected package tailscale.
(Reading database ... 74550 files and directories currently installed.)
Preparing to unpack .../tailscale_1.64.0_amd64.deb ...
Unpacking tailscale (1.64.0) ...
Selecting previously unselected package tailscale-archive-keyring.
Preparing to unpack .../tailscale-archive-keyring_1.35.181_all.deb ...
Unpacking tailscale-archive-keyring (1.35.181) ...
Setting up tailscale-archive-keyring (1.35.181) ...
Setting up tailscale (1.64.0) ...
Created symlink /etc/systemd/system/multi-user.target.wants/tailscaled.service → /lib/systemd/system/tailscaled.service.
Scanning processes...
Scanning linux images...

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

The error you get:

W: GPG error: https://pkgs.tailscale.com/stable/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 458CA832957F5868

indicates that apt is failing to find the public key (and not that the repo is not signed). Is this issue consistently reproducible on your end? If so, is it the same machine, or does it reproduce on multiple hosts? Did you do any other customization in /etc/apt/... that would prevent it from reading /usr/share/keyrings? And are you running apt update as root?

[David-Deming]

This is a weird one for sure. Yes, I am running update with root permissions (sudo). I have done some hardening to this install, but nothing that should prevent apt from reading /usr/share/keyrings. I checked to make sure there wasn't anything weird with permissions when I ran into the issue and everything was as it should be/is on my other machines. This is a fresh install, but this server was running 20.04 for a year and a half then 22.04 for 2 years and Tailscale functioned that entire time, so it shouldnt be any issue with the hardware itself. As far as consistently reproducible, yes, i have tried and failed handful of times, but only on that install, so I will attempt to reproduce on another machine tonight. For what its worth, I have other machines that run Tailscale on the same network and function just fine. I think I will try and actually run apt update as root, not just with sudo, to see if somehow that fixes it. Cant see why it would, but when nothing makes sense I might as well start trying everything.

[David-Deming]

Sorry for the delay. I could not replicate the issue anywhere with a fresh stock install until I did another install and hardened it to CIS level 2 server benchmark. Then I realized that the tailscale key in /usr/share/keyring had permissions of -rw-r-----. I did chmod 644 to match the perms of the other keys and then I could install Tailscale. I could have sworn I checked that on the previous install, but I am assuming I did not based on that being the change that fixed it.

The root cause was hardening the server, something in the process of remediating to CIS level 2 server made it so the permissions were altered when adding a key to /usr/share/keyring. I can't figure out what specific change caused it, but that is for sure the issue. A fresh install on the same machine had no problem. A fresh install that I hardened to CIS level 2 had the problem, changing the permissions on the key fixed it. Long story short, not an issue on your side at Tailscale. Sorry about that

[awly]

Thanks for the update, and good find!