Adding PGP signature verification for Arch Linux packaging scripts (PKGBUILD)

[yan12125]

This is a continuation of discussions on Arch Linux User Repository (AUR). /cc @oconnor663 as requested.

In the current Arch Linux packaging script (PKGBUILD), detached signatures of Debian packages are not verified. If I want to check whether the downloaded file is really from Keybase, I need to download https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_1.0.36-20171122003226.f68b10ef8_amd64.deb.sig and https://keybase.io/docs/server_security/code_signing_key.asc manually. I propose to apply emlun's patch

diff --git a/PKGBUILD b/PKGBUILD
index 5fda4e1..56a3da9 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -18,10 +18,13 @@ depends=(fuse gconf libxss gtk2) # don't change this without changing the SRCINF
 conflicts=(keybase keybase-release keybase-git)
 source_i686=(
   "https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_${deb_pkgver}_i386.deb"
+  "https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_${deb_pkgver}_i386.deb.sig"
 )
 source_x86_64=(
   "https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_${deb_pkgver}_amd64.deb"
+  "https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_${deb_pkgver}_amd64.deb.sig"
 )
+validpgpkeys=(222B85B0F90BE2D24CFEB93F47484E50656D16C7)
 install=keybase.install

 package() {
@@ -43,5 +46,5 @@ package() {
 rm -rf "$pkgdir/etc/cron.daily"
 }

-sha256sums_i686=(994998f8093474330a1249e75963cacc948b8011ef5dff6e2e98085fa58448ba)
-sha256sums_x86_64=(35bd09c324828c125ecb68e83b3c54a36f8c48d61b09352593a62d64476f2802)
+sha256sums_i686=(994998f8093474330a1249e75963cacc948b8011ef5dff6e2e98085fa58448ba 'SKIP')
+sha256sums_x86_64=(35bd09c324828c125ecb68e83b3c54a36f8c48d61b09352593a62d64476f2802 'SKIP')

So that makepkg (Arch Linux's default package builder) can do pgp signature verification automatically.

Some existing arguments for not adding this check:

• Verifying SHA256 checksums is enough

First, integrity != authenticity. Second, technically I can't verify that the checksum is correct. For example, the checksum for the x86_64 package version 1.0.36-20171122003226.f68b10ef8 35bd09c324828c125ecb68e83b3c54a36f8c48d61b09352593a62d64476f2802 does not appear anywhere other than the AUR package according to Google and DuckDuckGo. As a result, I need to trust several things before trusting the checksum (1) The machine that runs update_aur_packages.sh (2) The AUR server aur.archlinux.org

On the other hand, trusting the PGP signature is much simpler. I can get the public key from https://keybase.io/docs/server_security/our_code_signing_key or public key servers (MIT, Ubuntu, ...), and there are many evidences that 222B 85B0 F90B E2D2 4CFE B93F 4748 4E50 656D 16C7 is related to Keybase.io. Unless all of them are compromised, I'm safe.

Other related arguments:

Would an evil PKGBUILD include the right Keybase PGP fingerprint

A user should always verify the fingerprint listed validpgpkeys before building a package.

do we expect those targeted users to realize that a verification step should have been there?

Users of the keybase client (or any security-related software in general) should know how trust of software packages is established. They are in danger if they don't know it. Actually, it's shocking for me that the keybase-bin package does not include validpgpkeys - I was expecting a nice error message from makepkg to tell me which key I should fetch and verify for this package.

• It will bother users to force the pgp verification check

As more and more security-related and general softwares on AUR come with the validpgpkeys directive, I believe AUR users are quite familiar with them.

---

I appreciated usability for the keybase client as one of the core spirits honored by the team. I'd like to thank you all for greatly improved UI in comparison to existing security-softwares. In this case, usability of newcomers conflicts with convenience of know-how users. I think keybase should give nice instructions for newcomers to protect themselves, such as detailed instructions as comments or printed with echo in PKGBUILD, pinned comments on AUR, and so on.

As a fallback option, if you really hesitate on adding signature verification, please provide SHA256 sums of Debian packages somewhere on the website. https://keybase.io/docs/the_app/install_linux appears to be a good place.

[oconnor663]

Thanks for the thoughtful comments. Tell me what you think of the following: It's true that the hash by itself doesn't authenticate the .deb file. However, the HTTPS connection you're downloading it over does authenticate it. In order to ship malicious bits, Keybase (or Amazon) would need to host them and aur.archlinux.org would need to tweak/delete the SHA256 checksum from the PKGBUILD for that one user.

As you pointed out, PGP signature checking inside the PKGBUILD could help solve that problem, but only if the user is extremely disciplined in auditing the PKGBUILD file every time they install or update. Here's an example attack: The AUR replaces the .sig file extension with .sig (containing "CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I" instead of "i"). Amazon arranges to re-host the signature file by that modified name. Then when makepkg downloads it and checks the extension, it doesn't believe it's a signature file at all, and it silently skips the signature checking step. The only indication that something was amiss would be that the "Verifying source file signatures with gpg..." line was missing from the makepkg output, and even that could be fudged if some part of the build script used fancy ANSI control codes to rewrite the terminal output above it.

I think a much more realistic solution would be to have a PGP signature around the PKGBUILD file, to prove that it's the one we wrote. I don't think that would integrate very well with pacaur or similar helpers, but maybe someone who knows the tools better than I do could suggest an ergonomic way to do it.

[oconnor663]

Apologies for how utterly undocumented this is, but you can find SHA2 checksums for our package files if you look in the Debian repo metadata:

curl https://prerelease.keybase.io/deb/dists/stable/main/binary-amd64/Packages

Package: keybase
Version: 1.0.36-20171122003226.f68b10ef8
Architecture: amd64
Installed-Size: 294276
Maintainer: Keybase Bugs <bugs@keybase.io>
Priority: optional
Section: misc
Filename: pool/main/k/keybase/keybase_1.0.36-20171122003226.f68b10ef8_amd64.deb
Size: 79772106
SHA256: 35bd09c324828c125ecb68e83b3c54a36f8c48d61b09352593a62d64476f2802
SHA1: 2dc2f070c33ceb93ed57d416c0017f25ad376de7
MD5sum: 3a0db30b6abb7e1af82bf5675c47b734
Description: Keybase's command line client
Depends: libappindicator1, fuse, libgconf-2-4

[yan12125]

TLS protocol authenticates a host instead of hosted contents. It only proves that the relevant host has the private key for the interested domain name. Also, it's possible that an evil/compromised CA issued a valid certificate for s3.amazonaws.com and perform MITM attack, just like previous incidents of WoSign. Before Certificate Transparency becomes the new standard, HTTPS is not so trustworthy.

Not sure how S3 works. I guess a malicious party can also create another S3 account (e.g., prerelease.keybаse.io, where 'а' is U+0430) and change the link in PKGBUILD as long as AUR is compromised.

So, you're correct - adding *.sig files and validpgpkeys is not enough. Signing the whole PKGBUILD can make things better, while there's a hole - keybase.install is not authenticated. Another idea comes to my mind: as each AUR package itself is a git repository, how about signing all commits? Then it's possible to verify that all packaging-related files, including PKGBUILD and keybase.install, are really from the Keybase team. This can be implemented as an optional feature in AUR helpers, so newcomers won't be bothered with makepkg errors, either.

I remember AUR doesn't allow git push --force. Signing from the next commit is fine as git SHA1 IDs depends on all previous commits IIRC.

curl https://prerelease.keybase.io/deb/dists/stable/main/binary-amd64/Packages

Many thanks for this link! It would be great to add a comment and mention this link in PKGBUILD.

[oconnor663]

Added a comment like you suggested: https://github.com/keybase/client/commit/f970d7daf99332f2512e03111b6f663ceabc5bb5

[yan12125]

Thanks! Now signed files/commits is less important. Still looking forward to it as an easier approach for software verification.

[eli-schwartz]

I too would like PGP signature verification, and as I said in the AUR comments, "Anyway, at the end of the day I tend to err on the side of more capable users..."

That being said, the sha256sums are provided by the presumably-trusted build server, and the PGP signatures are I believe also provided by the trusted build server. Anyone who can hack the build server to push malicious updates everywhere, would also have the signing key.

...

On the topic of modifying the PKGBUILD with unicode names, it should be noted that that will be obvious in git log -p in a way that updating the pkgver and sha256sums will not be. Same with modifying the install file, or doing fancy ANSI control codes. The protections offered by PGP neither help nor hinder users who don't read the PKGBUILD at all before running it, and users who do read the PKGBUILD will notice sudden new code introduced to the PKGBUILD and install script.

[oconnor663]

Yes, in practice if you get your hands on our AUR account SSH key, you probably also have our code signing key, so there isn't really anything we can do to save ourselves in that scenario. I do think it's interesting to ask how much damage a rogue AUR admin could do, if they decided to mess with our package. Of course for users who don't audit the PKGBUILD carefully, they can do whatever they want, so the question is rather how much can they get away with even against paranoid expert users. I think the answer is still "murder" :) Here are some possible attacks against someone who's auditing the git history:

• The attacker might make a unicode-similar-character change in the middle of a larger refactoring, so that the relevant lines are already showing up in the diff for another innocuous-looking reason. (Unless the user runs everything through a fishy-unicode-detecting lint.)
• They might hide arbitrary changes in a merge commit, which git log doesn't display. (Unless the user manually inspects merges with git show -m ... or similar.)
• They could insert corrupt blob objects into the repo, to make changes without changing the commit history at all. (Unless the user overrides the fsckObjects defaults in their git config.)

These attacks just get fancier and fancier. I think it's super interesting to think about how you can mitigate each one, but pretty unrealistic to expect real users (even paranoid real users) to actually do these things. That's why I'm hesitant to add any extra steps to the install that everyone has to go through. Still, I'm happy to make these sorts of things possible for expert users, if they want to take more steps.

[oconnor663]

A relevant article that came out today: https://www.twistlock.com/2017/12/13/hiding-content-git-escape-sequence-twistlock-labs-experiment

[yan12125]

Oh git is too complex to be used safely. How about maintaining a binary package repository just like core/extra/community? The whole pkg.tar.xz file is signed, so it's unlikely there's an evil bit. My package_builder.py may help.

[oconnor663]

@yan12125 the Debian repo metadata I mentioned above does get signed over by our PGP key, and that includes the hashes of our binary packages. Take a look at:

curl prerelease.keybase.io/deb/dists/stable/InRelease

Following all the hashes that are signed over there is a little bit annoying, as it's designed for Debian tooling more than for doing anything by hand. But the upside is that it should be pretty stable, if you want to script it. Our repo is laid out with the reprepro tool.

[oconnor663]

Also if you take a look at https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/index.html, you'll find per-binary signature files.

Edit: Or did you mean some kind of signed repo that pacman understands?

[yan12125]

Or did you mean some kind of signed repo that pacman understands?

Yes!

[oconnor663]

I guess we could edit /etc/pacman.conf or something like that? That would be basically the same as what we do for .deb and .rpm. But the AUR is so much simpler than all that :(

[eli-schwartz]

The AUR is so much simpler, than saying "in addition, you can add this snippet to pacman.conf and run pacman-key --recv-keys $keyid && pacman-key --lsign-key $keyid to add our custom repo, and forever after just pacman -Syu"?

[yan12125]

@eli-schwartz is so fast in replying. Let me add some more details for the same argument.

To use binary repos, users has to modify their /etc/pacman.conf and import the public package signing key, and the Keybase team needs to maintain a folder on the server, just like the case for Debian packages. The good news is, Arch Linux packaging is much simpler. A minimal binary repo contains only 1 database file (e.g., keybase.db) and 1 package file (e.g., keybase-1.2.3.pkg.tar.xz), along with signature files.

An advantage besides security is that there's no need for users to run the build step in PKGBUILD. Upgrading of keybase-bin occurs with usual system upgrades (e.g., pacman -Syu). Considering the update frequency of keybase-bin, switching to binary repo would save quite some time for users.

AUR is simpler for package maintainers yet complex for users. Git, PKGBUILD auditing, ... Those are all burdens and difficult to do it correct IMO.

[eli-schwartz]

Note that making a pacman-compatible package does imply having pacman installed so you can use makepkg. I know that Gentoo has a pacman package (which uses --with-root-dir="${EPREFIX}/var/chroot/archlinux" for obvious reasons), not sure how that would work in a Debian build container though.

You could use the following instead of makepkg I guess: https://gist.github.com/Earnestly/bebad057f40a662b5cc3 And repo-add is a standalone script so you could use it as-is.