Open andersk opened 2 weeks ago
kou
commented
2 weeks ago Ah, https://apache.jfrog.io/artifactory/arrow/debian/apache-arrow-keyring.gpg is deprecated by apache-arrow-apt-source-latest*.deb and it's not updated.
You need to use https://downloads.apache.org/arrow/KEYS instead:
curl https://downloads.apache.org/arrow/KEYS | gpg --no-default-keyring --keyring /tmp/apache-arrow-apt-source.gpg --import -Similar issue: #40967
andersk
commented
2 weeks ago That doesn’t solve the issue. There are 16 distinct keys in that new keyring file, 5 of which are expired, so presumably it’s changing quite frequently. Based on the Wayback machine, the particular key in question was not in that keyring last year. Like I said, downloading new copies of the keyring every time it changes defeats the whole point of verifying PGP signatures, because an attacker who can tamper with the binary can also tamper with the keyring.
Zulip needs a stable root of trust for our external dependencies that can be baked into our release tarball. Ideally we would like it to be the one that’s already baked into our existing release tarball, so that our existing release tarball isn’t broken.
assignUser
commented
2 weeks ago Supply chain attacks are of course a valid concern but we can't avoid changes to the KEYS file, Arrow is a large project and individuals sign our releases, sometimes someone new takes on the responsibility (or new sub-projects are added) so changes are needed.
For apache/arrow releases there was one addition made in the last 2+ years and we added the key 6+ months before first using it to allow for the change to propagate. So it can't really be said that the file changes frequently. (there were a small number (2-3?) sub project additions unrelated to release singing for apache/arrow).
I think something that could make sense in the future is to announce changes to the KEYS file on the user mailing list with a hash of the new file, signed with an existing key from the file. This would provide a second channel to confirm the validity of any changes to the file. What do you think @kou?
In addition there was a change to the ASF release policy that now allows automatic release signing with a key not bound to an individual. This would further lower the frequency of changes to KEYS likely towards 0. But our releases process is quite involved and there are strict requirements for automatic signing to be implemented so this is not a change we can promise for the near future.
When we implement automatic release signing it might also be possible to additionally make use of https://github.blog/2024-05-02-introducing-artifact-attestations-now-in-public-beta/ to add another source of trust but I haven't looked into the requirements or uses cases for this so far.
If you have ideas (other than never changing KEYS :wink:) on how to improve this for downstream projects/users, contributions are of course always appreciated.
nikmolnar
commented
2 weeks ago This is also affecting apache-arrow-apt-source-latest-jammy.deb, and updating keys from https://downloads.apache.org/arrow/KEYS doesn't fix it.
kou
commented
2 weeks ago I think something that could make sense in the future is to announce changes to the KEYS file on the user mailing list with a hash of the new file, signed with an existing key from the file. This would provide a second channel to confirm the validity of any changes to the file. What do you think @kou?
It makes sense. Let's do it next time. We will do it on our release announce too.
kou
commented
2 weeks ago We may not be able to use apache-arrow-apt-source-lastest-XXX.deb{,.asc} with old https://downloads.apache.org/arrow/KEYS because apache-arrow-apt-source-lastest-XXX.deb.asc is generated with the latest https://downloads.apache.org/arrow/KEYS . The latest KEYS may include more keys that aren't included in an old KEYS.
How about the following instructions with KEYS ( https://apache.jfrog.io/artifactory/arrow/debian/apache-arrow-keyring.gpg is based on KEYS) that was added by https://github.com/zulip/zulip/commit/5ccb408f194ea54c6137f650384d6daa3173ceec (2023-07-28)?
.deb with the added KEYS.deb.deb with the KEYS included in the installed apache-arrow-apt-source_13.0.0-1.debapache-arrow-apt-source_15.0.0-1_all.deb{,.asc}FYI: We can detect available apache-arrow-apt-sources at https://apache.jfrog.io/ui/native/arrow/debian/pool/bullseye/main/a/apache-arrow-apt-source/ .
(We can update apache-arrow-apt-source by sudo apt update && sudo apt upgrade but if the old KEYS is too old, sudo apt update && sudo apt upgrade may not work.)
Describe the bug, including details regarding any error messages, version, and platform.
The PGP signature on the newly uploaded version of apache-arrow-apt-source-latest-bullseye.deb (and possibly others) fails verification, because it’s signed by a key that’s not in apache-arrow-keyring.gpg.
I would appreciate if this can be fixed by resigning the .deb with a key that’s already in the keyring, rather than adding new keys to the keyring, because we need to cut a new release of the Zulip server every time we need to change its included copy of the keyring. (Downloading new copies of the keyring every time it changes defeats the whole point of verifying PGP signatures, because an attacker who can tamper with the binary can also tamper with the keyring.)
Component(s)
Packaging