search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
140 stars 273 forks source link

SHA-1 GPG keys are deprecated, request to update GPG-keys and RPM repository packages #3124

Closed Jounimk closed 1 year ago

Jounimk commented 1 year ago

Is your feature request related to a problem? Please describe

SHA-1 keys are deprecated and it's not possible to install OpenSearch packages from the repository to RHEL 9, which does not accept anymore SHA-1 signed RPMs packages by default (there are considered distrusted).

“Importing GPG key 0x9310D3FC: Userid : "OpenSearch project [opensearch@amazon.com](mailto:opensearch@amazon.com)" Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC From : https://artifacts.opensearch.org/publickeys/opensearch.pgp Is this ok [y/N]: y warning: Signature not supported. Hash algorithm SHA1 not available. Key import failed (code 2). Failing package is: opensearch-2.4.1-1.x86_64”

https://www.redhat.com/en/blog/rhel-security-sha-1-package-signatures-distrusted-rhel-9

Describe the solution you'd like

I would like that OpenSearch GPG keys would be updated to SHA-256 (or SHA-512) algoritm and RPM packages in the repository would be signed with this new key to make them RHEL 9 compatible by default.

Describe alternatives you've considered

No response

Additional context

No response

Acceptance Criteria

peterzhuamazon commented 1 year ago

Hi @Jounimk we have not officially support RHEL9 related distros yet. https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/#operating-system-compatibility

Tho this is a good call out and we probably need to just disable sha1 checksum in the build process. Would label this as a future improvement for now.

Also, @bbarani @setiah please chime in on your thoughts. Thanks.

peterzhuamazon commented 1 year ago

Interesting part is we already use sha512 for signing but seems like sha1 checksum is still somewhat being verified by the host on default so this needs more researching.

dblock commented 1 year ago

@peterzhuamazon where is this code? Maybe @Jounimk can try and contribute a fix?

Jounimk commented 1 year ago

@dblock This issue is not in the code itself and I'm afraid it cannot be mitigated in the RPM/Deb build process. Problem lies inside old opensearch GPG keypair which was done using SHA-1 hash. Therefore opensearch.pgp public key cannot be imported anymore into the RHEL9 (or similar distributions). So package installation fails
in the preliminary step where opensearch.pgp public key import fails. Whoever is reponsible of the opensearch GPG keys should generate a new key pair with (with GnuPG 2.3.3 version or newer : "gpg --full-generate-key"). Then all .deb and .rpm packages need to be signed with the new private GPG-key. This will solve the described issue.

peterzhuamazon commented 1 year ago

@dblock This issue is not in the code itself and I'm afraid it cannot be mitigated in the RPM/Deb build process. Problem lies inside old opensearch GPG keypair which was done using SHA-1 hash. Therefore opensearch.pgp public key cannot be imported anymore into the RHEL9 (or similar distributions). So package installation fails in the preliminary step where opensearch.pgp public key import fails. Whoever is reponsible of the opensearch GPG keys should generate a new key pair with (with GnuPG 2.3.3 version or newer : "gpg --full-generate-key"). Then all .deb and .rpm packages need to be signed with the new private GPG-key. This will solve the described issue.

Thanks @Jounimk this is a much bigger issue than I initially anticipated. The best window for this might be after the current subkey expire again:

This means we also need to post 2 public keys for people to verify, old artifacts vs new ones later (possibly)

Need thoughts from @bbarani @CEHENKLE @dblock as this is not extending the key into a new signing subkey, but completely generate a new master key.

All steps are in this issue:

Thanks.

prudhvigodithi commented 1 year ago

This info could be helpful https://old.nixaid.com/gpg-migration-sha1-to-sha2/ to migrate to sha2. Can we try this @peterzhuamazon ? Thanks

peterzhuamazon commented 1 year ago

This info could be helpful https://old.nixaid.com/gpg-migration-sha1-to-sha2/ to migrate to sha2. Can we try this @peterzhuamazon ? Thanks

Yes @prudhvigodithi after doing some research I think if we can just migrate from sha1 to sha2, it will be the best case.

dblock commented 1 year ago

Thanks @peterzhuamazon for the details! @bbarani I think someone will need to take this on your team.

prudhvigodithi commented 1 year ago

Updated the Acceptance Criteria section of the issue. Thank you

peterzhuamazon commented 1 year ago

It requires some research on the migration without completely revoking the key. The key will expire (public key) in May per the extension last year, we will trying to migrate the secret key from sha1 to sha2 before extending the public key again.

bbarani commented 1 year ago

We are targeting to migrate to SHA-2 from 2.8.0 version. CC: @peterzhuamazon @prudhvigodithi

bbarani commented 1 year ago

@peterzhuamazon @prudhvigodithi Lets do the analysis and report the findings soon.

peterzhuamazon commented 1 year ago

Starting to do the key migration now before expiration day 20230512.

peterzhuamazon commented 1 year ago

More resources: https://support.axway.com/kb/178853/language/en https://superuser.com/questions/547335/gpg-use-sha1-even-with-digest-algo-sha512 https://old.nixaid.com/gpg-migration-sha1-to-sha2/ https://lists.archive.carbon60.com/gnupg/users/89181 https://pthree.org/2015/11/19/your-gnupg-private-key/ https://github.com/drduh/config/blob/master/gpg.conf

peterzhuamazon commented 1 year ago

Test on a rockylinux9 server on this:

# gpg -vv --version
gpg (GnuPG) 2.3.3
libgcrypt 1.10.0-unknown
Copyright (C) 2021 Free Software Foundation, Inc.
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Pubkey: RSA (1), ELG (16), DSA (17), ECDH (18), ECDSA (19), EDDSA (22)
Cipher: IDEA (S1), 3DES (S2), CAST5 (S3), BLOWFISH (S4), AES (S7),
        AES192 (S8), AES256 (S9), TWOFISH (S10), CAMELLIA128 (S11),
        CAMELLIA192 (S12), CAMELLIA256 (S13)
AEAD: EAX (A1), OCB (A2)
Hash: SHA1 (H2), RIPEMD160 (H3), SHA256 (H8), SHA384 (H9), SHA512 (H10),
      SHA224 (H11)
Compression: Uncompressed (Z0), ZIP (Z1), ZLIB (Z2), BZIP2 (Z3)
peterzhuamazon commented 1 year ago

Old key confirm to have sha1 (hash / digest-algo 2)

:signature packet: algo 1, keyid 39D319879310D3FC
......
digest algo 2
......
algo: 3, SHA1 protection, hash: 2
peterzhuamazon commented 1 year ago

We will try to change the cipher from CAST5 to AES256 (S3 to S9) while change hash from sha1 to sha512 (H2 to H10) in line with the rpm signature internally signed within the package.

peterzhuamazon commented 1 year ago

Note that the change means both subprivate key and master private key need to have this change. This does not affect the detached gpg signature but also affect package signing like rpm, which is currently still using master private key to sign due to not all the rpm version support subkey signing.

peterzhuamazon commented 1 year ago

A related bug seems not able to let me force a sha512: https://dev.gnupg.org/T1800

peterzhuamazon commented 1 year ago

Use this gpg.conf:


personal-cipher-preferences AES256
personal-digest-preferences SHA512
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
cipher-algo AES256
digest-algo SHA512
cert-digest-algo SHA512
compress-algo ZLIB
disable-cipher-algo 3DES
#weak-digest SHA1
s2k-cipher-algo AES256
s2k-digest-algo SHA512
s2k-mode 3
s2k-count 65011712
peterzhuamazon commented 1 year ago

Trying to use a older version of gpg:


gpg (GnuPG) 2.2.20
libgcrypt 1.8.5
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Pubkey: RSA (1), ELG (16), DSA (17), ECDH (18), ECDSA (19), EDDSA (22)
Cipher: IDEA (S1), 3DES (S2), CAST5 (S3), BLOWFISH (S4), AES (S7),
        AES192 (S8), AES256 (S9), TWOFISH (S10), CAMELLIA128 (S11),
        CAMELLIA192 (S12), CAMELLIA256 (S13)
Hash: SHA1 (H2), RIPEMD160 (H3), SHA256 (H8), SHA384 (H9), SHA512 (H10),
      SHA224 (H11)
Compression: Uncompressed (Z0), ZIP (Z1), ZLIB (Z2), BZIP2 (Z3)
peterzhuamazon commented 1 year ago

Seems like it will not able to change the existing sha1 key to sha2, period, unless you recreate a key initially in sha2. This is kinda contradict from many of the post online claiming it is possible. Yet I only see the results similar to this one:

https://security.stackexchange.com/questions/259483/changing-the-encryption-algorithm-used-for-pgp-gpg-private-key

It will keep showing as this:

    iter+salt S2K, algo: 7, SHA1 protection, hash: 2,
peterzhuamazon commented 1 year ago

RH official post: https://www.redhat.com/en/blog/rhel-security-sha-1-package-signatures-distrusted-rhel-9

                                                                                                                                                                                        104 MB/s | 679 MB     00:06
OpenSearch 2.x                                                                                                                                                                                                                                                                               33 kB/s | 3.1 kB     00:00
Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <opensearch@amazon.com>"
 Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
 From       : https://artifacts.opensearch.org/publickeys/opensearch.pgp
Is this ok [y/N]: y
warning: Signature not supported. Hash algorithm SHA1 not available.
Key import failed (code 2). Failing package is: opensearch-2.7.0-1.x86_64
 GPG Keys are configured as: https://artifacts.opensearch.org/publickeys/opensearch.pgp
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED
peterzhuamazon commented 1 year ago

Seems like as long as the public key is in sha2 then rpm install will let it through

:signature packet: algo 1, keyid 39D319879310D3FC
    version 4, created <>, md5len <>, sigclass <>
    digest algo 10, begin of digest <>

Installed:
  opensearch-2.7.0-1.x86_64
# cat /etc/*release
NAME="Rocky Linux"
VERSION="9.1 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.1 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.1"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"
Rocky Linux release 9.1 (Blue Onyx)
Rocky Linux release 9.1 (Blue Onyx)
Rocky Linux release 9.1 (Blue Onyx)
peterzhuamazon commented 1 year ago

Use this config and only change hash to SHA512 (H10), this results in the public key generated having 2 signature packets of both H2 and H10, to suit for both older and newer requirements.

gpg (GnuPG) 2.2.20
libgcrypt 1.8.5
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Pubkey: RSA (1), ELG (16), DSA (17), ECDH (18), ECDSA (19), EDDSA (22)
Cipher: IDEA (S1), 3DES (S2), CAST5 (S3), BLOWFISH (S4), AES (S7),
        AES192 (S8), AES256 (S9), TWOFISH (S10), CAMELLIA128 (S11),
        CAMELLIA192 (S12), CAMELLIA256 (S13)
Hash: SHA1 (H2), RIPEMD160 (H3), SHA256 (H8), SHA384 (H9), SHA512 (H10),
      SHA224 (H11)
Compression: Uncompressed (Z0), ZIP (Z1), ZLIB (Z2), BZIP2 (Z3)
personal-digest-preferences SHA512
digest-algo SHA512
cert-digest-algo SHA512
s2k-digest-algo SHA512
s2k-mode 3
s2k-count 65011712

The cipher remained to be on CAST5 for now.

Expire in one year at about 20240512.

Test on Rockylinux9:

Old key:

Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <opensearch@amazon.com>"
 Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
 From       : https://artifacts.opensearch.org/publickeys/opensearch.pgp
Is this ok [y/N]: y
warning: Signature not supported. Hash algorithm SHA1 not available.
Key import failed (code 2). Failing package is: opensearch-2.4.0-1.x86_64
 GPG Keys are configured as: https://artifacts.opensearch.org/publickeys/opensearch.pgp
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED

New key: (IGNORE post install script issues as I am testing on docker without systemd now)

Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <opensearch@amazon.com>"
 Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
 From       : /testcerts/opensearch_20240512.pgp
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                    1/1
  Running scriptlet: opensearch-2.4.0-1.x86_64                                                                                                                                                                                                                                                                          1/1
  Installing       : opensearch-2.4.0-1.x86_64                                                                                                                                                                                                                                                                          1/1
  Running scriptlet: opensearch-2.4.0-1.x86_64                                                                                                                                                                                                                                                                          1/1
warning: %post(opensearch-2.4.0-1.x86_64) scriptlet failed, exit status 127

Error in POSTIN scriptlet in rpm package opensearch
  Verifying        : opensearch-2.4.0-1.x86_64                                                                                                                                                                                                                                                                          1/1

Installed:
  opensearch-2.4.0-1.x86_64
peterzhuamazon commented 1 year ago

To clear old repo gpg check key use:

sudo rm -rf /var/cache/dnf/* (or specific to the */pubring or similar)

To clear old rpm imported gpg key:

rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n'
rpm -e <keyid>
peterzhuamazon commented 1 year ago

Ubuntu 20.04 ok: (IGNORE post install script issues as I am testing on docker without systemd now)

# apt-key list
......
pub   rsa4096 2021-05-11 [SC]
      C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
uid           [ unknown] OpenSearch project <opensearch@amazon.com>
sub   rsa2048 2021-05-11 [S] [expires: 2024-05-12]
......

# apt update
Get:1 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable InRelease [7535 B]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages [928 B]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Fetched 344 kB in 1s (507 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

# apt install opensearch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  opensearch
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 748 MB of archives.
After this operation, 976 MB of additional disk space will be used.
Get:1 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 opensearch amd64 2.7.0 [748 MB]
Fetched 748 MB in 9s (84.4 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package opensearch.
(Reading database ... 7978 files and directories currently installed.)
Preparing to unpack .../opensearch_2.7.0_amd64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.7.0) ...
Setting up opensearch (2.7.0) ...
Running OpenSearch Post-Installation Script
/var/lib/dpkg/info/opensearch.postinst: line 56: systemd-tmpfiles: command not found
dpkg: error processing package opensearch (--configure):
 installed opensearch package post-installation script subprocess returned error exit status 127
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Errors were encountered while processing:
 opensearch
E: Sub-process /usr/bin/dpkg returned an error code (1)
peterzhuamazon commented 1 year ago

Try centos7 ok: (IGNORE errors as there is no systemd here)

                                                                                                                                                       | 679 MB  00:00:08
Retrieving key from file:///<>
Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <opensearch@amazon.com>"
 Fingerprint: c5b7 4989 65ef d1c2 924b a9d5 39d3 1987 9310 d3fc
 From       : <>
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Failed to get D-Bus connection: Operation not permitted
Failed to get D-Bus connection: Operation not permitted
  Installing : opensearch-2.7.0-1.x86_64                                                                                                                                                                                                                                                                                1/1
Failed to get D-Bus connection: Operation not permitted
warning: %post(opensearch-2.7.0-1.x86_64) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package opensearch-2.7.0-1.x86_64
  Verifying  : opensearch-2.7.0-1.x86_64                                                                                                                                                                                                                                                                                1/1

Installed:
  opensearch.x86_64 0:2.7.0-1
peterzhuamazon commented 1 year ago

Test verify detached signature on the oldest releases:

# gpg --verify opensearch-1.0.0-linux-x64.tar.gz.sig
gpg: assuming signed data in 'opensearch-1.0.0-linux-x64.tar.gz'
gpg: Signature made Mon Jul 12 19:22:11 2021 UTC
gpg:                using RSA key C2EE2AF6542C03B4
gpg: Good signature from "OpenSearch project <opensearch@amazon.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
     Subkey fingerprint: 2187 3199 B103 0FCD 49DA  83F8 C2EE 2AF6 542C 03B4
peterzhuamazon commented 1 year ago

Update this: https://github.com/opensearch-project/opensearch-build/issues/2040#issuecomment-1125449385

peterzhuamazon commented 1 year ago

We will publish the new key as part of this issue:

peterzhuamazon commented 1 year ago

We have released the new key now to public: https://opensearch.org/verify-signatures.html

Thanks.

peterzhuamazon commented 1 year ago

Is you try to install rpm without yum, but directly using the rpm package by rpm --import or similar, RHEL9 might not accept the new public key, tho yum accept.

You can always enable the legacy crypto: