BadSignature when using P-256 (prime256v1) curve with sha512

[vehlwn]

Checklist

• [x] I've searched the issue tracker for similar requests: https://github.com/rustls/rustls/issues/1367

Is your feature request related to a problem? Please describe. I was very confused when my self signed certificate failed to verify with WebPkiServerVerifier when openssl verify command successfully worked.

Describe the solution you'd like Apparently rusttls 0.23.11 is lacking support for P-256 sha512 verification method witch should be supported by TLS 1.2 and 1.3 standards.

Describe alternatives you've considered Alternatively I can regenerate my certificates with sha256 or sha384. Or use openssl.

Additional context Here is a bash script I use to generate self signed CA authority and sign with it site certificate:

Spoiler warning

```bash #!/bin/bash readonly OUT_DIR=ssl readonly CURVE=prime256v1 readonly HASH=sha512 readonly SAN="DNS:localhost, IP:127.0.0.1" readonly CA_SUBJ="/CN=demo self signed CA" readonly SITE_SUBJ="/CN=demo" mkdir -p "${OUT_DIR}" function dump_cert() { openssl x509 -in "$1" -noout -text } function dump_req() { openssl req -in "$1" -noout -text } echo Generating CA... openssl req -x509 -newkey EC -pkeyopt ec_paramgen_curve:"${CURVE}" -"${HASH}" \ -out "${OUT_DIR}/ca.crt" -keyout "${OUT_DIR}/ca.key" -noenc \ -subj "${CA_SUBJ}" -days 365 \ -reqexts reqexts \ -config <(cat << EOF [reqexts] basicConstraints = critical, CA:true keyUsage = critical, digitalSignature, keyCertSign, cRLSign extendedKeyUsage = serverAuth, clientAuth EOF ) dump_cert "${OUT_DIR}/ca.crt" echo Generating site csr... openssl req -newkey EC -pkeyopt ec_paramgen_curve:"${CURVE}" -"${HASH}" \ -out "${OUT_DIR}/site.csr" -keyout "${OUT_DIR}/key.pem" -noenc \ -subj "${SITE_SUBJ}" dump_req "${OUT_DIR}/site.csr" echo Signing site csr... openssl x509 -req -in "${OUT_DIR}/site.csr" \ -CA "${OUT_DIR}/ca.crt" \ -CAkey "${OUT_DIR}/ca.key" \ -out "${OUT_DIR}/certificate.pem" \ -CAcreateserial \ -days 365 \ -"${HASH}" \ -extfile <(cat << EOF subjectAltName = ${SAN} basicConstraints = critical, CA:false keyUsage = critical, digitalSignature extendedKeyUsage = serverAuth, clientAuth EOF ) dump_cert "${OUT_DIR}/certificate.pem" cat "${OUT_DIR}/certificate.pem" "${OUT_DIR}/ca.crt" > "${OUT_DIR}/fullchain.pem" ```

Example output:

Spoiler warning

```bash Generating CA... ----- Certificate: Data: Version: 3 (0x2) Serial Number: 2d:72:4c:c3:4b:29:b6:bf:ae:6e:b2:73:7c:e7:db:63:ac:b7:2c:61 Signature Algorithm: ecdsa-with-SHA512 Issuer: CN=demo self signed CA Validity Not Before: Jul 25 06:24:48 2024 GMT Not After : Jul 25 06:24:48 2025 GMT Subject: CN=demo self signed CA Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:fa:a7:c0:05:c5:f0:81:0c:9f:aa:c3:0a:3c:26: f5:b6:63:01:28:cf:8e:2e:f6:9a:33:ae:62:9f:d4: 75:15:db:3f:60:9e:16:ba:fa:17:81:f3:3f:e2:3f: 9c:26:02:58:33:70:c4:77:00:0a:38:a0:48:ce:38: 8a:4c:e0:1e:0b ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Basic Constraints: critical CA:TRUE X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: 69:A2:F5:98:19:4A:D9:A2:25:BF:D8:ED:FC:0F:E3:A7:8E:27:75:1D Signature Algorithm: ecdsa-with-SHA512 Signature Value: 30:46:02:21:00:9d:4f:13:fd:fa:86:82:7e:27:a9:76:61:91: 03:3f:94:e1:23:ec:ff:6e:4c:e4:2d:fc:6b:55:4c:de:64:dd: 5e:02:21:00:c3:bf:56:a9:93:d2:bd:21:c4:d7:21:a2:89:41: e6:86:1c:c1:27:e2:22:8c:60:c3:4e:5b:b4:09:30:3f:87:fb Generating site csr... ----- Certificate Request: Data: Version: 1 (0x0) Subject: CN=demo Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:af:8b:c5:61:3b:67:77:48:18:ef:5b:36:49:a8: 0f:60:43:cf:18:a1:7c:8c:be:65:00:73:41:a5:8f: 52:56:38:f8:a2:e3:8e:c2:93:98:9c:fd:33:41:fa: 47:70:07:da:b0:1d:f8:a6:76:c9:9d:eb:df:41:a6: 5c:51:5d:71:42 ASN1 OID: prime256v1 NIST CURVE: P-256 Attributes: (none) Requested Extensions: Signature Algorithm: ecdsa-with-SHA512 Signature Value: 30:45:02:20:5a:55:70:b7:ba:99:aa:39:ff:bc:41:ce:32:1f: 1b:ca:ed:53:e5:ab:00:5a:18:4f:f6:6a:c9:14:17:6a:d8:76: 02:21:00:ac:28:b8:2b:ff:bb:8f:6b:71:17:5a:d7:00:86:76: 4f:0a:88:1e:8f:a0:aa:f1:b4:91:af:d5:81:60:6e:f2:23 Signing site csr... Certificate request self-signature ok subject=CN=demo Certificate: Data: Version: 3 (0x2) Serial Number: 70:82:c5:bf:88:75:3f:2d:7b:f6:83:cb:83:b2:17:3e:42:0a:32:ee Signature Algorithm: ecdsa-with-SHA512 Issuer: CN=demo self signed CA Validity Not Before: Jul 25 06:24:48 2024 GMT Not After : Jul 25 06:24:48 2025 GMT Subject: CN=demo Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:af:8b:c5:61:3b:67:77:48:18:ef:5b:36:49:a8: 0f:60:43:cf:18:a1:7c:8c:be:65:00:73:41:a5:8f: 52:56:38:f8:a2:e3:8e:c2:93:98:9c:fd:33:41:fa: 47:70:07:da:b0:1d:f8:a6:76:c9:9d:eb:df:41:a6: 5c:51:5d:71:42 ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Subject Alternative Name: DNS:localhost, IP Address:127.0.0.1 X509v3 Basic Constraints: critical CA:FALSE X509v3 Key Usage: critical Digital Signature X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: 38:F7:A9:93:DD:83:F0:59:75:5B:9A:0F:5E:68:1A:8E:C2:AD:FF:8A X509v3 Authority Key Identifier: 69:A2:F5:98:19:4A:D9:A2:25:BF:D8:ED:FC:0F:E3:A7:8E:27:75:1D Signature Algorithm: ecdsa-with-SHA512 Signature Value: 30:45:02:21:00:80:e9:18:77:7e:13:90:23:6a:94:8f:58:47: 08:bf:db:bb:2d:08:d9:41:f4:a5:4a:cc:82:14:81:c6:e5:f9: 19:02:20:7c:d6:24:b2:02:f6:df:e4:0f:1c:67:53:43:c4:8a: b4:24:3e:a4:99:e7:a3:3c:59:b4:1f:ba:35:6b:01:6e:f3 ```

You can verify signatures with:

openssl verify -CAfile ssl/ca.crt ssl/fullchain.pem

Failing rust code:

Spoiler warning

```rust use rustls::client::danger::ServerCertVerifier; use rustls::pki_types::CertificateDer; use rustls::pki_types::ServerName; use std::sync::Arc; pub fn load_root_ca(ca_file: &str) -> anyhow::Result { let mut cert_pool = rustls::RootCertStore::empty(); let certs = load_certificates(ca_file)?; let (added, _ignored) = cert_pool.add_parsable_certificates(certs); if added == 0 { anyhow::bail!("No valid certificats found in '{ca_file}'"); } return Ok(cert_pool); } fn load_certificates(path: &str) -> anyhow::Result>> { let mut reader = std::io::BufReader::new(std::fs::File::open(path).unwrap()); return Ok(rustls_pemfile::certs(&mut reader) .map_while(|x| match x { Ok(ok) => Some(ok), Err(e) => { log::error!("Failed to read certificate: {e}"); return None; } }) .collect()); } fn main() { rustls::crypto::ring::default_provider() .install_default() .expect("Failed to install rustls crypto provider"); let verifier = rustls::client::WebPkiServerVerifier::builder(Arc::new( load_root_ca("ssl/ca.crt").unwrap(), )) .build() .unwrap(); let chains = load_certificates("ssl/fullchain.pem").unwrap(); let (end_entity, intermediates) = chains.split_first().unwrap(); for server_name in ["localhost", "127.0.0.1"] { verifier .verify_server_cert( end_entity, intermediates, &ServerName::try_from(server_name).unwrap(), &[], rustls::pki_types::UnixTime::now(), ) .unwrap(); } } ```

It gives:

thread 'main' panicked at verify_server_cert called Result::unwrap() on an Err value: InvalidCertificate(BadSignature)

Example certificate files: Archive.zip

[ctz]

Apparently rusttls 0.23.11 is lacking support for P-256 sha512 verification method witch should be supported by TLS 1.2 and 1.3 standards.

Unfortunately I think you're wrong in that assertion for TLS 1.3. Here is the full list of standardised schemes: https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-signaturescheme

Note that there is no ecdsa_secp256r1_sha512. This enumeration controls both signatures in the TLS handshake and certificates, in practice some implementations are more accepting in the latter (outside of protocol negotiation).

I would recommend sticking to cabforum guidelines as this provides a good basis for expecting software support: https://cabforum.org/working-groups/server/baseline-requirements/requirements/#71322-ecdsa

[vehlwn]

Thanks. I misread that TLS 1.2 supports only the following signature algorithms:

• ecdsa_secp256r1_sha256;
• ecdsa_secp384r1_sha384;
• ecdsa_secp521r1_sha512.

Can be closed then.