Certificate verification failing when Digital signature and TLS1.3 enabled (IDFGH-14097)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

5.3.1

Espressif SoC revision.

ESP32-C3 QFN32(Rev v0.3)

Operating System used.

Windows

How did you build your project?

VS Code IDE

If you are using Windows, please specify command line type.

None

Development Kit.

ESP32-C3-MINI-1

Power Supply used.

USB

What is the expected behavior?

It looks like MBEDTLS_PK_RSA is the appropriate key type when digital signing peripheral is in use. But device chooses MBEDTLS_PK_RSASSA_PSS option and finally failing to sign/verify in the MBEDTLS_SSL_CLIENT_CERTIFICATE_VERIFY state.

It seems ssl_preset_default_sig_algs is containing MBEDTLS_TLS1_3_SIG_RSA_PSS_RSAE_SHA512 MBEDTLS_TLS1_3_SIG_RSA_PSS_RSAE_SHA384 and MBEDTLS_TLS1_3_SIG_RSA_PSS_RSAE_SHA256
which is not really supported by DS peripheral.

What is the actual behavior?

See the logs below for more details: The device is rejecting the certificate signature and sending alert to the server. I tried by disabling MBEDTLS_X509_RSASSA_PSS_SUPPORT macro from mbedtls_config.h. But the error is still happening.

Steps to reproduce.

Perform the TLS 1.3 handshake with certificates(RootCA and temporary cert).

Debug Logs.

Standard setting
----------------
I (8944) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls.c:4525 client state: MBEDTLS_SSL_CLIENT_CERTIFICATE_VERIFY

I (8954) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1132 => write certificate verify

I (8964) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1093 CertificateVerify signature failed with rsa_pss_rsae_sha256

I (8984) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1095 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

I (8994) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1093 CertificateVerify signature failed with rsa_pss_rsae_sha384

I (9004) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1095 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

I (9024) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1093 CertificateVerify signature failed with rsa_pss_rsae_sha512

I (9034) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1095 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

W (9044) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1112 no suitable signature algorithm

I (9064) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1150 <= write certificate verify

I (9074) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:5168 => send alert message

I (9084) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2943 => write record

I (9084) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:935 => encrypt buf

I (9094) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:1474 <= encrypt buf

I (9104) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2353 => flush output

I (9114) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2367 message length: 37, out_left: 37

I (9124) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2374 ssl->f_send() returned 37 (-0xffffffdb)

I (9134) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2401 <= flush output

I (9144) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:3080 <= write record

I (9154) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:5180 <= send alert message

I (9164) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls.c:4617 <= handshake

E (9174) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x6E00
I (9174) esp-tls-mbedtls: Certificate verified.
E (9184) esp-tls: Failed to open new connection
E (9194) transport_base: Failed to open a new connection
E (9194) HTTP_CLIENT: Connection failed, sock < 0

------------------------------------------------
MBEDTLS_X509_RSASSA_PSS_SUPPORT disabled setting
------------------------------------------------
I (8814) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls.c:4525 client state: MBEDTLS_SSL_CLIENT_CERTIFICATE_VERIFY

I (8824) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1140 => write certificate verify
I (8864) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1100 CertificateVerify signature failed with rsa_pss_rsae_sha256

I (8874) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1102 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

I (8944) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1100 CertificateVerify signature failed with rsa_pss_rsae_sha384

I (8954) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1102 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

I (9024) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1100 CertificateVerify signature failed with rsa_pss_rsae_sha512

I (9034) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1102 mbedtls_pk_sign_ext() returned -16128 (-0x3f00)

W (9094) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1120 no suitable signature algorithm

I (9104) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls13_generic.c:1158 <= write certificate verify

I (9114) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:5168 => send alert message

I (9124) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2943 => write record

I (9134) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:935 => encrypt buf

I (9144) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:1474 <= encrypt buf

I (9154) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2353 => flush output

I (9164) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2367 message length: 37, out_left: 37

I (9174) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2374 ssl->f_send() returned 37 (-0xffffffdb)

I (9184) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:2401 <= flush output

I (9194) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:3080 <= write record

I (9204) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_msg.c:5180 <= send alert message

I (9214) mbedtls: IDF\components\mbedtls\mbedtls\library\ssl_tls.c:4617 <= handshake

E (9214) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x6E00
I (9224) esp-tls-mbedtls: Certificate verified.
E (9234) esp-tls: Failed to open new connection
E (9234) transport_base: Failed to open a new connection
E (9244) HTTP_CLIENT: Connection failed, sock < 0

More Information.

No response

espressif / esp-idf