search

risacher / p11-capi

A pkcs11 implementation on top of win32 (or win64) capi
GNU Lesser General Public License v2.1
23 stars 5 forks source link

SSL_ERROR_SIGN_HASHES_FAILURE with TLS 1.2 on Win10 #7

Open schiermi opened 5 years ago

schiermi commented 5 years ago

I try to use p11capi_w64.dll with Windows 10 64 bit and Firefox 67.0.4. When the server requests the certificate the selection dialog pops up as configured. After selecting the users only available certificate firefox displays an SSL error page with the error code: SSL_ERROR_SIGN_HASHES_FAILURE.

Firefox selected cipher from servers available cipher list: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

schiermi commented 5 years ago

debug output: 00045753 57.82608032 [4892] S9: registered sesson id 00045754 57.82609940 [4892] C_OpenSession: 0 00045755 57.82611465 [4892] C_SignInit: enter
00045756 57.82613373 [4892] S9: found and locked session 00045757 57.82637405 [4892] copying container name 67ecd2b999046a689e7515264b0caa68_d766644d-7784-49a4-a71b-29cfc50bfb65 00045758 57.82640457 [4892] S9: unlocked session 00045759 57.82642365 [4892] C_SignInit: 0
00045760 57.82643509 [4892] C_GetAttributeValue: enter
00045761 57.82645416 [4892] S1: found and locked session 00045762 57.82647324 [4892] S1: unlocked session 00045763 57.82648468 [4892] C_GetAttributeValue: 0
00045764 57.82649612 [4892] C_Sign: enter
00045765 57.82651520 [4892] S9: found and locked session 00045766 57.82733154 [4892] Container name 67ecd2b999046a689e7515264b0caa68_d766644d-7784-49a4-a71b-29cfc50bfb65; provider name Microsoft Enhanced Cryptographic Provider v1.0
00045767 57.82738876 [4892] parse_rsa_pkcs_der_hash at ../src/p11-capi-rsa.c, line 88
00045768 57.82740402 [4892] n_input 51
00045769 57.82742691 [4892] input: 3031300D060960864801650304020105000420BB0B480D9A9F333B001E371149F4DF223EF0AED74357303635D9D27E016627F8
00045770 57.82744598 [4892] oid_str: 2.16.840.1.101.3.4.2.1
00045771 57.82884598 [4892] response form CertOIDToAlgId: 0xffffffff 00045772 57.82885742 [4892] forcing algorithm to CALG_SHA_256 (0x800c)
00045773 57.82950592 [4892] checkpoint at ../src/p11-capi-rsa.c, line 282
00045774 57.82951736 [4892] algorithm 0x800C 00045775 57.82954025 [4892] capifail 80090008 at ../src/p11-capi-rsa.c, line 327 00045776 57.82955933 [4892] checkpoint (ret = 32) at ../src/p11-capi-session.c, line 886 00045777 57.82957077 [4892] S9: unlocked session 00045778 57.82958984 [4892] C_Sign: 32
00045779 57.82960129 [4892] C_CloseSession: enter
00045780 57.82962036 [4892] S9: found and locked session 00045781 57.82963181 [4892] S9: removed session from list
00045782 57.82964325 [4892] S9: unlocked session 00045783 57.82968140 [4892] S9: destroyed

schiermi commented 5 years ago

It seems to me that PROV_RSA_FULL in Windows 10 is finally not capable/willing to handle CALG_SHA_256 due to error 80090008.

I found some C# code where the key associated to a PROV_RSA_FULL provider is re-used with PROV_RSA_AES: https://referencesource.microsoft.com/#System.IdentityModel/System/IdentityModel/Tokens/X509AsymmetricSecurityKey.cs,434

Unfortunately I'm not savvy enough to translate this into C.