search

libtom / libtomcrypt

LibTomCrypt is a fairly comprehensive, modular and portable cryptographic toolkit that provides developers with a vast array of well known published block ciphers, one-way hash functions, chaining modes, pseudo-random number generators, public key cryptography and a plethora of other routines.
https://www.libtom.net
Other
1.51k stars 449 forks source link

pkcs_1_pss_decode.c #638

Closed headscott closed 7 months ago

headscott commented 7 months ago

I encountered a problem in pkcs_1_pss_decode. It's located at src/pk/pkcs1/pkcs_1_pss_decode.c

While doing some changes for a TLS project I used eduardsui/tlse (from Github) and found a problem in parsing the certificate verify packet. eduardsui used this library for calculating values and to verify data from TLS packets. The parse method used your pkcs_1_pss_decode. Inside this method it fails in the following part:

for (x = 0; x < modulus_len - saltlen - hLen - 2; x++) { if (DB[x] != 0x00) { err = CRYPT_INVALID_PACKET; goto LBL_ERR; } }

and returns the error CRYPT_INVALID_PACKET. I am not sure exactly where the problem is, but I found out, that salt is allocated inside this method, but never used. There are just some comments, where salt is mentioned, but there is no usage. Could this be the problem? And if not, can you just remove these lines, where salt is allocated, set free and the memory set to zero?

Thanks in advance. F. T.

sjaeckel commented 7 months ago

Hi,

you're right, salt is allocated and never used, but that's not the problem.

Removing the verification that PS starts with 0x0 is absolutely no option.

Looking at the calling code and the TLS RFC I suspect that the salt length is wrong.

Here's a potential patch 0001-Fix-length-of-the-salt.patch.txt which could probably solve the issue, but I couldn't get tlssimple.c to connect to something since the certificates fail to load or I don't know what, so I couldn't test it. (Looks like I'm hitting https://github.com/eduardsui/tlse/issues/88 as well).

Please get this sorted out with @eduardsui

headscott commented 7 months ago

Hey,

I fixed these lines now. It doesn't fail anymore in the check for DB == 0x00, but it fails here:

if (XMEMCMP(mask, hash, hLen) == 0) { *res = 1; }

i am not sure if this is a problem of the TLS code or of the pkcs_1_pss_decode method. I would be very happy if you could help me here again. @eduardsui is not answering since months.

headscott commented 7 months ago

I found the solution btw. Thank you anyways. If you are interested for the solution, look at https://github.com/eduardsui/tlse/issues/89 There is my comment for that issue.