When decrypted with a different private key, `OpenSSL::PKey::RSAError` does not occur

[seconoid]

Hi,

Is there a way to determine if the wrong key is being used when decrypting a string? In ruby 2.7, when a different key was used, it would raise OpenSSL::PKey::RSAError .

ruby 2.7.6 and OpenSSL 2.1.3

RUBY_VERSION #=> "2.7.6"
OpenSSL::VERSION #=> "2.1.3"
OpenSSL::OPENSSL_LIBRARY_VERSION #=> "OpenSSL 1.1.1n  15 Mar 2022"

correct_private_key = OpenSSL::PKey::RSA.new(correct_private_key.pem)
wrong_private_key = OpenSSL::PKey::RSA.new(wrong_private_key.pem)

str = "PLAIN TEXT"
encrypted_str = correct_private_key.public_encrypt(str)
correct_private_key.private_decrypt(encrypted_str) #=> "PLAIN TEXT"
wrong_private_key.private_decrypt(encrypted_str) #=> OpenSSL::PKey::RSAError: padding check failed

However, when upgrading to ruby 3, even if the wrong key is used, OpenSSL::PKey::RSAError no longer occurs.

ruby 3.2.3 and OpenSSL 3.1.0

RUBY_VERSION #=> "3.2.3"
OpenSSL::VERSION #=> "3.1.0"
OpenSSL::OPENSSL_LIBRARY_VERSION #=> "OpenSSL 3.2.1 30 Jan 2024"

correct_private_key = OpenSSL::PKey::RSA.new(correct_private_key.pem)
wrong_private_key = OpenSSL::PKey::RSA.new(wrong_private_key.pem)

str = "PLAIN TEXT"
encrypted_str = correct_private_key.public_encrypt(str)
correct_private_key.private_decrypt(encrypted_str) #=> "PLAIN TEXT"
wrong_private_key.private_decrypt(encrypted_str) #=> "\x96h\xE4\xFF\xFD?4|h-g!\xECu\xB1\xC3\x02;\f?\xD5-M\x9D\xF0\xF1\xEC\b\xE0\x9D\xA3ps\xE0\xE3\xDD\xCF\xDBG5\xE1\xDE\x89N\xA2\x97\x95\x00\xA75\xE0h\xF1\xF0&L\xF7\x85\ni\xC1\xD9\x91\xCB\xF4\a\xE1FpG\xF0\x18\x1C\xD2\xD8\x1A\xD6\xFD\xAA\xEF9\x95?\x10\xCA\xA5@\x8A\x88\x91n\x81q\xA5a\xD9`\x8D\xAD|\xB9\xFB\xEF[\xD6#\x94CiU\xC5\x13L\xEB\xCC\xC4\xAE\xD0\x90\ao\x12\xC1\xF5\xF4$s\x8E\x12\x8D\xDBL\xE4\xC5\xE9\x15\xD5\r<\xCE\xFD\x8B\xDC\x03\x93"
wrong_private_key.decrypt(encrypted_str)         #=> "\x96h\xE4\xFF\xFD?4|h-g!\xECu\xB1\xC3\x02;\f?\xD5-M\x9D\xF0\xF1\xEC\b\xE0\x9D\xA3ps\xE0\xE3\xDD\xCF\xDBG5\xE1\xDE\x89N\xA2\x97\x95\x00\xA75\xE0h\xF1\xF0&L\xF7\x85\ni\xC1\xD9\x91\xCB\xF4\a\xE1FpG\xF0\x18\x1C\xD2\xD8\x1A\xD6\xFD\xAA\xEF9\x95?\x10\xCA\xA5@\x8A\x88\x91n\x81q\xA5a\xD9`\x8D\xAD|\xB9\xFB\xEF[\xD6#\x94CiU\xC5\x13L\xEB\xCC\xC4\xAE\xD0\x90\ao\x12\xC1\xF5\xF4$s\x8E\x12\x8D\xDBL\xE4\xC5\xE9\x15\xD5\r<\xCE\xFD\x8B\xDC\x03\x93"

I'm not sure if this is a specification or a bug, but I would like to know if there is a way to determine it.

[rhenium]

This is a change in OpenSSL 3.2 (the C library, not ruby/openssl) to mitigate a timing attack with PKCS#1 v1.5 padding. It is also backported to older versions as a security fix in some distributions, such as Fedora or Ubuntu.

https://www.openssl.org/docs/man3.2/man3/EVP_PKEY_decrypt.html

You can use wrong_private_key.decrypt(encrypted_str, rsa_pkcs1_implicit_rejection: 0) if the old behavior is absolutely necessary. This option is documented in https://www.openssl.org/docs/man3.2/man1/openssl-pkeyutl.html

[seconoid]

thanks!