search

qxmpp-project / qxmpp

Cross-platform C++ XMPP client and server library
412 stars 196 forks source link

Strange behaviour using qxmpp together with OpenSSL RSA decoding #328

Closed GoodOldGamer closed 3 years ago

GoodOldGamer commented 3 years ago

I'm not sure if this is a problem with qxmpp or a general problem of Qt.

I've got a strange behaviour when using qxmpp together with the RSA encoding/decoding from OpenSSL. Encoding with the private key works without problems. But after successfully decoding a RSA encoded string via public key decoding, QXmppClient throws an error and loses the connection:

error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01, error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed

Any idea why this is happening? The only connection I can see is that QSslSocket internally uses OpenSSL, too.

I pretty much followed this example for RSA encoding/decoding with OpenSSL: https://www.programmersought.com/article/65971451482/

Qt: 5.15.2 qxmpp: 1.3 OpenSSL: 1.1.1d

lnjX commented 3 years ago

QXmpp uses the QSslSocket for TLS encryption. QSslSocket uses the crypto library from your OS (which can be OpenSSL in some cases). The error however just looks like there's something wrong with your RSA code? Why do you think this is related to QXmpp?

For RSA encryption with Qt this might be interesting for you: https://github.com/QuasarApp/Qt-Secret

GoodOldGamer commented 3 years ago

As I wrote I'm not sure if this is related to qxmpp or if this is a problem with Qt/QSslSocket.

Sure, there may be something wrong with the RSA code, although encoding and decoding works fine with it. The thing that's confusing me is that I'm losing my XMPP connection after this error. Even if there are errors in the RSA encryption code, the socket from QXmppClient shouldn't get closed, right? Or is OpenSSL using some globals so that an error in any other OpenSSL dependant code affects QSslSocket?

I will take a look into Qt-Secret, thank you!

lnjX commented 3 years ago

Sure, there may be something wrong with the RSA code, although encoding and decoding works fine with it. The thing that's confusing me is that I'm losing my XMPP connection after this error.

Ahhh, okay. Sorry, I missed that. Unfortunately I still have no idea. QXmpp doesn't do anything special with the QSslSocket, so I guess the problem is in Qt (or even OpenSSL).

Or is OpenSSL using some globals so that an error in any other OpenSSL dependant code affects QSslSocket?

That could be possible, but I've got no idea how OpenSSL internally works. Sorry :(

GoodOldGamer commented 3 years ago

I've created a small example which produces the error. As soon as the OpenSSL code gets called in the connected slot, the client gets the error and closes it's connection. Without TLS enabled, the code is working.

I'm currently breaking down the connection code from QXmppOutgoingClient to get more information for Qt support about a potential bug in QSslSocket. But connecting to host and encryption via STARTTLS are working. Next is the SASL authentication, but that's already specific to XMPP ...

example.zip

GoodOldGamer commented 3 years ago

Ok, found my mistake. Used the wrong array length in decryption so that RSA_public_decrypt got an empty part handed over. And this seems to have affected the Socket.

So everything is fine with QXmpp. Ticket closed :)