crypto: openssl: Pad DH key buffers with leading zeros for smaller keys

LibVNC / libvncserver

LibVNCServer/LibVNCClient are cross-platform C libraries that allow you to easily implement VNC server or client functionality in your program.

GNU General Public License v2.0

1.08k stars 483 forks source link

crypto: openssl: Pad DH key buffers with leading zeros for smaller keys #550

Closed gujjwal00 closed 1 year ago

gujjwal00 commented 1 year ago

Re: https://github.com/LibVNC/libvncserver/issues/493 Re: https://github.com/bk138/multivnc/issues/202

@bk138 , I would like to combine dh_generate_keypair() & dh_compute_shared_key() into a single function. We are doing a lot of unnecessary juggling by splitting these two.

DH & BIGNUM objects are allocated, freed in dh_generate_keypair(), and then reallocated again in dh_compute_shared_key()
Storing DH private key outside of DH object is not needed. We simply copy it out of dh_generate_keypair() and pass to dh_compute_shared_key()
Same situation is present in crypto_libgrypt.c

bk138 commented 1 year ago

Tested: works now when connecting MultiVNC for Android to UltraVNC w/ "New MS Logon" enabled. Thanks very much @gujjwal00!

Regarding your question: let's leave the code as-is for now, I'm currently quite limited w/ review time, so that would be an extra burden. I'm satisfied if the code works as-is, The Big Cleanup can happen after LibVNCServer 1.0.0.

gujjwal00 commented 1 year ago

Regarding your question: let's leave the code as-is for now, I'm currently quite limited w/ review time, so that would be an extra burden. I'm satisfied if the code works as-is, The Big Cleanup can happen after LibVNCServer 1.0.0.

No problem! I will try to help in at-least testing some of the PRs.