Improve performance of Diffie-Hellman key exchange

[steerlink]

Resolve the following issues:

1. SSH connection stucked with NET::SSH_7.0.1 and OpenSSH_8.0 with FIPS enabled
2. Diffie-Hellman implementation causes key exchange to hang and consume CPU

Improve performance of Diffie-Hellman key exchange by generating a cryptographically strong random number instead of a probable prime. RFC 4419 does not require or suggest x (private key) be prime.

Background

• net-ssh, built on top of openssl, implements the diffie-hellman-group-exchange-sha256 key exchange and supports prime modulus in the range 1024 - 8192.
• Generating probable primes using Java's BigInteger class is highly variable and really slows down for primes with bit lengths > 2048.
• When a SSH server sends a prime modulus with a 8192 bit length, the jruby-openssl algorithm generates a BigInteger probable prime for x with the same bit length. Usually the server hangs up before the prime is available.
• RFC 4419 and RFC 4253 both indicate the private key is a random number, not a prime.
• Other implementations of the Diffie-Hellman key exchange algorithm don't generate primes for the private key. Here are a few examples:
  • Bouncy Castle
  • Go
  • Java
  • OpenSSL C

Benchmarks

☝️ Box plots (log scale) showing how long it takes to generate a probable prime of various bit lengths using the following constructor:

// https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/math/BigInteger.html#%3Cinit%3E(int,int,java.util.Random)
BigInteger(int bitLength, int certainty, Random rnd);

Hoping to get this in the next release of JRuby.

@kares @headius

[kares]

Thanks Sheldon, this is a very good catch!

You're right the prime requirement seems a bit off, looking at history this existed all the way back when the PKeyDH class was introduced: https://github.com/jruby/jruby-openssl/commit/659dd94fdda65041186f86893748a0850d5afb51

I do not see a reason to have it in but would like to spent some time reading the RFC. I was already planning to do a JOSSL release soon, this should definitely be included.

There's still one missing compat feature I'd like to look into, so it might take a week or few...