Error 5007 reported occasionally during session handshake

Occasionally, error 5007 (LIBSSH2_ERROR_KEY_EXCHANGE_FAILURE) is reported during session handshake. This is somewhat reproducible when opening and closing sessions in quick succession (for example, when running integration tests).

The root cause of this failure is unknown. It appears to be an issue in either libssh2, WinCNG, or OpenSSH.

This is a debug log of the error, using libssh2 1.11.0: https://gist.github.com/logmanoriginal/3f785fbd7bd245c6ae2ded4efbce58f9

It indicates a failure to verify the server's hostkey signature:

[libssh2] 5843.285799 Key Ex: Server's MD5 Fingerprint: f2:9a:95:ba:15:5b:e8:83:80:ce:ee:ca:60:97:19:de
[libssh2] 5843.285799 Key Ex: Server's SHA1 Fingerprint: 85:61:d1:4f:68:7f:50:8d:fe:5e:21:08:70:f8:c8:fc:72:86:1d:41
[libssh2] 5843.285799 Key Ex: Server's SHA256 Fingerprint: 605osBjPZxbfbXjyol6DXYAmo9dmO5w2vYOD5m/oIPA=
[libssh2] 5843.296793 Key Ex: Failed hostkey sig_verify(): rsa-sha2-512: -1
[libssh2] 5843.297794 Failure Event: -11 - Unable to verify hostkey signature DH-SHA
[libssh2] 5843.297794 Failure Event: -8 - Unrecoverable error exchanging keys
[libssh2] 5843.297794 Failure Event: -8 - Unable to exchange encryption keys

For comparison, this is a debug log for a successful case: https://gist.github.com/logmanoriginal/26e3e0bc94151d30a9be7cc7b09df5d1

[libssh2] 11184.086917 Key Ex: Server's MD5 Fingerprint: f2:9a:95:ba:15:5b:e8:83:80:ce:ee:ca:60:97:19:de
[libssh2] 11184.086917 Key Ex: Server's SHA1 Fingerprint: 85:61:d1:4f:68:7f:50:8d:fe:5e:21:08:70:f8:c8:fc:72:86:1d:41
[libssh2] 11184.086917 Key Ex: Server's SHA256 Fingerprint: 605osBjPZxbfbXjyol6DXYAmo9dmO5w2vYOD5m/oIPA=
[libssh2] 11184.097940 Key Ex: Sending NEWKEYS message

In both cases, the rsa-sha2-512 algorithm is used (verified through the debugger).

The signature is verified using the BCryptVerifySignature function of the WinCNG backend: https://github.com/libssh2/libssh2/blob/1c3f1b7da588f2652260285529ec3c1f1125eb4e/src/wincng.c#L679-L680

In case of failure, the returned value is 0xC000A000 => "The cryptographic signature is invalid." The expected value is 0.

At this time I am at a loss as to what is causing this issue and put this here in the hopes that a solution will present itself.

Workaround Fortunately the issue is not persistent. Reconnecting to the server appears to work.

logmanoriginal / lvssh2

Error 5007 reported occasionally during session handshake #6