How to make tls handshake to invoke a custom signature method to do private key signature?

[xuyouming1986]

We have encounted a project in which out client program should do bidirectional tls handshake with server. Our program is integrated into a device which we can not access private key , but we have an interface to invoke to get private key signature result. The question is how to integrate this customized method in openssl to complete tls handshake progress? We haven't find a callback thing from openssl api，is there a simple way, any suggestion or example?

[mattcaswell]

In OpenSSL 1.1.1 and below you can use an ENGINE for this. In OpenSSL 3.0 you can write a provider.

[ljluestc]

#include <openssl/ssl.h>

// Callback function to obtain the PSK for client-side TLS
unsigned int psk_client_callback(SSL *ssl, const char *hint, char *identity,
                                 unsigned int max_identity_len,
                                 unsigned char *psk, unsigned int max_psk_len) {
    // Invoke your interface to get the private key signature result
    // Store the result in 'psk' buffer and return its length
    // Return 0 if the PSK cannot be obtained
}

int main() {
    // Initialize OpenSSL
    SSL_library_init();
    SSL_load_error_strings();
    ERR_load_BIO_strings();
    OpenSSL_add_ssl_algorithms();

    // Create an SSL context
    SSL_CTX *ctx = SSL_CTX_new(TLS_client_method());

    // Set the PSK client callback function
    SSL_CTX_set_psk_client_callback(ctx, psk_client_callback);

    // Perform TLS handshake
    // Your program will need to initiate the TLS handshake using SSL_connect()

    // Clean up OpenSSL
    SSL_CTX_free(ctx);
    ERR_free_strings();
    EVP_cleanup();
    CRYPTO_cleanup_all_ex_data();

    return 0;
}

[nhorman]

Marking as inactive, to be closed at the end of 3.4 dev, barring further input