tls does not work

[runner365]

Hi, I try to us wss, but it doesn't work.

X509* rtc_dtls::s_certificate    = nullptr;
EVP_PKEY* rtc_dtls::s_privatekey = nullptr;
SSL_CTX* rtc_dtls::s_ssl_ctx     = nullptr;

void rtc_dtls::dtls_init(const std::string& key_file, const std::string& cert_file) {
    FILE* fh = fopen(cert_file.c_str(), "r");
    if (!fh) {
        MS_THROW_ERROR("open cert file(%s) error", cert_file.c_str());
    }

    s_certificate = PEM_read_X509(fh, nullptr, nullptr, nullptr);
    if (!s_certificate) {
        MS_THROW_ERROR("read certificate file error");
    }

    fclose(fh);

    fh = fopen(key_file.c_str(), "r");
    if (!fh) {
        MS_THROW_ERROR("open key file(%s) error", key_file.c_str());
    }

    s_privatekey = PEM_read_PrivateKey(fh, nullptr, nullptr, nullptr);
    if (!s_privatekey) {
        MS_THROW_ERROR("read key file error");
    }

    fclose(fh);

    s_ssl_ctx = SSL_CTX_new(DTLS_method());
    if (!s_ssl_ctx) {
        MS_THROW_ERROR("create ssl context error");
    }

    int ret = SSL_CTX_use_certificate(s_ssl_ctx, s_certificate);
    if (!ret) {
        MS_THROW_ERROR("use certificate error");
    }

    ret = SSL_CTX_use_PrivateKey(s_ssl_ctx, s_privatekey);
    if (!ret) {
        MS_THROW_ERROR("use privatekey error");
    }

    ret = SSL_CTX_check_private_key(s_ssl_ctx);
    if (!ret) {
        MS_THROW_ERROR("check privatekey error");
    }

    // Set options.
    SSL_CTX_set_options(s_ssl_ctx,
                    SSL_OP_CIPHER_SERVER_PREFERENCE | SSL_OP_NO_TICKET |
                    SSL_OP_SINGLE_ECDH_USE | SSL_OP_NO_QUERY_MTU);

    SSL_CTX_set_session_cache_mode(s_ssl_ctx, SSL_SESS_CACHE_OFF);

    SSL_CTX_set_read_ahead(s_ssl_ctx, 1);

    SSL_CTX_set_verify_depth(s_ssl_ctx, 4);

    SSL_CTX_set_verify(s_ssl_ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT,
                    on_ssl_certificate_verify);

    SSL_CTX_set_info_callback(s_ssl_ctx, ssl_info_callback);

    ret = SSL_CTX_set_cipher_list(s_ssl_ctx, "DEFAULT:!NULL:!aNULL:!SHA256:!SHA384:!aECDH:!AESGCM+AES256:!aPSK");
    if (!ret) {
        MS_THROW_ERROR("set cipher list error");
    }

    SSL_CTX_set_ecdh_auto(s_ssl_ctx, 1);

    int index = 0;
    std::string dtls_srtp_crypto;
    for (auto item : srtp_crypto_suite_vec) {
        if (index != 0) {
            dtls_srtp_crypto += ":";
        }

        dtls_srtp_crypto += item.name;
        index++;
    }

    log_infof("dtls srtp crypto: %s", dtls_srtp_crypto.c_str());

    ret = SSL_CTX_set_tlsext_use_srtp(s_ssl_ctx, dtls_srtp_crypto.c_str());
    if (ret != 0) {
        MS_THROW_ERROR("init srtp tlsext error");
    }

    for (auto& item : string2finger_print_algorithm) {
        const std::string& algorithm_str      = item.first;
        finger_print_algorithm_enum algorithm = item.second;
        uint8_t bin_finger_print[EVP_MAX_MD_SIZE];
        char finger_print_sz[(EVP_MAX_MD_SIZE * 3) + 1];
        unsigned int size = 0;

        const EVP_MD* hash_function;

        switch(algorithm)
        {
            case FINGER_SHA1:
            {
                hash_function = EVP_sha1();
                break;
            }
            case FINGER_SHA224:
            {
                hash_function = EVP_sha224();
                break;
            }
            case FINGER_SHA256:
            {
                hash_function = EVP_sha256();
                break;
            }
            case FINGER_SHA384:
            {
                hash_function = EVP_sha384();
                break;
            }
            case FINGER_SHA512:
            {
                hash_function = EVP_sha512();
                break;
            }
            default:
            {
                MS_THROW_ERROR("unkown finger print type:%d", (int)algorithm);
            }
        }
        int ret = X509_digest(s_certificate, hash_function, bin_finger_print, &size);
        if (!ret) {
            MS_THROW_ERROR("X509_digest error");
        }
        for (unsigned int index = 0; index < size; index++)
        {
            std::sprintf(finger_print_sz + (index * 3), "%.2X:", bin_finger_print[index]);
        }
        finger_print_sz[(size * 3) - 1] = '\0';
        log_infof("dtls %s fingerprint: %s", algorithm_str.c_str(), finger_print_sz);

        finger_print_info info;

        info.algorithm = string2finger_print_algorithm[algorithm_str];
        info.value     = finger_print_sz;

        s_local_fingerprint_vec.push_back(info);
    }

    return;
}

I try to use s_ssl_ctx as input parameter in your api: Server(uv_loop_t loop, SSL_CTX ctx = nullptr);

But it fail when wss client is connecting.

Pls help.

[Matheus28]

Sorry but I'm unable to help you with ssl issues.

[runner365]

Hi, I have rewrte ssl for the ws28 about the wss, And the wss does work fine in server mode. You can see in my project.

new ssl: https://github.com/runner365/cpp_media_server/blob/v1.1/src/net/tcp/ssl_server.hpp ws28: https://github.com/runner365/cpp_media_server/tree/v1.1/src/net/websocket/ws28

If you want, I can submite a pr for your ws28.

[Matheus28]

I haven't had issues with ssl on ws28 with the current implementation...

If you're not initializing openssl yourself, you must call ws28::TLS::InitSSL();. I also see you're using DTLS not TLS in SSL_CTX_new...