Getting SIGSEGV in EVP_MD_CTX_free

Hi Everyone, Recently I have moved to using OpenSSL 3.0.8 from OpenSSL 1.0.2 in SoftHSMv2 library.

I've been getting SIGSEGV 11 while doing EVP_MD_CTX_free(ctx) with the following error trace

`*Before first iteration ** In HashInit *: In HashInit before EVPDigestInitex **: * In getEVPHash() ** *End of HashInit : * In HashUpdate : * In Hashupdate before EVPDigestUpdate : * End of HashUpdate : * In HashUpdate : * In Hashupdate before EVPDigestUpdate : * End of HashUpdate : * In HashFinal : 0x7fa730151470 * In HashFinal before resize : * In getEVPHash() * In HashFinal EVP_MD_SIZE returns 32 : In HashFinal after resize : * In HashFinal before EVPDigestFinal : * In HashFinal After EVPDigestFinal : * In HashFinal before resize - 2 ***** In HashFinal after resize - 20x7fa730151470 **OPENSSL -> ENtered EVP_MD_CTX_free * 806687856 **OPENSSL -> EVP_MD_CTX_free after null check * 806687856 OPENSSL -> Entered evp_md_ctx_reset_ex 806687856 OPENSSL -> evp_md_ctx_reset_ex after null check806687856 **OPENSSL -> evp_md_ctx_reset_ex after evp_md_ctx_clear_digest 806687856 OPENSSL -> evp_md_ctx_reset_ex after OPENSSL_cleanse 806687856 **OPENSSL -> EVP_MD_CTX_free after EVP_MD_CTX_reset * 806687856 **OPENSSL -> EVP_MD_CTX_free after OPENSSL_free * 806687856 After EVP_MD_CTX_FREE: 0x7fa730151470 End of HashFinal **: 0 After first iteration

***iteration count : 1709

* In HashInit ***: In HashInit before EVPDigestInitex : * In getEVPHash() *End of HashInit : * In HashUpdate : * In Hashupdate before EVPDigestUpdate : * End of HashUpdate : * In HashFinal : 0x7fa730151470 * In HashFinal before resize : * In getEVPHash() * *** In HashFinal EVP_MD_SIZE returns 32 : * In HashFinal after resize ***: In HashFinal before EVPDigestFinal : **** In HashFinal After EVPDigestFinal : In HashFinal before resize - 2 *** In HashFinal after resize - 20x7fa730151470 ***OPENSSL -> ENtered EVP_MD_CTX_free 806687856 *OPENSSL -> EVP_MD_CTX_free after null check 806687856 OPENSSL -> Entered evp_md_ctx_reset_ex 806687856 OPENSSL -> evp_md_ctx_reset_ex after null check806687856 OPENSSL -> evp_md_ctx_reset_ex pctx value-720710000 EVP_PKEY_CTX_free - value of ctx here -720710000 **** EVP_PKEY_CTX_free - if not returned from here

ATAL ERROR : Caught a fatal signal or exception.\n\n\n\n\nFATAL ERROR : Aborting the DTM process due to fatal signal or exception.\n\n\n\n\nFATAL ERROR : Signal Received: SIGSEGV (11)\n***\n\n\nStack trace produced by process [pmdtm(:wf_mtt_01000017000000000002.s_mtt_01000017000000000002_e44ba218a6a545f18bdc664fea40cbf2)] with pid [47635].\n. /libpmasrt.so(PmDumpStackTrace+0x79)[0x7fa7d4830d69]\n /cldagnt/apps/Data_Integration_Server/67.0.1.1/ICS/main/bin/rdtm/pmdtm[0x5e89fa]\n/cldagnt/jdk/jre/lib/amd64/server/libjvm.so(+0x9771fb)[0x7fa7ae0951fb]\n /cldagnt/jdk/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x325)[0x7fa7ae09a095]\n /cldagnt/jdk/jre/lib/amd64/server/libjvm.so(+0x96e478)[0x7fa7ae08c478]\n /lib64/libpthread.so.0(+0xf630)[0x7fa7d6cdc630]\n. /libcrypto.so.3(EVP_PKEY_CTX_free+0x5b)[0x7fa7b53b3033]\n. /libcrypto.so.3(+0x1f7e0d)[0x7fa7b5379e0d]\n./ libcrypto.so.3(EVP_MD_CTX_reset+0x1d)[0x7fa7b5379eb9]\n. /libcrypto.so.3(EVP_MD_CTX_free+0x55)[0x7fa7b537a017]\n. /libsofthsm2.so(+0xbb844)[0x7fa7b5b1a844]\n. /libsofthsm2.so(+0xcbcbc)[0x7fa7b5b2acbc]\n. /libsofthsm2.so(+0xcc4e9)[0x7fa7b5b2b4e9]\n. /libsofthsm2.so(+0xccb2d)[0x7fa7b5b2bb2d]\n. /libsofthsm2.so(+0xf003d)[0x7fa7b5b4f03d]\n. /libsofthsm2.so(+0xee352)[0x7fa7b5b4d352]\n. /libsofthsm2.so(+0x73283)[0x7fa7b5ad2283]\n. /libsofthsm2.so(C_InitToken+0x3e)[0x7fa7b5ab1d19]\n. /libiCPSDK.so.1(_ZN16CryptoOpsManager9initTokenERK17IHSMConfiguration+0x376)[0x7fa7b66ee36a]\n. /libiCPSDK.so.1`

Below is the code for reference.

EVPHashAlgorithm.cpp link to file

`// Destructor OSSLEVPHashAlgorithm::~OSSLEVPHashAlgorithm() { std::cout << "*****EVP Destructor called\n"; if(curCTX != NULL) EVP_MD_CTX_free(curCTX); }

// Hashing functions bool OSSLEVPHashAlgorithm::hashInit() { std::cout << "* In HashInit : "; if (!HashAlgorithm::hashInit()) { return false; }

// Initialize the context curCTX = EVP_MD_CTX_new(); if (curCTX == NULL) { ERROR_MSG("Failed to allocate space for EVP_MD_CTX");

return false;

} std::cout << "* In HashInit before EVPDigestInitex ***: "; // Initialize EVP digesting if (!EVP_DigestInit_ex(curCTX, getEVPHash(), NULL)) { ERROR_MSG("EVP_DigestInit failed"); std::cout << "DigestInit failed\n"; EVP_MD_CTX_free(curCTX); curCTX = NULL;

ByteString dummy;
HashAlgorithm::hashFinal(dummy);

return false;

} std::cout << "*End of HashInit : "; return true; }

bool OSSLEVPHashAlgorithm::hashUpdate(const ByteString& data) { std::cout << "* In HashUpdate : "; if (!HashAlgorithm::hashUpdate(data)) { return false; }

// Continue digesting if (data.size() == 0) { return true; } std::cout << "* In Hashupdate before EVPDigestUpdate : "; if (!EVP_DigestUpdate(curCTX, (unsigned char*) data.const_byte_str(), data.size())) { ERROR_MSG("EVP_DigestUpdate failed"); std::cout << "*****DigestUpdate failed\n"; EVP_MD_CTX_free(curCTX); curCTX = NULL;

ByteString dummy;
HashAlgorithm::hashFinal(dummy);

return false;

} std::cout << "* End of HashUpdate : "; return true; }

bool OSSLEVPHashAlgorithm::hashFinal(ByteString& hashedData) { std::cout << " In HashFinal : " << curCTX; if (!HashAlgorithm::hashFinal(hashedData)) { return false; } std::cout << " In HashFinal before resize : "; size_t size = EVP_MD_size(getEVPHash()); std::cout << " In HashFinal EVP_MD_SIZE returns " << size <<" : "; hashedData.resize(size); std::cout << " In HashFinal after resize : "; unsigned int outLen = hashedData.size(); std::cout << " In HashFinal before EVPDigestFinal : "; if (!EVP_DigestFinal_ex(curCTX, &hashedData[0], &outLen)) { ERROR_MSG("EVP_DigestFinal failed"); std::cout << "****EVP_DigestFinal failed\n"; EVP_MD_CTX_free(curCTX); curCTX = NULL;

return false;

} std::cout << "* In HashFinal After EVPDigestFinal ***: "; std::cout << " In HashFinal before resize - 2"; hashedData.resize(outLen); std::cout << "* In HashFinal after resize - 2" << curCTX; if (curCTX != NULL) { EVP_MD_CTX_free(curCTX); curCTX = NULL; } std::cout << "* After EVP_MD_CTX_FREE: " << curCTX;

std::cout << "* End of HashFinal : " << curCTX; return true; } `

These are getting called from RFC4880.cpp link to file

`// This function derives a 256-bit AES key from the supplied password data bool RFC4880::PBEDeriveKey(const ByteString& password, ByteString& salt, AESKey** ppKey) { // Check that a proper salt value was supplied; it should be at least 8 bytes long if (salt.size() < 8) { ERROR_MSG("Insufficient salt data supplied for password-based encryption");

return false;

}

// Check other parameters if ((password.size() == 0) || (ppKey == NULL)) { return false; }

// Determine the iteration count based on the last byte of the salt unsigned int iter = PBE_ITERATION_BASE_COUNT + salt[salt.size() - 1];

// Get a hash instance HashAlgorithm* hash = CryptoFactory::i()->getHashAlgorithm(HashAlgo::SHA256);

if (hash == NULL) { ERROR_MSG("Could not get a SHA-256 instance");

return false;

}

// Perform the first iteration which takes as input the salt value and // the password ByteString intermediate; std::cout << "**Before first iteration"; if (!hash->hashInit() || !hash->hashUpdate(salt) || !hash->hashUpdate(password) || !hash->hashFinal(intermediate)) { ERROR_MSG("Hashing failed");

CryptoFactory::i()->recycleHashAlgorithm(hash);

return false;

} std::cout << "**After first iteration"; // Perform the remaining iteration std:: cout << "***iteration count : " << iter << "****"; while (--iter > 0) { if (!hash->hashInit() || !hash->hashUpdate(intermediate) || !hash->hashFinal(intermediate)) { ERROR_MSG("Hashing failed");

    CryptoFactory::i()->recycleHashAlgorithm(hash);

    return false;
}

}

// Create the AES key instance ppKey = new AESKey(256); (ppKey)->setKeyBits(intermediate);

// Release the hash instance CryptoFactory::i()->recycleHashAlgorithm(hash);

return true; }`

opendnssec / SoftHSMv2

Getting SIGSEGV in EVP_MD_CTX_free #725