Power-up self test failed and exception thrown with attached debugger

Hi all,

We are using a CryptoPP 64-bit Dll version 8.7.0 built with VS 2022.

The dll works fine in debug and release build as long as no debugger is attached. But with an debugger attached, during the DoDllPowerUpSelfTest() in fipstest.cpp the function bool IntegrityCheckModule(const char *moduleFilename, const byte *expectedModuleMac, SecByteBlock *pActualMac, unsigned long *pMacFileLocation) fails and as a result the exception Cryptographic algorithms are disabled after a power-up self test failed. is thrown when using a cryptographic function.

We guess that something similar to the lines 406 - 416 might solve the problem:

#if (_MSC_VER >= 1400 && _MSC_VER < 1920) && (defined(_M_IX86) || defined(_M_X64))
    // first byte of _CRT_DEBUGGER_HOOK gets modified in memory by the debugger invisibly, so read it from file
    if (IsDebuggerPresent())
    {
        if (subSectionMemStart <= (byte *)&_CRT_DEBUGGER_HOOK && (byte *)&_CRT_DEBUGGER_HOOK < subSectionMemStart + subSectionSize)
        {
            subSectionSize = (byte *)&_CRT_DEBUGGER_HOOK - subSectionMemStart;
            nextSubSectionStart = (byte *)&_CRT_DEBUGGER_HOOK - sectionMemStart + 1;
        }
    }
#endif

We circumvented the problem by changing the return value as follows without really solving the problem.

#if defined(_MSC_VER) && (_MSC_VER >= 1400) && (defined(_M_IX86) || defined(_M_X64))
    if (IsDebuggerPresent())
    {
        OutputDebugStringA("Crypto++ DLL integrity check failure ignored due to attached debugger.\n");
        return true;
    }
#endif

We would appreciate if the problem could be solved. At least, if there is no better solution than our hack.

There is a wiki page with information on filing useful bug reports. If you have some time please visit http://www.cryptopp.com/wiki/Bug_Report on the wiki. The executive summary is:

Windows 10 Professional 64-bit
Crypto++ 8.7.0
Visual Studio 2022, using provided project files

compiler command line

/Yu"pch.h" /ifcOutput "x64\cryptdll\Release\" /GS /W4 /wd"4231" /wd"4251" /wd"4275" /wd"4355" /wd"4505" /Gy /Zc:wchar_t /Zi /Gm- /O2 /Ob2 /Fd"x64\cryptdll\Release\vc143.pdb" /Zc:inline /fp:precise /D "NDEBUG" /D "CRYPTOPP_EXPORTS" /D "CRYPTOPP_ENABLE_COMPLIANCE_WITH_FIPS_140_2=1" /D "USE_PRECOMPILED_HEADERS" /D "_WINDLL" /errorReport:none /GF /WX- /Zc:forScope /Gd /Oy /Oi /MT /FC /Fa"x64\cryptdll\Release\" /EHsc /nologo /Fo"x64\cryptdll\Release\" /Fp"x64\cryptdll\Release\cryptopp.pch" /diagnostics:column

linker command line

/OUT:"x64\DLL_Output\Release\\cryptopp.dll" /ALLOWBIND /MANIFEST /NXCOMPAT /PDB:"x64\DLL_Output\Release\\cryptopp.pdb" /DYNAMICBASE:NO "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /IMPLIB:"C:\Develop\CSB-Image-Meater\thirdParty\CryptoPP\x64\DLL_Output\Release\\cryptopp.lib" /DEBUG /DLL /BASE:"0x42900000" /MACHINE:X64 /OPT:REF /PGD:"x64\DLL_Output\Release\cryptopp.pgd" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"x64\cryptdll\Release\cryptopp.dll.intermediate.manifest" /LTCGOUT:"x64\cryptdll\Release\cryptopp.iobj" /OPT:ICF /ERRORREPORT:PROMPT /ILK:"x64\cryptdll\Release\cryptopp.ilk" /NOLOGO /TLBID:1

weidai11 / cryptopp

Power-up self test failed and exception thrown with attached debugger #1153