Need cryptest.cmd for Windows testing

[noloader]

On Linux, OS X and Unix we have a "test script from hell": cryptest.sh. The script tests the library under various configuration options on a number of different platforms. Its value comes from detecting problems before code is checked in, and detecting problems before users experience issues in the field.

We need a test script for Windows that builds the various Visual Studio configurations under a Developer Command Prompt using Visual Studio 2010 and above with msbuild. That will allow us to _easily_ test changes like Issue 143: Support for Universal Windows Platform (UWP). The script could be named cryptest.cmd. It can run as a batch/command file, as a command-based script host, or as a PowerShell script.

If you are interested in this, then don't loose sight of the goal: we want something that we (1) "Open Developer Command Prompt" and then (2) torture tests library under various configurations.

[zabulus]

torture tests library under various configurations.

What configurations do we want to test? I suppose that we need to adopt defines from here: https://cryptopp.com/wiki/Release_process#Test_Script

What VC compilers script should support? In wider sense: what prerequisites for running of this script should be? I know that stand-alone (without setting up) compiler + msbuild exists only for VC14. Lower versions need Visual Studio to be installed.

[noloader]

@zabulus,

What configurations do we want to test?

Currently there are about 24 of them, and they are detailed at Visual Studio | Projects and Configurations.

On Windows desktop/server, we want to test {Debug,Release}x{32-bit,64-bit}x{No DLL, DLL-Import}.

On mobile devices, we can forgo the DLL since its a FIPS DLL. In this case, we want to torture test {Debug,Release}x{32-bit,64-bit}.

In addition to the matrix that results in the 24 configurations, we want to test {Debug,Release}x{32-bit,64-bit} mated with options like CRYPTOPP_NO_UNALIGNED_DATA_ACCESS, NO_OS_DEPENDENCE, and CRYPTOPP_USE_FIPS_202_SHA3 (and friends). We also want to exercise CRYPTOPP_NO_UNALIGNED_DATA_ACCESS, NO_OS_DEPENDENCE, and CRYPTOPP_USE_FIPS_202_SHA3 (and friends) on mobile.

---

What VC compilers script should support?

I think msbuild means its Visual Studio 2010 and above. It will allow us to automate testing where it matters most - VS2010 and above.

I currently test back to .Net 2002 on Windows 2000 and .Net 2003 on Windows XP. Its a special kind of hell :) I can manually run these tests under Visual Studio and detect potential breaks. Its not as aggressive and efficient as the script, but it does work.

[zabulus]

Can CMake here be used? In this case CMake generation of .sln also will be tested

[noloader]

Can CMake here be used? In this case CMake generation of .sln also will be tested

On Windows, the build system is Visual Studio. Anything we do has to work with Windows and Visual Studio.

We can help out the folks who want Cmake, but its not the build system. I don't have it on any of my Windows test systems (Windows 2000 through Windows 8).

Also, my limited testing of Cmake on Linux and Unix has turned up more problems than it solves. I finally gave up on testing it because there were so many issues. Its also the source of a number of bugs in our GitHub bug reporter. I'm not very happy with Cmake at the moment :(

[zabulus]

On Windows, the build system is Visual Studio. Anything we do has to work with Windows and Visual Studio.

How do you want to organize solution files/configurations/targets? My opinion: currently used the vs2010.zip is something inconvinient in many senses: versioning, using sln files and so on.

I suggest to unpack it in something like .\Build\<win?>\vs2010 and use it from there. Further, we could build solution projects using msbuild partial files. For example list of files to compile is declared in one place, and all other .vcxproj just include them, declaring specific compilation options.

[noloader]

My opinion: currently used the vs2010.zip is something inconvinient in many senses...

In the long term, vs2010.zip won't be a problem. We are cutting over to them at the next major bump (5.7 or 6.0). The ZIP file was added adhoc because (1) VCUpgrade absolutely sucks and was butchering the conversion leading to an endless stream of problems and questions; and (2) we have some visually impaired users who needed VS2010 integration with screen readers.

If you are scripting it, you can perform an unzip -aoq -d . to unpack it in-place; or unzip -aoq -d <some dir> to unpack it to <some dir>.

At the end of the day, I want to be able to open a developer prompt of my choice (WIn32, Win64, Phone, ARM, etc), cd to <cryptopp dir>, execute .\cryptest.cmd and "things just work". For example, to test your RNG patch, I will do it under developer prompts for Win32, Win64 and Windows Phone using platforms Windows 7 and Windows 8. I will manually test the compile under Windows 2000 and Windows XP to ensure the patch does not break down level clients.

Also see Issue 89: Make VS2010 default sometime in 2016.

[zabulus]

(1) VCUpgrade absolutely sucks and was butchering the conversion;

Why don't place vs 2010 & 2008 solution and project files to separate folders?

[noloader]

(1) VCUpgrade absolutely sucks and was butchering the conversion;

Why don't place vs 2010 & 2008 solution and project files to separate folders?

This causes support problems. Some users are inexperienced and can't find the output artifacts like Win32's cryptlib.lib. I don't want to overload them with choices like clicking on the wrong solution file. (You won't have this problem, but some folks will).

I don't mean to sound like a dictator. I'm really a nobody who has check-in rights. Its Wei Dai's library, and I'm just a steward. Most choices I make are tied to two principals: (1) does doing some meet the project's goals; and (2) does doing something benefit the library users. Once something like vs2010.zip passes the sniff test, then it becomes a question of making things "just work" for regular users.

[zabulus]

I don't mean to sound like a dictator. I'm really a nobody who has check-in rights.

Sure, no problem. I just need to know boundaries of what I can do in this script related to organizing files to not to cross lines.

[zabulus]

I've still unzipped vs2010.zip, but renamed cryptest.sln from archive. Can't work without versioning. I've finished script to build in 24 configurations. Looks like it works, but I still need 2010 studio to install to fully test it (I've temporally upgraded compilers to v140 toolset). One more issue with using 2010 as default build. It works with MSBuild < 12.0 (I think 4.0, can't find now). I wrote it using MSBuild 12, but don't know if there will be issues with script downgrade to 4.0. There is nuget package https://www.nuget.org/packages/MSBuild that can be downloaded during build without additional dependencies.

[noloader]

I've still unzipped vs2010.zip, but renamed cryptest.sln from archive. Can't work without versioning.

I'm not sure what you mean. But if it works, then great.

This does not need to be elegant or fancy, like production C/C++ code. This can be more of a bull-in-a-china shop as long as it achieves the goals. Remember, the workflow is simple: (1) "Open Developer Command Prompt" and then (2) torture tests library under various configurations. We do it so we know things "just work" for users in the field. They don't care how we got to that state for them.

---

Looks like it works, but I still need 2010 studio to install to fully test it (I've temporally upgraded compilers to v140 toolset).

OK, very good. Would you like me to test it? If so, make it available. Eventually, make it available as a Pull Request so you get credit for it at Contributors.

[zabulus]

https://github.com/zabulus/cryptopp/tree/cryptopp_cmd Branch is here. I'll make PR as soon as I'll test it with vs2010.

[noloader]

@zabulus,

This is good stuff. Thanks.

So here is what I am seeing under Windows 7, x64 with VS2010 and VS2012 installed.

1. Download the Crypto++ ZIP from GitHub; unpack to %DESKTOP%\cryptopp
2. Unpack vs2010.zip to %DESKTOP%\cryptopp
3. Save cryptest.cmd to %DESKTOP%\cryptopp
4. Program Files -> VS2010 -> Tools -> VS2010 Developer Prompt
5. cd %DESKTOP%\cryptopp
6. Run the script

!! Step 4
Setting environment for using Microsoft Visual Studio 2010 x86 tools.

!! Step 5
C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC>cd C:\Users\Jeffrey Walton\Desktop\cryptopp

!! Step 6
C:\Users\Jeffrey Walton\Desktop\cryptopp>.\cryptest.cmd

C:\Users\Jeffrey Walton\Desktop\cryptopp>SETLOCAL

C:\Users\Jeffrey Walton\Desktop\cryptopp>SET SolutionDir=C:\Users\Jeffrey Walton\Desktop\cryptopp\

C:\Users\Jeffrey Walton\Desktop\cryptopp>msbuild cryptopp.proj /t:BuildLibrary /
verbosity:diag /p:PreprocessorDefinitions=CRYPTOPP_NO_UNALIGNED_DATA_ACCESS
Microsoft (R) Build Engine version 4.6.1055.0
[Microsoft .NET Framework, version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1009: Project file does not exist.
Switch: cryptopp.proj

C:\Users\Jeffrey Walton\Desktop\cryptopp>if 1 EQU 1 exit /b 1

C:\Users\Jeffrey Walton\Desktop\cryptopp>

And here's what the directory looks like after Step 3, when the unpacks are complete:

C:\Users\Jeffrey Walton\Desktop\cryptopp>dir *.vc* *.sln
 Volume in drive C is Windows
 Volume Serial Number is C60A-1028

 Directory of C:\Users\Jeffrey Walton\Desktop\cryptopp

04/24/2016  07:20 PM            81,166 cryptdll.vcproj
11/22/2015  10:05 AM            95,469 cryptdll.vcxproj
10/20/2015  05:04 AM            11,932 cryptdll.vcxproj.filters
11/04/2015  04:56 AM               143 cryptdll.vcxproj.user
04/24/2016  07:20 PM            38,426 cryptest.vcproj
04/22/2016  01:53 PM            46,531 cryptest.vcxproj
04/22/2016  01:54 PM            10,426 cryptest.vcxproj.filters
10/22/2015  05:56 AM               252 cryptest.vcxproj.user
04/24/2016  07:20 PM           210,495 cryptlib.vcproj
04/21/2016  11:54 AM           247,416 cryptlib.vcxproj
04/21/2016  11:57 AM            25,091 cryptlib.vcxproj.filters
11/04/2015  04:56 AM               143 cryptlib.vcxproj.user
04/24/2016  07:20 PM             7,990 dlltest.vcproj
11/22/2015  10:05 AM            12,003 dlltest.vcxproj
11/04/2015  04:56 AM               143 dlltest.vcxproj.user

 Directory of C:\Users\Jeffrey Walton\Desktop\cryptopp

10/22/2015  05:01 AM             6,793 cryptest.sln

---

When I change the script to:

msbuild cryptlib.vcxproj /t:BuildLibrary /verbosity:diag 
/p:PreprocessorDefinitions=CRYPTOPP_NO_UNALIGNED_DATA_ACCESS
...
msbuild cryptest.vcxproj /t:BuildDynamicLibrary /verbosity:diag 
/p:PreprocessorDefinitions=CRYPTOPP_NO_UNALIGNED_DATA_ACCESS
...

It results in:

 <lots of output>

 Detailed Build Summary
 ======================
  ...
Build FAILED.

"C:\Users\Jeffrey Walton\Desktop\cryptopp\cryptlib.vcxproj" (BuildLibrary target) (1) ->
  C:\Users\Jeffrey Walton\Desktop\cryptopp\cryptlib.vcxproj : error MSB4057: The target "BuildLibrary" does not exist in the project.

    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:07.25

[zabulus]

1. Unpack vs2010.zip to %DESKTOP%\cryptopp
2. Save cryptest.cmd to %DESKTOP%\cryptopp

After that

4. Save cryptopp.proj to %DESKTOP%\cryptopp cryptopp.proj file is msbuild script that contains what projects and configuration should msbuild build.

[zabulus]

And this "/p:PreprocessorDefinitions=CRYPTOPP_NO_UNALIGNED_DATA_ACCESS" needs additional changes in vcxproj files, that should be added to content of vs2010.zip

[noloader]

cryptopp.proj file is msbuild script that contains what projects and configuration should msbuild build.

Well, that kind of begs the question.... What are we testing?

I'm fairly certain we are not testing the Crypto++ project if we are using Cmake produced project files with random flags and macros defined.

---

And this "/p:PreprocessorDefinitions=CRYPTOPP_NO_UNALIGNED_DATA_ACCESS" needs additional changes in vcxproj files, that should be added to content of vs2010.zip

This brings up a good point. What I hoped to eventually do was read the project's settings, and then add flags like CRYPTOPP_NO_UNALIGNED_DATA_ACCESS and NO_OS_DEPENDENCE (and friends) as a test configuration.

[zabulus]

I'm fairly certain we are not testing the Crypto++ project if we are using Cmake produced project files with random flags and macros defined.

It's not CMake produced file. I've wrote it yesterday :)

Well, that kind of begs the question.... What are we testing?

At first, default configurations, without special defines.

This brings up a good point. What I hoped to eventually do was read the project's settings, and then add flags like CRYPTOPP_NO_UNALIGNED_DATA_ACCESS and NO_OS_DEPENDENCE (and friends) as a test configuration.

You mean to build it *nix like. MSBuild works with .vcxproj in different way.

[noloader]

You mean to build it *nix like. MSBuild works with .vcxproj in different way.

Actually, you can configure them at build time; see MSBuild Properties on MSDN and How to override a configuration property on Stack Overflow. That's why I wanted a MSbuild based script.

[zabulus]

It is the same thing, but workaround for C++. For C++ it doesn't work without the workaround because PreprocessorDefinitions is not PropertyGroup See https://github.com/zabulus/cryptopp/blob/cryptopp_cmd/cryptlib.vcxproj#L43, https://github.com/zabulus/cryptopp/blob/cryptopp_cmd/cryptlib.vcxproj#L157. This adds PropertyGroupthat can be injected via /p:ExternalPreprocessorDefinitions=<>

[zabulus]

I've run into problem with dependencies. I'm trying to build in following sequence: cryptlib->cryptest->cryptdll->cryptdlltest. But cryptest in *_DLL configuration depends on cryptdll. Cryptdll depends cryptest on post-build to cryptest mac_dll. Looks like I need to split cryptest build into two parts.

[noloader]

I've run into problem with dependencies.... Cryptdll depends cryptest on post-build to cryptest mac_dll. Looks like I need to split cryptest build into two parts.

Yeah :) This has been a pain point for some time. I've spent so much time on it its ridiculous. See, for example, How to control order of Batch Build → Build All. I put a bounty on it but did not get any satisfactory answers.

What ever you do, the _first_ project that should be built in all cases is Win32's cryptest.exe (even when building x64). Don't leave it to chance with Visual Studio because it lacks a way for us to express the dependencies we need.

By the way, the _Batch Build → Build All_ and then subsequent runs is what I want to automate. But instead of doing it just once, I want it done multiple times with different configurations, like "project settings"+CRYPTOPP_NO_UNALIGNED_DATA_ACCESS, "project settings"+NO_OS_DEPENDENCE, etc.

[zabulus]

Ok, looks like I've done with default configurations. But, cryptest.exe in *DLL combinations throws CryptoPP::Exception caught: Cryptographic algorithms are disabled after a power-up self test failed.

[zabulus]

@noloader, I don't really understand what the point in additional cryptest.nmake? IMHO, it slightly duplicates CMake stuff, but in another way.

And I've made .vcxproj based script cryptopp.cmd. But it depends on changes in vs2010.zip content.

[noloader]

@zabulus, can you spot what is wrong here?

• First, copy cryptopp.cmd to my local Crypto++ directory
• Second, unzip vs2010.zip
• Third, open a x64 Developer Command Prompt
• Fourth, cd to Crypto++ directory

Here's the pain point. I can duplicate from a VS2010 and VS2013 developer command prompts.

C:\Users\Jeffrey Walton\Desktop\cryptopp>.\cryptopp.cmd

C:\Users\Jeffrey Walton\Desktop\cryptopp>SETLOCAL ENABLEEXTENSIONS ENABLEDELAYED
EXPANSION

C:\Users\Jeffrey Walton\Desktop\cryptopp>SET SolutionDir=C:\Users\Jeffrey Walton
\Desktop\cryptopp\

C:\Users\Jeffrey Walton\Desktop\cryptopp>RD /s /q .\Win32
The system cannot find the file specified.
...

C:\Users\Jeffrey Walton\Desktop\cryptopp>msbuild cryptopp.proj /t:BuildAll /verb
osity:m /p:"PreprocessorDefinitions="
Microsoft (R) Build Engine version 12.0.21005.1
[Microsoft .NET Framework, version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1009: Project file does not exist.
Switch: cryptopp.proj

When I change it to:

msbuild cryptlib.vcxproj /t:BuildAll /verbosity:m /p:"PreprocessorDefinitions=%1"
if %errorlevel% equ 1 exit /b %errorlevel%
msbuild cryptest.vcxproj /t:BuildAll /verbosity:m /p:"PreprocessorDefinitions=%1"
if %errorlevel% equ 1 exit /b %errorlevel%

Its results in:

C:\Users\Jeffrey Walton\Desktop\cryptopp>msbuild cryptlib.vcxproj /t:BuildAll /v
erbosity:m /p:"PreprocessorDefinitions="
Microsoft (R) Build Engine version 4.0.30319.34209
[Microsoft .NET Framework, version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. All rights reserved.

C:\Users\Jeffrey Walton\Desktop\cryptopp\cryptlib.vcxproj : error MSB4057: The
target "BuildAll" does not exist in the project.

C:\Users\Jeffrey Walton\Desktop\cryptopp>msbuild cryptest.vcxproj /t:BuildAll /v
erbosity:m /p:"PreprocessorDefinitions="
Microsoft (R) Build Engine version 4.0.30319.34209
[Microsoft .NET Framework, version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. All rights reserved.

C:\Users\Jeffrey Walton\Desktop\cryptopp\cryptest.vcxproj : error MSB4057: The
target "BuildAll" does not exist in the project.
...

[noloader]

Closing.