Wiki article on how to run Unit test please

[doingnz]

It is necessary to make a debug build of the SDK to run the tests?

I can checkout netmf, run build_sdk and build_solution for STM32F4DISCOVERY using the default build environment. TinyBooter and TinyCLR load into the STM32F4DISCOVERY.

When I then try execute RunTests.exe with no parameters, it throws up errors. Same errors if I specify command line parameters to direct the test to be via USB to the device.

ERROR: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.SPOT.Platform.Test.TestSystem.RunMFTests.get_InstallRoot() in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\Pl
atform\Tools\MFTestSystem\TestSystem.cs:line 957
   at Microsoft.SPOT.Platform.Test.TestSystem.RunMFTests.SetResultPaths() in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\Pla
tform\Tools\MFTestSystem\TestSystem.cs:line 1316
   at Microsoft.SPOT.Platform.Test.TestSystem.RunMFTests.Run(String[] args) in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\P
latform\Tools\MFTestSystem\TestSystem.cs:line 562
   at Microsoft.SPOT.Platform.Test.TestSystem.RunTests() in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\Platform\Tools\MFTes
tSystem\TestSystem.cs:line 125
TestSystem: An uncaught exception was thrown when invoking RunTests: System.NullReferenceException: Object reference not set to an i
nstance of an object.
   at Microsoft.SPOT.Platform.Test.TestSystem.RunMFTests.get_InstallRoot() in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\Pl
atform\Tools\MFTestSystem\TestSystem.cs:line 957
   at Microsoft.SPOT.Platform.Test.TestSystem.RunMFTests.SetResultPaths() in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\Pla
tform\Tools\MFTestSystem\TestSystem.cs:line 1316
   at Microsoft.SPOT.Platform.Test.TestSystem.RunMFTests.Run(String[] args) in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\P
latform\Tools\MFTestSystem\TestSystem.cs:line 562
   at Microsoft.SPOT.Platform.Test.TestSystem.RunTests() in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\Platform\Tools\MFTes
tSystem\TestSystem.cs:line 160
   at Microsoft.SPOT.Platform.Test.TestSystem.Main(String[] args) in C:\Projects\BPplus\netmf-dev\netmf-interpreter\Test\Platform\To
ols\MFTestSystem\TestSystem.cs:line 90

[smaillet-ms]

Unit tests should be able to run with the standard SDK. There is rarely ever any reason to build the SDK as debug. (Unless you are debugging the VS integration or MFDeploy itself). @mortezag wrote up some details on that but it was mostly focused on our internal test support. I'm re-working our automated build/test system so it is something that can be cloned and run on any developers system or mirrored build servers so I'll try to extract the important parts of the docs out to the WIKI as I go on that.

[doingnz]

What is the time frame for a release of tests we can use in the public domain?

I cannot get the existing Unit Tests to run with Release build of the SDK on Windows nor on the supplied STM32F4DISCOVERY solution. Yes, I would also expect them to run straight out the box for an unmodified environment?

Do they run on any other person's system? Maybe it is just my system that is failing?

Hence my questions, as I need to know if there is something fundamentally wrong with what I am doing.

Most of the tests run and Pass with a Debug build of the SDK when run on the Windows Emulator.

I can get a few to run and pass when executed individually on a Debug build of my NetMF v4.4 port to the ARM920T processor (Meridian SoM from Device Solutions.)

I need to urgently debug and qualify my port as working, otherwise I am likely to be stuck with an old hacked version of NetMF 4.1 for a long time, and there will be internal pressure to avoid NetMF going forward as it is too hard (even if we upgrade to a new Cortex-M CPU) to test. (We need objective evidence for a Medical Device.)

I would like to contribute improvements for the Dallas Onewire driver/API, SD Card support above 2GB, etc..., but need a tested and trusted NetMF 4.4 on my Hardware to get to the point of contributing the changes.

If the new test system is still under development, can we get the older system working in the interim?

[smaillet-ms]

There isn't any new test system, just a new automated system to run the tests that can be re-hosted by others. The unit tests and test runner are not changing. The daily build presently runs the tests against the release SDK emulator using the following script:

call setenv_vs.cmd 14
@echo on
BuildOutput\public\Release\Test\Server\dll\RunTests.exe -testlist %1 -transport Emulator -device Microsoft

The test list is a simple text file with a unit test listed on each line. The tests run on daily builds are:

Test\Platform\Tests\CLR\mscorlib\arrays\arrays.sln
Test\Platform\Tests\CLR\mscorlib\attributes\attributes.sln
Test\Platform\Tests\CLR\mscorlib\basicconcepts\basicconcepts.sln
Test\Platform\Tests\CLR\mscorlib\bitconverter\bitconverter.sln
Test\Platform\Tests\CLR\mscorlib\classes\classes1\classes1.sln
Test\Platform\Tests\CLR\mscorlib\classes\classes2\classes2.sln
Test\Platform\Tests\CLR\mscorlib\classes\classes3\classes3.sln
Test\Platform\Tests\CLR\mscorlib\collections\collections.sln
Test\Platform\Tests\CLR\mscorlib\conversions\conversions.sln
Test\Platform\Tests\CLR\mscorlib\conversions\basic\basic.sln
Test\Platform\Tests\CLR\mscorlib\conversions\basic2\basic2.sln
Test\Platform\Tests\CLR\mscorlib\conversions\expenum\expenum.sln
Test\Platform\Tests\CLR\mscorlib\conversions\expenum2\expenum2.sln
Test\Platform\Tests\CLR\mscorlib\conversions\user\user.sln
Test\Platform\Tests\CLR\mscorlib\conversions\user\selection\selection.sln
Test\Platform\Tests\CLR\mscorlib\delegates\delegates.sln
Test\Platform\Tests\CLR\mscorlib\enums\enums.sln
Test\Platform\Tests\CLR\mscorlib\exceptions\CSharp\exceptions.sln
Test\Platform\Tests\CLR\mscorlib\expressions\expressions1\expressions1.sln
Test\Platform\Tests\CLR\mscorlib\expressions\expressions2\expressions2.sln
Test\Platform\Tests\CLR\mscorlib\lexical\lexical.sln
Test\Platform\Tests\CLR\mscorlib\namespaces\namespaces.sln
Test\Platform\Tests\CLR\mscorlib\statements\statements.sln
Test\Platform\Tests\CLR\mscorlib\structs\structs.sln
Test\Platform\Tests\CLR\mscorlib\systemlib\systemlib1\systemlib1.sln
Test\Platform\Tests\CLR\mscorlib\systemlib\systemlib2\systemlib2.sln
Test\Platform\Tests\CLR\mscorlib\threads\threads1\threads1.sln
Test\Platform\Tests\CLR\mscorlib\threads\threads2\threads2.sln
Test\Platform\Tests\CLR\mscorlib\threads\threads3\threads3.sln
Test\Platform\Tests\CLR\mscorlib\types\types.sln
Test\Platform\Tests\CLR\mscorlib\variables\variables.sln

These are intended to test the behavior of the runtime itself (as opposed to any particular port).

[smaillet-ms]

Did a bit of research on this - Turns out the test runner will NOT run if you only have the SDK. It looks like when the SDK was split up to the core framework MSI and the VSIX packages some of the test runtime payload got lost, so needed files are missing. Thus you'd need to have a full repository and run build_sdk to get all the right files. :( Have to think about how to resolve that - hate to have to build a new MSI.

[smaillet-ms]

Well, ya learn something new everyday... A bit embarrassing this wasn't documented anywhere... There is a separate MSI for the test kit. Unfortunately, due to the lack of docs it was never updated to v4.4, 😦 Tagging this as a bug to fix the MSI generation and to publish the MSI.

Have to think about merging this into the SDK in the future. I think running with a separate tool like this is somewhat counter intuitive to the now normal VS approach. I've been wanting to leverage the VS extensibility for testing to support NETMF tests directly in VS like the built-in .NET test engine or nUnit, etc.. can do.

[doingnz]

Than you for resolving the problems with the UnitTests.

I have grouped some of the remaining tests not in mscorelibtests.txt

fileiotests.txt

Test\Platform\Tests\CLR\System\IO\IOTests.sln
Test\Platform\Tests\CLR\System\IO\Directory\DirectoryTests.sln
Test\Platform\Tests\CLR\System\IO\File\FileTests.sln
Test\Platform\Tests\CLR\System\IO\FileStream\FileStreamTests.sln
Test\Platform\Tests\CLR\System\IO\MemoryStream\MemoryStreamTests.sln
Test\Platform\Tests\CLR\System\IO\Path\PathTests.sln

all above test pass except for 3 within Test\platform\tests\clr\system\io\directory\directorytests.sln

securitytests.txt

Test\Platform\Desktop\CLR\Microsoft.SPOT.Net.Security\ClientServerDesktop.sln
Test\Platform\Tests\CLR\System\Security\SecurityTests.sln
Test\Platform\Tests\CLR\System\Security\ASYMM\AsymmetricTests.sln
Test\Platform\Tests\CLR\System\Security\HASHING\HashTests.sln
Test\Platform\Tests\CLR\System\Security\KEYS\KeyTests.sln
Test\Platform\Tests\CLR\System\Security\MISC\MiscSecurity.sln
Test\Platform\Tests\CLR\System\Security\PADDING\PaddingTests.sln
Test\Platform\Tests\CLR\System\Security\RSA\RSATests.sln
Test\Platform\Tests\CLR\System\Security\STREAMS\ StreamTests.sln
Test\Platform\Tests\CLR\System\Security\X509Certificate\CertTests.sln
Test\Platform\Tests\CLR\System\Security\X509Certs\CertsTest2.sln

all of the above tests pass

graphicstests.txt

Test\Platform\Tests\CLR\Microsoft.SPOT.TinyCore\Controls\Controls.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.Graphics\Bitmap.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.TinyCore\Presentation\Presentation.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.TinyCore\Shapes\Shapes.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.TinyCore\Media\Media.sln

all of the above tests pass.

networktests.txt

Test\Platform\Tests\CLR\Microsoft.SPOT.Net\netInfoTests\netInfoTests.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.Net\WiFiTests\WiFiTests1\WiFiTests1.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.Net.Security\ClientServer\Device.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.Net.Security\SslStream\SslStreamTests.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.Time\TimeServiceTests\TimeServiceTests.sln
Test\Platform\Tests\CLR\System\Http\HttpTests.sln
Test\Platform\Tests\CLR\System\netTests\netTests.sln
Test\Platform\Tests\CLR\System\socketTests\socketTests.sln

SSL stream tests pass, the others are either all skipped or fail. Specifically HttpTests, and TimeServiceTests fail when I run them on the Emulator.

there are a few remaining tests I grouped as othertests.txt for now.

these pass

Test\Platform\Tests\Native_MathMethods\Math\Math\MathTests.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.Hardware\LargeBuffer\LargeBuffer.sln
Test\Platform\Tests\CLR\System\xmlTests\xmlTests.sln
Test\Platform\Tests\CLR\System\SPOTXmlTests\SPOTXmlTests.sln
Test\Platform\Tests\Desktop\MFDeploy\MFDeployBasicTests\MFDeployBasicTests.sln   

these are all skipped

Test\Platform\Tests\CLR\Microsoft.SPOT.Hardware\I2C\I2C.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.Hardware\UsbClient\usbclient.sln
Test\Platform\Tests\CLR\Microsoft.SPOT.Interop\Interop.sln
Test\Platform\Tests\CLR\System\Text\Regexp\RegEx.slnproj
Test\Platform\Tests\CLR\System\Text\StringBuilder\StringBuilder.slnproj

these two fail

Test\Platform\Tests\CLR\Microsoft.SPOT\ExtendedWeakReferenceTests\ExtendedWeakReferenceTests.sln  
Test\Platform\Tests\CLR\Microsoft.SPOT.Hardware\EmulateHardware\EmulateHardware.sln

I have not tried the performance tests

Test\Platform\Tests\Performance\ProfilerTests\ProfilerCollectionsTests\ProfilerCollectionsTests.sln
Test\Platform\Tests\Performance\ProfilerTests\ProfilerStringTests\ProfilerStringTests.sln
Test\Platform\Tests\Performance\ProfilerTests\ProfilerThreadTests\ProfilerThreadTests.sln
Test\Platform\Tests\Performance\ProfilerTests\ProfilerXmlTests\ProfilerXmlTests.sln
Test\Platform\Tests\Performance\Strings\Strings.sln
Test\Platform\Tests\Performance\Sockets\Sockets.sln
Test\Platform\Tests\Performance\Stress\Sockets\Loopback\Loopback.sln
Test\Platform\Tests\Performance\Stress\Sockets\MultiThreadStress\MultiThreadStress.sln
Test\Platform\Tests\Performance\Stress\Sockets\NormalMode\SocketClient\SocketClientTest.sln
Test\Platform\Tests\Performance\Stress\Sockets\NormalMode\SocketServer\SocketServerTest.sln

[smaillet-ms]

Some of those failing tests require a specialized setup to succeed. Sadly, documentation on the details is not provided with the tests.

[doingnz]

I listed the summary in the hope that more people might help get the remaining tests working, or to add to this discussion (or source code) the steps to setup the required environments to run the tests.

[doingnz]

Some notes on single stepping MFTestSystem.

As a first step I wanted to improve MFTestSystem's handling of -testlist option. (improve handling of errors in the file with the list of solutions.)

To single step the PC application, I followed these steps.

1. Delete BuildOutput folder
2. Set FLAVOR=Debug
3. Set FLAVOR_SDK=Debug
4. Set FLAVOR_WIN=Debug
5. Set FLAVOR_DAT=Debug
6. Build_sdk.cmd

Then used the following to launch VS2015. I output the ENVIRONMENT variables so that I can check the context VS2015 is starting with.

@echo off

:ARGSOK
setlocal
call setenv_vs.cmd 14

@echo -------------------------------------------------------------------------------------------
set
@echo -------------------------------------------------------------------------------------------
@echo on
devenv
endlocal

In VS2015, browse and load the MFTestSystem.sln

\netmf\netmf-interpreter\Test\Platform\Tools\MFTestSystem

With VS2015 in Debug, it is then possible to single step into MFTestSystem..

Note that if you switch VS2015 to Release, the project will not run unless you build a regular Release SDK. (see step 1. above where the BuildOutput folder was deleted before doing the Debug build.)

[doingnz]

Do the mscoretests.txt list of test run on the STM32 Discovery evaluation board?

[smaillet-ms]

No idea, we have only regularly run them on the emulator and the MCBSTM32F400.. Can't think of a reason why they wouldn't run on the discovery as they are not hardware specific the ones in that list are intended to test the core of the runtime itself independent of a given port.

[doingnz]

With the solution built using MDK, the mscorelibtests.txt all pass as long as the device being tested starts with no application loaded and non of the tests fail catastrophically.

bitconverter tests appear to fail and thereafter the test system does not recover till that app is deleted from the device. May need to add a recovery process to MSTestSystem that reboots and stays in the booter and then deletes a rouge application.

What I am now trying to do is build the same solution using ARM DS5. This was not working. Loaads and runs, but gets "lost" after USB enumerates the system. i.e. MFDeploy shows correct device name, but a ping returns no connection. I don;t have a means to single step the code, so cannot see where it is falling over.

Ultimate goal is to show DS5 build is OK in and effort to resolve why unit testing ARM920T port will not run the test list over USB. (thinking now, I should try over serial link to try isolate to USB. An over night exercise given the speed limit)

.