[MRG] Use Azure pipelines for testing

[mstimberg]

I had a look into testing with the Microsoft Azure pipeline framework (of which @thesamovar made me aware recently). I tried it out here for Brian2GeNN, but in the long run it might of course be useful for Brian itself.

The advantage over our current testing infrastructure is that it has support for Linux, OS X, and Windows, we'd therefore no longer would have to deal with two separate CI configurations (travis for Linux/OS X, appveyor for Windows). Another advantage is that currently our Windows tests on appveyor are running all in sequence which takes quite a long time after all the Linux/OS X tests already ran through (much more relevant for Brian 2 of course). With Azure, up to 10 tests can run in parallel, regardless of the platform. Finally, with travis and appveyor we have a maximum run time per job of ~1h, which can be problematic. With Azure, the time out is at 6h or something like that.

So, all this sounds good and the configuration is pretty straightforward. Unfortunately, I did not get the Windows configuration to compile properly. It has an installation of Visual Studio 2017, but this does not seem to be configured properly, or there is something that we are doing wrong in setting things up in Brian2GeNN or GeNN.

If I run things without any further configuration, I get an error about a missing Microsoft.Cpp.default.props file:

2018-11-22T16:53:17.7340448Z Project "D:\a\1\genn\lib\generate.vcxproj" on node 1 (default targets).
2018-11-22T16:53:17.7341836Z D:\a\1\genn\lib\generate.vcxproj(26,3): error MSB4019: The imported project "D:\Microsoft.Cpp.default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
2018-11-22T16:53:17.7342217Z Done Building Project "D:\a\1\genn\lib\generate.vcxproj" (default targets) -- FAILED.

(full log)

I googled a bit around, and this error disappears when I set the VCTargetsPath variable to C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\v140:

2018-11-22T17:13:49.7993123Z Project "D:\a\1\genn\lib\generate.vcxproj" on node 1 (default targets).
2018-11-22T17:13:49.7994283Z Project "D:\a\1\genn\lib\generate.vcxproj" (1) is building "D:\a\1\genn\lib\lib_genn.vcxproj" (2) on node 1 (default targets).
2018-11-22T17:13:49.7995056Z C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\v140\Microsoft.Cpp.Current.targets(64,5): error MSB4062: The "SetEnv" task could not be loaded from the assembly C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\v140\Microsoft.Build.CppTasks.Common.dll. Could not load file or assembly 'Microsoft.Build.Utilities.Core, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [D:\a\1\genn\lib\lib_genn.vcxproj]
2018-11-22T17:13:49.7997401Z Done Building Project "D:\a\1\genn\lib\lib_genn.vcxproj" (default targets) -- FAILED.
2018-11-22T17:13:49.8363575Z Done Building Project "D:\a\1\genn\lib\generate.vcxproj" (default targets) -- FAILED.

(full log)

A similar error when I set it to C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets:

2018-11-22T12:35:44.6962740Z Project "D:\a\1\genn\lib\generate.vcxproj" on node 1 (default targets).
2018-11-22T12:35:44.6974576Z C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.Cpp.DesignTime.targets(491,5): error MSB4062: The "SetEnv" task could not be loaded from the assembly C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.Build.CppTasks.Common.dll. Could not load file or assembly 'Microsoft.Build.Utilities.Core, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [D:\a\1\genn\lib\generate.vcxproj]
2018-11-22T12:35:44.6975283Z Done Building Project "D:\a\1\genn\lib\generate.vcxproj" (default targets) -- FAILED.

(full log)

Googling further only led to me to suggestions to install Visual Studio 2015 in parallel and things like that, I'm not sure that this is a viable solution to do for every test... Given that I know next to nothing about building stuff with Visual Studio on Windows, maybe @neworderofjamie or @thesamovar have an idea?

[neworderofjamie]

That does sound very cool! I can confirm that GeNN does work fine with Visual Studio 2017 but, the VCTargetsPath issue sounds symptomatic of something not being set up right. To my knowledge the first (MSBuild) path is the correct one, but it should be set through the registry if the C++ compiler is properly configured.

[thesamovar]

Agree - very nice. I remember I didn't manage to get GeNN building with MSVC on my machine, and it may be the same issue. I'll have another look at it when I can!

[neworderofjamie]

@thesamovar let me know if you find any issues - as long as you have a Visual Studio >= 2013 and < the bleeding edge that's too new for CUDA it should work fine.

[neworderofjamie]

@mstimberg - https://blogs.msdn.microsoft.com/vcblog/2017/11/15/side-by-side-minor-version-msvc-toolsets-in-visual-studio-2017/ also suggests that that is the wrong version for Visual Studio 2017 - it should be using v141

[mstimberg]

Good news, I managed to get it working! I'm not sure whether it is a misconfiguration of the server or a problem in distutils, but the routine to find vcvarsall.bat did not find the one from VS 2017 and then used a different one from the fell back to the one from the v140 toolset. Giving the path manually (via our Brian preference) worked.

[neworderofjamie]

Awesome! Quite how vcvarsall.bat ever gets found is a mystery to me

[mstimberg]

In related news: vswhere is a new tool to "locate Visual Studio 2017 and newer installations"... :roll_eyes:

[mstimberg]

Hmm, while the documentation states that the VMs "can run jobs for up to 360 minutes (6 hours).", I just had an OS X job canceled because it "has exceeded the maximum execution time of 60." :disappointed:

[neworderofjamie]

Maybe the default is lower. It looks like there's a cancelTimeoutInMinutes property you can set

[mstimberg]

:man_facepalming: Good catch! https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=vsts&tabs=yaml#timeouts

[mstimberg]

(For documentation: the relevant setting is timeoutInMinutes not cancelTimeoutInMinutes, the latter is the timeout before forcefully killing a job after cancelling it.)

[mstimberg]

Everything's looking good, we test against GeNN's latest version from master and against the last release (3.2) currently, for single precision and double precision, for Python 2.7 and 3.6, and for Linux + OS X + Windows. To avoid too much of combinatorial explosion, the Python 3.6 tests are only performed against GeNN's release and with double precision. All good to merge, I'd say.

[tnowotny]

sounds very good indeed!