Port HJCFIT tests to run in new Jenkins' Windows node

[raquelalegre]

This'll help figure out what's needed for things to work at Remi's Windows machine.

[jenshnielsen]

At the moment we have visual studio 2013 (i.e. 12.0) installed on the windows nodes. We should probably also install 2015 in addition to that to get new c++11/14 features on windows

[jenshnielsen]

The Zacros windows job at https://github.com/UCL-RITS/jenkins-job-builder-files/blob/master/jobs/zacros.yml#L102 might be useful to see how to set this up

[raquelalegre]

Awaiting installation of VS2015 in the Jenkins Windows node since it's the only version compatible with python3.5.

In the meanwhile, we have tested the latest version of the develop branch on a Windows 7 64-bit Virtual Machine and it works. These are the steps I followed on my VM to get things to work:

Requirements:

• VS2015
• Python3.5, preferably installed with Miniconda3
• Python packages: behave, numpy, scipy, six (also jupyter and matplotlib would be nice for the notebooks)
• Swig
• CMake

Procedure:

1. Clone HJCFIT develop's branch:

   git clone -b develop https://github.com/DCPROGS/HJCFIT.git

2. Get into the project's folder and create a build directory where all the cmake temp files will be stored:

   cd HJCFIT
   mkdir build
   cd build

3. Load 64-bit VS2015 stuff: "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64
   • Check it went fine by looking at output of cl, it should look like: Microsoft (R) C/C++ Optimizing Compiler Version 19.00.23026 for x64, which corresponds to VS2015 64-bit version of cl.

4. Compile the code:

   cmake .. -DCMAKE_BUILD_TYPE=Release -G "NMake Makefiles" 

   • Optionally pass DCMAKE_INSTALL_PREFIX if desired to install dcprogx elsewhere than default. Default is the path where the python interpreter is.

5. Install the code:

   nmake install

   • Note it has happened to me several times that this step breaks with a rare error: Failed to write the updated manifest to the resource of file. Re-running nmake seems to fix it. See here.

6. Tests can be run optionally like this:

   ctest -T  test --output-on-failure -C Release

   • Note for debugging purposes it was very useful to run only one random python test with extra verbose output and fix my VM's configuration for next step:

   ctest -R python_qmatrix -VV

7. (not needed anymore) Something else is needed in order to correctly import dcprogs from a python prompt:
   • PYTHONPATH needs to be set and point to CMAKE_INSTALL_PREFIX: set PYTHONPATH=C:\Users\<username>\Miniconda3\envs\dcprogs\dcprogs\python-pkg
   • The path to the DLLs needs to be added to %PATH% as well: set PATH=C:\Users\<username>\Miniconda3\envs\dcprogs\dcprogs\DLLs;%PATH%

[jenshnielsen]

Awaiting installation of VS2015 in the Jenkins Windows node since it's the only version compatible with > python3.5.

While it is recommended to build with the same version of VS that python was build with (and 3.5 is build with 2015 and 2.7 is build with 2010) this has nothing to do with Python3.5. The problem is that we need a 64 bit compiler to build against a 64bit Python and the free version of VS 2013 does not ship with a 64 bit compiler and we can't get the licensed version since we have to pay a fee for every single user :(

[jenshnielsen]

The PATH and PYTHON_PATH is a bit of a hack. It should not be necessary to set python path thats only because its install into the wrong directory i.e. dcprogs\dcprogs\python-pkg

[raquelalegre]

Ah, I misunderstood, sorry. I though it was both the 64-bit problem and the incompatibility of python3.5 with VS2013.

I saw nmake installs both the C++ headers and the python stuff under Miniconda3/envs/dcprogs: the python stuff goes to dcprogs/dcprogs/python-pkg and the c headers to dcprogs/include. I'm not sure how it should be, but it's the way the tests run by ctest work so I wrongly assumed it was the only way. I'll have a look and see if I can make it work without any hacks.

[jenshnielsen]

For reference it does the correct thing on the mac (at least when you don't use a venv)

-- Install configuration: ""
-- Installing: /usr/local/include/dcprogs/DCProgsConfig.h
-- Installing: /usr/local/lib/liblikelihood.dylib
-- Up-to-date: /usr/local/include/dcprogs/likelihood/qmatrix.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/errors.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/idealG.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/occupancies.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/time_filter.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/recursion_formula.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/exact_survivor.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/root_finder.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/determinant_equation.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/asymptotes.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/approx_survivor.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/missed_eventsG.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/laplace_survivor.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/likelihood.h
-- Up-to-date: /usr/local/include/dcprogs/likelihood/brentq.h
-- Installing: /usr/local/lib/python2.7/site-packages/dcprogs/likelihood/_likelihood.so
-- Up-to-date: /usr/local/lib/python2.7/site-packages/dcprogs/likelihood/random.py
-- Up-to-date: /usr/local/lib/python2.7/site-packages/dcprogs/likelihood/_methods.py
-- Up-to-date: /usr/local/lib/python2.7/site-packages/dcprogs/likelihood/optimization.py
-- Installing: /usr/local/lib/python2.7/site-packages/dcprogs/likelihood/likelihood.py
-- Up-to-date: /usr/local/lib/python2.7/site-packages/dcprogs/likelihood/__init__.py
-- Up-to-date: /usr/local/lib/python2.7/site-packages/dcprogs/data/CH82.scn
-- Up-to-date: /usr/local/lib/python2.7/site-packages/dcprogs/data/CCO.scn
-- Up-to-date: /usr/local/lib/python2.7/site-packages/dcprogs/data/CO.scn
-- Up-to-date: /usr/local/lib/python2.7/site-packages/dcprogs/__init__.py

[jenshnielsen]

python files should go to prefix/lib/python2.7/site-packages/dcprogs and not prefix/dcprogs/python-pkg as far as I can see

[raquelalegre]

Found it, I think. The dcprogs/python-pkg bit is hardcoded in the AllPythonBindings make module. I'll change it.

[raquelalegre]

The hack is now gone and there's no need for step 7 in the earlier comment.

Windows test has now gone live, although it'll keep on failing until we get VS2015 installed.

[raquelalegre]

VS2015 has been installed in the staging Jenkins Windows node. It uses the correct version of cl and seems to run fine, but fails in the documentation tests for Python. This is a bit odd, since the other python tests don't fail. We'll need to investigate. We also need to ask for VS2015 to be installed in the production Windows node.

Almost there!

[jenshnielsen]

I am thinking that the documentation failures may be related to running the tests inside an conda virtual environment. I would suggest sshing into the node and try building with and without the env and see what passes and what fails

[jenshnielsen]

Looks like the ctest way of setting the env PYTHONPATH does not work. Possibly because it is inside the conda env? We should add something to the test that prints the python path to confirm?

Upgrading to cmake 3.4.1 did not fix it

Alternatively we can just work around by doing an nmake install

[jenshnielsen]

Up and running in production apart from the import issue. I fixed another issue related to the xml output handling which was looking for the wrong path

[raquelalegre]

Can we close this?

[jenshnielsen]

Yes Lets do that

[jenshnielsen]

@raquel-ucl Can you make sure the documentation for windows is added to the wiki?