ctest setup

[coldfix]

When testing the redesign-cpymad-backend branch, it failed because the test setup did not take care to install a dependency. Also, I have the suspicion that there are sometimes problems stemming from previous builds (uncleaned build directory?). So I had a look into the ctest files.

To be honest, I don't really understand the ctest setup: which files get invoked when, why and from where? Also, I noticed, that the ctest logic for testing and control (environment, submission) is intermixed.

IMO, there should be

• one command to run tests within the current (arbitrary) python environment, like python setup.py test
• one command that knows how to invoke the test command for multiple environments (python versions, madx-versions, etc), like in .travis.yml. This should also be responsible for providing:
• a clean runtime environment everytime the tests are run: (fresh virtualenv and new clone/build directory)
• a command to submit the test results to a dashboard
• no need to modify configuration (like source/build directory) before running the tests

Do you have any thoughts on how to simplify/modularize the ctest setup? I will research for a solution that is comparable to Travis regarding simplicity, but this might take a while.

[Eothred]

• There is only one command to run the tests: "ctest". In addition, "python setup.py build" is ran to build the modules..
• Settings like you mention are, if I understand you correctly, defined in the template script which (can) differ for each client.

The logic is a bit different than that of Travis. CDash merely aggregates the results from different clients, and any machine can be used as a test server with in principle minimal configuration. Travis is a complete testing server system (but we are then limited to only test on this server, not on any client we want).

Do we really need virtualenv now? I would like it if it was not needed for testing, but I guess I could reconfigure otherwise..

[coldfix]

The virtualenv is not needed, it is merely a more convenient way to guarantee that the builds are independent from the global python environment and it does the PYTHONPATH modifications more safely for us.

I don't think you should hurry into changing the setup from the outside. Rather, when doing the test builds, just provide a more flexible entrypoint:

• clone the tested branch into a new directory with no leftovers whatsoever
• run a command like test.sh in the repository folder

For now the test.sh could simply execute ctest. But, later on, it will be easier to modify the behaviour by just modifying the pymad source:

• install dependencies (without sudo of course)
• setup a virtualenv
• run against different python versions
• add command line options for
  • python versions
  • test coverage reports
  • dashboard submission
  • slow/fast tests
• possibly stop using ctest

I believe, that this is also a requirement if you want to use the ctest on Travis. Since on Travis we already are inside a virtualenv and should not modify the PYTHONPATH or explicitly set the python interpretor, but rather just use what's there in our environment.

[coldfix]

I'd suggest to use tox to provide the environment and dependencies. This will make the test execution on your machines very similar to Travis and greatly simplifies the configuration.

Optimally, I'd recommend to drop the ctest script completely (too much configuration hazzle, and ctest is really not the greatest solution for python), but I am not sure how to submit test results to CDash or another similar service without it.

[Eothred]

Yes, cdash is most convenient for projects that also build using cmake. tox documentation suggest Jenkins, but I have no experience installing that. If I find some spare time at some point I can have a look and see if I manage to set it up.

[Eothred]

Right, so while looking into Jenkins I remembered another reason why chosing cdash. Cdash is purely php-based, so no need for special setup on the server side. On the other hand, Jenkins seems to be based on Java, so we would need a special web server to run it. I do not have that available (and in any case I might be leaving CERN soon, in which case this server would disappear anyway).

[coldfix]

Java, ugh... A friend recommended buildbot, which I will look into.

Regarding your plans to leave CERN: how soon would that be? Does this mean, you will quit pymad development as well?

[Eothred]

Difficult to say, depends on the new job I guess. I'm gone by June...

[coldfix]

I am currently working on a buildbot branch, which is a feature-rich and extremely customizable CI framework.

Short overview: The server (so called master) is a python application that can be started via buildbot start. The builds are run on clients (so called slaves), that are started via buildslave start. Both applications are compatible with python2.6 and greater, see here and here.

Features:

• perform builds on updates in any branch (like travis)
• includes a http status page, other status targets (like github PR comments) can be configured
• nightly builds on fixed set of branches
• connect multiple build slaves, each running on an independent platform and python version (windows compatible)
• manually trigger builds on all machines
• listen for changes from multiple repositories (say separate repositories for models and cpymad) and trigger builds if any of these changes
• (future option: automatically create+upload binaries from connected machines)

I think, it makes the most sense to keep the buildbot configuration in a separate (very small) repository. It shouldn't be too hard to make the buildbot reconfigure itself automatically, when the config changes. Passwords are best kept in a separate YAML file, that is not added to the repository.

Do you still have access to the CERN machines to setup build slaves? The master could be hosted elsewhere, if you don't have access to any CERN machine with a public IP.

[Eothred]

I have access to the CERN machines yes, but they do not have a public IP. This means that the master has to be able to communicate to the slaves somehow, or are the slaves triggering by themselves somehow? The former will be problematic for the CERN machines I think...

[coldfix]

When started a slave connects to the master (and tries to keep the connection alive/reconnect when something goes wrong). So the master is the only machine that needs a public IP.

The slaves do not need any configuration of their own, except for the IP/dns of the master and the password to authenticate with the master.

[Eothred]

OK, then this should be a working solution as long as we can get hold of a master I guess. Hopefully more flexible than cdash/ctest...

[coldfix]

I am currently considering to rent a V-server for personal use, which could make a nice master as well. Not sure, whether you like this solution though.

[Eothred]

As long as you're fine with spending your resources on it I have no problem with it (assuming it is reasonably easy to migrate to a more official server in the future should that ever be available and needed).

[coldfix]

It's not too hard to migrate to another server. I have put up a rudimentary configuration and setup instructions for both master and slaves here.

To make reconfiguration easier, I plan on reading the list of tests (as well as install commands) from a YAML configuration file and make the master automatically reconfigure itself when a change on the config file occurs.

[coldfix]

Closing my opened issues for now, as I will handle these in the fork.