Setting up regression tests

[olexandr-konovalov]

I'd like to volunteer to establish a set of regression tests for the SWC Windows installer using Jenkins (I have access to two Jenkins slaves, one with Cygwin and one without).

I think the 1st step should be to have a Jenkins job which builds the installer as described in https://github.com/swcarpentry/windows-installer/blob/master/README.md. It will require one-time installation of Python, py2exe and Inno Setup on the Jenkins slave, and then the script will just execute three lines

python setup.py install
python setup.py py2exe
ISCC.exe swc-installer.iss

and produce the release candidate to test. It can then publish it somewhere for those who will test it manually. The build could be weekly test, or could be triggered by pushes - so far, the latter does not seem to be a problem with the current commit activity.

This is not yet a full release cycle, but at least something to start with and catch any problems at the build stage - I hope that this already will be useful. Setting this up, I hope to understand better some internals, and then the next stage should be adding more automated checks to verify the usability of the installer. Probably one test per component (Make, nano, SQLite, R and some other "health-checks), so we would be easily see which parts pass the tests. For example, Make test may try to call make to perform some simple build and check whether it worked or not, and fail the build if not.

The tricky part of this second stage is to reset the machine to the state in which it was prior to the test. I am not familiar with SWC Windows installer yet, but as far as I understand, it downloads and installs some other software. Is there a way to uninstall that packages for a clean test? What will happen if I will run the installer next time on the same machine? What will happen if those packages are updated? Can we explicitly specify that e.g. SWC Windows installer works with this version of Git installer, and prevent it from trying to install newer version (my understanding was that this was the source of the recent problem). I would appreciate any advice on these questions, while I will be trying to set up the 1st step to automate regular builds of the installer.

[ethanwhite]

Hi @alex-konovalov - thanks again for offering to do this.

The build could be weekly test, or could be triggered by pushes

In the short-term I think triggered on pushes makes sense. In the longer-term I think we probably want weekly tests if we can also test the external dependencies since the breaks tend to come from changes in external things, not changes in the installer itself.

This is not yet a full release cycle, but at least something to start with and catch any problems at the build stage - I hope that this already will be useful.

Definitely.

Is there a way to uninstall that packages for a clean test?

Yes. All you need to do is remove the ~\.swc directory and the line(s) the installer adds to ~\.bash_profile (which on a testing system probably just means deleting ~\.bash_profile).

What will happen if I will run the installer next time on the same machine? What will happen if those packages are updated?

In concept it should just write over the existing files in ~.\swc and therefore provide the updated packages. @wking would have a better idea on exactly what would happen here.

Can we explicitly specify that e.g. SWC Windows installer works with this version of Git installer, and prevent it from trying to install newer version

In concept we could do that. From experience it wouldn't work to just say "get version 1.3" because a lot of students will just hit the first download button they see (or be overwhelmed by a list of choices), so we'd have to hard link to it from the workshop website if possible. Looking at the repo/releases [1] hard links seem possible and are likely to be pretty stable at least for a while.

[1] https://github.com/git-for-windows/git/releases

[wking]

On Tue, Nov 24, 2015 at 03:50:49AM -0800, Ethan White wrote:

Is there a way to uninstall that packages for a clean test?

Yes. All you need to do is remove the ~\.swc directory and the line(s) the installer adds to ~\.bash_profile (which on a testing system probably just means deleting ~\.bash_profile).

And remove ~\nano.rc too 1.

What will happen if I will run the installer next time on the same machine? What will happen if those packages are updated?

In concept it should just write over the existing files in ~.\swc and therefore provide the updated packages.

Actually, it tries to avoid doing this. In most cases, it skips an install if the target directory already exists [1,2,3]. The new Make code installs two zips to the same directory, so it uses a particular file path to decide if a given Make zip has been installed (8107c0d).

Can we explicitly specify that e.g. SWC Windows installer works with this version of Git installer, and prevent it from trying to install newer version

In concept we could do that. From experience it wouldn't work to just say "get version 1.3" because a lot of students will just hit the first download button they see (or be overwhelmed by a list of choices), so we'd have to hard link to it from the workshop website if possible. Looking at the repo/releases 1 hard links seem possible and are likely to be pretty stable at least for a while.

I expect this is more trouble than it's worth. The shift to MSys2 is pretty major, and we've had a few bumps while we adjust, but I don't expect the Git for Windows folks to make another major tooling change like that in the next few years. On one hand, if we get automatic integration testing, it would make this sort of thing reasonably cheap. On the other hand, anything we offload to instructors for website configuration is one more thing the may forget to do (e.g. commenting CHECKS in the current installation tester, which I'm trying to address with wking/swc-setup-installation-test#9).

[olexandr-konovalov]

Thanks @ethanwhite and @wking, this makes things more clear. I presume it should it be fine to test the installer on the same machine where it is built? When building cygwin applications, we test them on a different machine which is Cygwin free, but I can't see the reasons to do the same here. Or am I missing something, perhaps?

[olexandr-konovalov]

I have set up Jenkins test to build the installer and archive the resulting SWCarpentryInstaller.exe artifact. If the build will fail, I will be notified by email. For now, it is scheduled to run daily (and it takes less than a minute to run). I've just sent build log to @ethanwhite and @wking by email. It seems to work, and I've tried the installer (on the same machine), and it seems to work as well.

There is an obstacle for setting up an automated test of the installer - it requires human intervention to click buttons and enter admin password... Are there any ways to run it in the command line?

[ethanwhite]

I presume it should it be fine to test the installer on the same machine where it is built?

Yes

Are there any ways to run it in the command line?

If you just want to run the Python script then it can be run from the command line. I don't know if it's possible to run an Inno Setup installer from the command line. Presumably there are tools to facilitate this sort of thing for remotely deploying changes to multiple Windows machines, but that's outside of my area of expertise.

[gvwilson]

Is this one still live?

[olexandr-konovalov]

@gvwilson yes, this runs regularly, in the state reported on Nov 25th. Would be good to improve it though - it builds the installer, but does not have any automated checks that it actually works - to do that, one should login into Jenkins, download the build artefact and start it manually...