Refactoring regression tests

[pllim]

Currently, SSB regression tests live in a private and hard to access SVN repository. Cron jobs are set up on certain machines (Mac OSX, RHEL6, and RHEL5) to update test scripts from that repo once a day and run with pandokia, compare results with "truth" images/tables, and generate HTML reports. Developers then look at the reports and can choose to "okify" failed tests (i.e., make the new results the new "truth").

Pros

• It is already working.
• It is familiar to us.
• pandokia is developed in-house (free, no licensing issues).

Cons

• pandokia has limited documentation. Some internal wiki instructions but that's about it.
• pandokia lacks Python 3 support, as it was written about 10 years ago.
• pandokia is not flexible in letting tests reading in big data from an arbitrary path. For example, CALWF3 input data are required to be on the current machine in the current directory.
• SVN repository not always in-sync across machines. For example, test updated on a machine and pushed to SVN repository might not get picked up by the other machine. This problem is suspected to be caused by SVN version is too old on one or more of the machines (but upgrading the SVN version would cause a different kind of problem).
• This system has a single point of failure, in the sense that only Joe or Christine (and perhaps Vicki if she has time) knows how to fix it. Okay, up to three points of failure.
• Not all exceptions are caught properly by pandokia. For example, a syntax error will result in the test being omitted from final report entirely (not to be confused with being reporting as missing in the report).
• Tests are disconnected from the actual codes that they are testing, in the sense that they are not under the same version control.
• A test messing up can affect unrelated tests. That is, the whole regression test system can fail because of a single test.
• Regression test codes are invisible to anyone outside SSB.
• It is difficult to access the current SVN (e.g., no Trac site).

Proposed Changes

• Move test codes out of the hidden and hard-to-access SVN repository back to the respective code repositories. For example, pysynphot tests go back to pysynphot GitHub repo. An exception can be made for HSTCAL (because it's written in C but its tests are in Python) and legacy codes (e.g., PyRAF, pytools). I am willing to absorb CALACS tests into acstools, but CALWF3 has opposing view. There is no reason why tests for all the different packages need to be in the same repository. It is trivial to skip/xfail a test requiring big data that is not present.
• Replace pandokia with a new test system. For example, a private version of Travis CI or Jenkins CI.
• Replace SVN with git, to be consistent with our recent move to GitHub. Also, this way, we do not need a special global account to modify the tests. Ideally, we can open pull requests and merge like we do with "real" codes.
• Switch from nose to pytest. As we move forward with more and more codes depending on Astropy or its template, this is unavoidable.
• Create a new test workflow. There is no reason for tests to run every night for all the packages. Tests should only be run when there are changes relevant to the codes being tested (e.g., a new commit or a new data file). Also, they must be able to be run easily manually as needed. For example, when there is a new pull request, we can checkout the codes from the pull request and just type py.test packagename [args] or python setup.py test packagename [args].
• Figure out a cleaner way to store input and output test data. While smaller data can live in the code repository (see first point above), big data should be somewhere else. Also we should address things like: Do we need to version control input data? Do we need to keep old "truths" once we "okify" new ones? How will the "okify" process work with Travis or Jenkins CI?
• While we're at it, we can review the affected tests and discard those that are outdated and do not make sense anymore. This will reduce maintenance costs going forward and lessen total run time.
• Replace RHEL5 and RHEL6 with RHEL7. Or at the very least, get rid of RHEL5. Also, if applicable, upgrade Mac OSX test machine.
• Do we even want to consider a Windows test machine? Maybe Windows 10?

c/c: @sosey @nden @cdsontag @jhunkeler @vglaidler @stsci-hack @justincely and whoever else in SSB that is interested

p/s: This is how I envision the change -- https://www.youtube.com/watch?v=mZ6_0wGGsuY

[sosey]

You need to explain what "Move regression tests back to respective repos" actually means

[pllim]

I thought the contents of this issue explained it, no?

[sosey]

no. The regression tests haven't gone anywhere to be moved back from. You're discussing changing the way in which we write, store, manage and execute our regression tests. That's a lot more than "move back to respective repos" unless you mean something entirely different, which should be explained.

[pllim]

Ok. How about now?

[sosey]

You just changed the title. Edit the original description to contain the current way we do tests and your suggested changes, that would be the most clear for everyone. Not everyone here is familiar with our current pandokia system.

[nden]

I agree it would be best to have a summary of what you propose. There may be other proposals for changes (or may be "wishes" is a better word) so it'll be easier to understand what each one involves without going through emails.

[pllim]

Thanks for the suggestions. I'll put something more coherent together when I get the chance today...

[pllim]

Okay. What about now? I think I have addressed all the comments and concerns. Thanks.