(+) module testcases using the unittest functionality

[korobkin]

@2sn @fherwig @adam-paul I've started a new branch to collect testcases, with a separate module NuGridPy/testcases.py, which uses the unittest functionality (see unittest docs). unittest allows various interesting ways to run the test suites, the simplest way is to run a command:

   python -m NuGridPy.testcases

(the parent directory of NuGridPy must be the PYTHONPATH). What do you think?

[2sn]

@korobkin Oleg, may we have test data in a separate git repo? yt, e.g., just downloads test data from a server to temp directory using a plain url. That would also be an option. Cache the data locally, in temp directory (python has support for temporary files and directories).

[korobkin]

@2sn Yes, absolutely. We could start a new tests repository, e.g. TestData, just for storing the tests... But I think that having a module testsuites, or perhaps selftest, in NuGridPy, which is just a lightweight master test runner, is not a bad idea. A user could run,

   python -m NuGridPy.selftest

and the test module automatically downloads detailed unit tests from the TestData url, maintain cache and cleans up after the tests are done. Maybe later we can share this functionality to other modules in the project.

BTW at last, my first successful NuGrid plot ...

[fherwig]

@korobkin - are this ppn test? Single-zone network output? You can find test data on the our NuGrid CADC VOspace storage, I am in the processes of transferring a library of 1-zone i-process simulations: data-team/iprocess-nucleosynthesis/iprocess-1zone/Z2e-2XH0p10T200rho1e2 - check ABout NuGrid for members web page on the plone for howto access.

[korobkin]

@fherwig these are only tests of python scripts from NuGridPy with some output datafiles that I copied from elsewhere. Does this work for you? If you check out this branch, set PYTHONPATH to the parent directory of NuGridPy and run,

   python -m NuGridPy.selftest

it should correctl make two plots, provided you are in the directory with abundances datafiles.

Thanks for the link! I will work on remaking these tests to download data from your storage.

[2sn]

@korobkin This is great. I agree, the test suite should be part of NuGridPy and is well organised to live in its own sub-module.
@korobkin yes, having a separate repo for the data is good, alternatively could be a web page/url @fherwig using CADC is not an option because of the lack of python3 integration. Let's go for a simple solution for now. git stuff can be checked out using python, I have even written an extended interface for git (to import all my KEPLER history from tgz files and cvs).

[fherwig]

@korobkin ok - you have some data to plot, good. The one I am transferring may go away again, so not good for long-term solution.

How about having the nuppn test suite create test data? We do have nugrid servers that we can ask to do whatever we want. We could ask these servers to execute the nuppn tests on a regular basis, followed by the nugridpy tests. maybe this an be integrated with the nice travis interface. I may discuss this with @adam-paul tomorrow.

[fherwig]

@2sn CADC VOspace does have a web interface to pull data from. try http://www.canfar.phys.uvic.ca/vosui/#/nugrid

you can pull any file from there individually using wget if you like.

I am just arguing against a multiplication of sites and things we have. If we have a solution for something we should use it, unless there is a good reason for it.

[fherwig]

@2sn Re CADC VOspace and python 2/3 - I assume we are performing these tests both under python2 and 3 - right? @adam-paul ?!?!

It turns out that we do have test data for plotting already on github: https://github.com/NuGrid/NuPPN/tree/master/examples/ppn_Hburn_simple see Notebook and RUN_MASTER dirs. @korobkin - can you execute the Notebook with the new merge? In both python2 and 3? @OClarkson

[2sn]

@korobkin as I write above, CADC may not work, let's use a git repo for the test data for now. @fherwig :(

On another note, I have not found out how I can reply to specific comments in the history on the github site. Any ideas?

[2sn]

@fherwig OK, for the test cases we should pull data using the web interface. All good with that then. The main test suite will not be a notebook but using python's unittest - all command line for portability and automated tools, as done by @korobkin . It is OK to have examples & tutorials in notebooks, of course, but not the test suite we every time the code is changed, there is a new checking, etc.

[2sn]

@korobkin Where do I get the data files? Ideally, they should be downloaded to some temporary directory (using python's tempfile)

~/NuGrid>python3 -m NuGridPy.selftest
Error: no iso_massf named files exist in Directory
Now returning None
EE.........
======================================================================
ERROR: test_abu_chart (__main__.TestAbuChart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/NuGrid/NuGridPy/selftest.py", line 45, in test_abu_chart
    raise IOError("Cannot locate a set of abundance data files")
OSError: Cannot locate a set of abundance data files

======================================================================
ERROR: test_abu_evolution (__main__.TestAbuChart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/NuGrid/NuGridPy/selftest.py", line 72, in test_abu_evolution
    x=ppn.xtime('.')
  File "/home/alex/NuGrid/NuGridPy/ppn.py", line 147, in __init__
    self._readFile(fname,sldir)
  File "/home/alex/NuGrid/NuGridPy/ppn.py", line 160, in _readFile
    f=open(fname,'r')
FileNotFoundError: [Errno 2] No such file or directory: './x-time.dat'

----------------------------------------------------------------------
Ran 11 tests in 0.314s

FAILED (errors=2)

very nice!

[fherwig]

@2sn

On another note, I have not found out how I can reply to specific comments in the history on the github site. Any ideas?

like this? use the "Insert quote button and copy-paste.

One more thing @2sn and @korobkin - I am happy to try the new version, but you need to

put some notes into the nugridpy wiki how users should go about using nugridpy now? Some examples etc would be useful. I think that would be essential.

[2sn]

@fherwig

I am happy to try the new version, but you need to put some notes into the nugridpy wiki how users should go about using nugridpy now? Some examples etc would be useful. I think that would be essential.

as we write, NuGridPy itself should not be in the PYTHONPATH but its parent directory. Then it heaves like any other module, but now is the NuGridPy module. So you would write

from NuGridPy import utils as u
from NuGridPy import *
from NuGridPy.data_plot import DataPlot

etc.

@korobkin @adam-paul @fherwig Oleg did a great job starting the test suite setup as discussed during the nugrid meeting, Adam should use this framework for the test service.

@fherwig A next step will now be to make NuGriPy a package for pipy so it can be installed using pip. I have never done this...this is also what I need to do for the things needed to the decay network (Falk, your other email). I will probably first do this for the FORTRAN binary reader module that was discussed to replace h5py - much faster and easier integration with both Python and FORTRAN. Currently we are, e.g., stuck with lack of support upgrading to support unicode strings native to Python3 in the FORTRAN reader (OK, it was not really a big deal to change this - I do not know, though, where the se.py module for this lives, I have my own). Important will be to agree on a file format that includes a version tag (number), maybe a file type ID as well, at the very beginning. This has served me well in the past, I can still read KEPLER dumps from the late 80ies. Maybe this should become a separate thread - NuGridPy-dev / NuGridPy-ideas.

@fherwig @marcop46 @sam How will versioning, releases, branches, devel, tagging, etc., be done with NuGridPy? What is policy on this?

[2sn]

@fherwig

I am happy to try the new version, but you need to put some notes into the nugridpy wiki how users should go about using nugridpy now? Some examples etc would be useful. I think that would be essential.

the current wiki doc looks OK to me.

[korobkin]

@fherwig

I am happy to try the new version, but you need to put some notes into the nugridpy wiki how users should go about using nugridpy now? Some examples etc would be useful. I think that would be essential.

I've added some notes into the nugridpy wiki -- basically just reused the README file from master branch. Once we have #7 (unittest) working, I'll add test instructions to the wiki too.

[2sn]

@korobkin If you can provide the test data you used, I can check the suite as well if you like.

[2sn]

@korobkin

~/NuGrid>python2 -m NuGridPy.selftest
~/NuGrid>python2 -m NuGridPy.selftest
error: Directory, /NuPPN/examples/ppn_C13_pocket/master_results not found
Now returning None
Eerror: Directory, /NuPPN/examples/ppn_Hburn_simple/RUN_MASTER not found
Now returning None
E.........
======================================================================
ERROR: test_abu_chart (__main__.TestAbuChart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/NuGrid/NuGridPy/selftest.py", line 46, in test_abu_chart
    raise IOError("Cannot locate a set of abundance data files")
IOError: Cannot locate a set of abundance data files

======================================================================
ERROR: test_abu_evolution (__main__.TestAbuChart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/NuGrid/NuGridPy/selftest.py", line 82, in test_abu_evolution
    x.plot('time',spec,logy=True,logx=True,shape=utils.linestyle(i)[0],show=False,title='')
  File "NuGridPy/data_plot.py", line 434, in plot
    listY=self.get(atriy)
  File "NuGridPy/ppn.py", line 204, in get
    data_column=zeros(self.ilines)
AttributeError: 'xtime' object has no attribute 'ilines'

----------------------------------------------------------------------

I suppose I still do not have the test data

[adam-paul]

@korobkin

I am also seeing the KeyError:

======================================================================
ERROR: test_abu_evolution (__main__.TestAbuChart)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/npod1/home/adampaul/NuGridPy/selftest.py", line 82, in test_abu_evolution
    x.plot('time',spec,logy=True,logx=True,shape=utils.linestyle(i)[0],show=False,title='')
  File "/astro/adampaul/NuGridPy/data_plot.py", line 434, in plot
    listY=self.get(atriy)
  File "/astro/adampaul/NuGridPy/ppn.py", line 206, in get
    data_column[i]=self.data[i][self.col_num[col_str]]
KeyError: 'PROT'

The issue is in the get method of the xtime class and I guess PROT refers to proton number. I suspect that for our test data, 'PROT' is being returned as None and the test can't handle that data type.

[adam-paul]

@2sn @korobkin @fherwig

Sorry for the delay, I had a lot of other stuff on the go and there was lots of debugging to do on the Travis end of things also. The unittest functionality is now working well with Travis CI automation and I have added some first-draft documentation to both the NuGridDoc repo and the NuGridPy wiki outlining the regression testing schema.

In any case, could we now merge this pull request and begin development of new tests on their own branches?

[2sn]

Sorry for the delay, I had a lot of other stuff on the go and there was lots of debugging to do on the Travis end of things also. The unittest functionality is now working well with Travis CI automation and I have added some first-draft documentation to both the NuGridDoc repo and the NuGridPy wiki outlining the regression testing schema.

@fherwig @adam-paul There should be only one place where things need to be added? At least not the same or similar info.

[adam-paul]

@2sn

Yes I agree with that. The way the documentation is set up now is that in NuGridDoc is the general information regarding regression tests and Travis CI (e.g. logging in and integrating it with GitHub, links to Travis CI documentation, etc.), and in the NuGridPy wiki and other specific module wikis is information on how to build a particular regression test for that module.

[adam-paul]

@fherwig

This branch can now be merged with master once the error you planted by copying one of the AbuChart files with the wrong name is corrected! I am not sure how to access the VOspace exactly to do that or if I have the adequate permissions. Feel free to do it though and then merge.

[fherwig]

@adam-paul build #64.1 (and even less so #64.2) do fail but if this is because of the planted error, why are we not told about the error? It seems we do not see an error message from that test specifically. Is this just the way it is? Or are we missing something?

[adam-paul]

@fherwig

How do you mean? Here is the unittest error message in the Travis build:

This indicates that the test which is failing is the "test_ppnHburn_abucharts" test in the ImageCompare class.

[adam-paul]

@fherwig

Should we remove the error and merge?