Add CI builds

[Chilipp]

hey @justinalsing! As part of the software review in https://github.com/openjournals/joss-reviews/issues/1157 I create this issue to discuss the installation of automated tests using Continuous integration services (such as Travis) (see my comment (https://github.com/openjournals/joss-reviews/issues/1157#issuecomment-455693322) and the one of @taqtiqa-mark (https://github.com/openjournals/joss-reviews/issues/1157#issuecomment-459463115).

In your README file you state that the tutorial serves as a test for your suite (see https://github.com/justinalsing/dlmmc/commit/ccf6b831de09f5a55a2b4ed4839c82b2df834756). Nevertheless, this is not an automated test and (in terms of transparency) you should use any of the available CI services that runs your tests on every commit (e.g. Travis CI). I know, this is tedious but all open source projects that you are relying on went through this process, so why shouldn't yours as well?

Minimal requirements for your automated tests are:

• [ ] Compile the models
• [ ] test all models in your package (not only the one provided in the tutorial)
• [ ] Execute the tutorial notebooks to make sure they run without any exceptions (e.g. by using jupyter nbconvert --execute on your notebooks

Additionally, citing https://github.com/openjournals/joss-reviews/issues/1157#issuecomment-459463115

• [x] You should definitely think about tests to check the mathematical correctness of your models since this is the central part of your code.

  Fits null and alternate specifications to simulated and empirical data. The tests would be simply checking parameters vales, and goodness of fit metrics. The 'true' or expected values should come from a reference implementation that is clearly identified, e.g. some R-project, etc. library result set, and the script used to generate the reference values would be part of the project.

  Comparing to some reference R-project, etc. would be ideal, of course. Alternatively you could just use the trend of your tutorial dataset (here you know, that this one is correct) and implement a unittest that runs your four models and compares it to this reference trend. You are the expert of your software so it's up to you which metric and which accuracy you are expecting of your model (but nevertheless, please justify your choice).

• [ ] Parallel test suite that exercises the parallel calculations. You could for example run your dlm_lat_alt_mpi_run.py script in your test suite

If you address this issue in a commit or pull request, please include the link to this issue (https://github.com/justinalsing/dlmmc/issues/5) in the commit/PR-message. In that way, it will be easier for us to follow the implementation.

[taqtiqa-mark]

I wonder if it isn't worth breaking this issue into three? To be completed in this order.

• Vanilla Test suite.
• Parallel test suite.
• CI build that executes the above.

@justinalsing, just a suggestion based on personal experience: Each of these can be non-trivial to do well, and all the moving parts can be overwhelming if you try to tackle them together.
Also be aware that CI build centric iterations can be much slower than running tests on your local machine. Generally the CI build of a branch involves all tests and is triggered by a commit that merges several changes from a scratch branch.... on your localhost you'll likely be running subsets of tests with regularity.

[justinalsing]

Hi @Chilipp @taqtiqa-mark

Following the discussion on the main review thread about the JOSS requirements, I’ve added a new notebook (dlm_validation_tests.ipynb) with a suite of validation tests, for each of the DLM models included in the package. For each model, I (1) Generate mock data from the model (for user specified input parameters), (2) Run the DLM model on the mock data, (3) Plot the inferred model parameters against the inputs, showing that the inputs are recovered (within posterior uncertainties).

I’ve added a section to the README pointing the user to this notebook as a set of tests/validation for the full set of implemented models. I’ve also added comments in the CONTRIBUTING file, guiding contributors to include a similar test/demo of any new models implemented in this notebook.

Regarding testing the MPI script, as I discussed on the main review thread I really included this to make non MPI-savvy users lives easier rather than as an integral/critical part of the package. But I’ve emphasized in the README that the MPI script can be tested by running it with certain parameters set so such that it runs very short MCMC chains (and hence runs through very quickly).

Look forward to hearing whether you think this is all OK!

Best, Justin

[Chilipp]

Thanks @justinalsing ! I just started and apparently there is a file missing: utils/dlm.py

# Import required modules
import pystan
import matplotlib.pyplot as plt
import numpy as np
import time
import scipy.interpolate as interpolate
import netCDF4
import pickle
import scipy.stats as stats
from utils.utils import *
from utils.dlm import *
%matplotlib inline

---------------------------------------------------------------------------

ModuleNotFoundError                       Traceback (most recent call last)

<ipython-input-1-890c5b3f6cfd> in <module>
      9 import scipy.stats as stats
     10 from utils.utils import *
---> 11 from utils.dlm import *
     12 get_ipython().run_line_magic('matplotlib', 'inline')

ModuleNotFoundError: No module named 'utils.dlm'

[justinalsing]

@Chilipp ah, thanks! I forgot to add it - I just added it now and pushed so it should be there now (sorry for that!)

[Chilipp]

Thank you @justinalsing. Unfortunately it it seems like I cannot run the test 2 for dlm_vanilla_ar2 because of the following error. Can you confirm this error?

    ---------------------------------------------------------------------------

    RuntimeError                              Traceback (most recent call last)

    <ipython-input-4-c33d4d85b5cc> in <module>
         64 
         65 # Ok, let's run it
    ---> 66 fit = dlm_model.sampling(data=input_data, iter=4000, warmup=1000, chains=1, init = [initial_state], verbose=True, pars=('sigma_trend', 'sigma_seas', 'sigma_AR', 'rhoAR1', 'rhoAR2', 'trend', 'slope', 'beta', 'seasonal', 'ar'))
         67 
         68 # Plot the results:

    ~/miniconda/envs/dlmmc/lib/python3.7/site-packages/pystan/model.py in sampling(self, data, pars, chains, iter, warmup, thin, seed, init, sample_file, diagnostic_file, verbose, algorithm, control, n_jobs, **kwargs)
        670 
        671         seed = pystan.misc._check_seed(seed)
    --> 672         fit = self.fit_class(data, seed)
        673 
        674         m_pars = fit._get_param_names()

    stanfit4anon_model_3780e43ed9ee465806912a94b90d6e06_7690838862458284755.pyx in stanfit4anon_model_3780e43ed9ee465806912a94b90d6e06_7690838862458284755.StanFit4Model.__cinit__()

    RuntimeError: Exception: variable does not exist; processing stage=data initialization; variable name=nx; base type=int  (in 'unknown file name' at line 6)

Here is some information about the working environment:

conda list

``` # packages in environment at /Users/psommer/miniconda/envs/dlmmc: # # Name Version Build Channel appnope 0.1.0 py37_1000 conda-forge attrs 19.1.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge blas 1.1 openblas conda-forge bleach 3.1.0 py_0 conda-forge bzip2 1.0.6 h1de35cc_1002 conda-forge ca-certificates 2019.3.9 hecc5488_0 conda-forge certifi 2019.3.9 py37_0 conda-forge cftime 1.0.3.4 py37h917ab60_1000 conda-forge curl 7.64.0 heae2a1f_2 conda-forge cycler 0.10.0 py_1 conda-forge cython 0.29.6 py37h0a44026_0 conda-forge decorator 4.3.2 py_0 conda-forge defusedxml 0.5.0 py_1 conda-forge entrypoints 0.3 py37_1000 conda-forge freetype 2.9.1 h597ad8a_1005 conda-forge hdf4 4.2.13 hf3c6af0_1002 conda-forge hdf5 1.10.4 nompi_h646315f_1105 conda-forge icu 58.2 h0a44026_1000 conda-forge ipykernel 5.1.0 py37h24bf2e0_1002 conda-forge ipython 7.3.0 py37h24bf2e0_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.4.2 py_0 conda-forge jedi 0.13.3 py37_0 conda-forge jinja2 2.10 py_1 conda-forge jpeg 9c h1de35cc_1001 conda-forge jsonschema 3.0.1 py37_0 conda-forge jupyter 1.0.0 py_1 conda-forge jupyter_client 5.2.4 py_3 conda-forge jupyter_console 6.0.0 py_0 conda-forge jupyter_core 4.4.0 py_0 conda-forge kiwisolver 1.0.1 py37h04f5b5a_1002 conda-forge krb5 1.16.3 hcfa6398_1001 conda-forge libcurl 7.64.0 he376013_2 conda-forge libcxx 7.0.0 h2d50403_1 conda-forge libedit 3.1.20170329 hcfe32e1_1001 conda-forge libffi 3.2.1 h0a44026_1005 conda-forge libgfortran 3.0.1 0 conda-forge libnetcdf 4.6.2 h6b88ef6_1001 conda-forge libpng 1.6.36 ha441bb4_1000 conda-forge libsodium 1.0.16 h1de35cc_1001 conda-forge libssh2 1.8.0 hb1dc21d_1004 conda-forge llvm-meta 7.0.0 0 conda-forge markupsafe 1.1.1 py37h1de35cc_0 conda-forge matplotlib 3.0.3 py37_0 conda-forge matplotlib-base 3.0.3 py37hf043ca5_0 conda-forge mistune 0.8.4 py37h1de35cc_1000 conda-forge mpi 1.0 openmpi conda-forge mpi4py 3.0.1 py37h27a7d74_0 conda-forge nbconvert 5.4.1 py_2 conda-forge nbformat 4.4.0 py_1 conda-forge ncurses 6.1 h0a44026_1002 conda-forge netcdf4 1.4.3.2 py37hf8bc7f3_0 conda-forge notebook 5.7.6 py37_0 conda-forge numpy 1.16.2 py37_blas_openblash486cb9f_0 [blas_openblas] conda-forge openblas 0.3.3 hdc02c5d_1001 conda-forge openmpi 3.1.3 h71abe9c_1001 conda-forge openssl 1.1.1b h1de35cc_1 conda-forge pandoc 2.6 1 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.3.4 py_0 conda-forge pexpect 4.6.0 py37_1000 conda-forge pickleshare 0.7.5 py37_1000 conda-forge prometheus_client 0.6.0 py_0 conda-forge prompt_toolkit 2.0.9 py_0 conda-forge ptyprocess 0.6.0 py37_1000 conda-forge pygments 2.3.1 py_0 conda-forge pyparsing 2.3.1 py_0 conda-forge pyqt 5.6.0 py37hc26a216_1008 conda-forge pyrsistent 0.14.11 py37h1de35cc_0 conda-forge pystan 2.17.1.0 py37hf8a1672_2 conda-forge python 3.7.1 hbdd33cc_1002 conda-forge python-dateutil 2.8.0 py_0 conda-forge pyzmq 18.0.1 py37h4cc6ddd_0 conda-forge qt 5.6.2 h822fa55_1013 conda-forge qtconsole 4.4.3 py_0 conda-forge readline 7.0 hcfe32e1_1001 conda-forge scipy 1.2.1 py37_blas_openblash486cb9f_0 [blas_openblas] conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 40.8.0 py37_0 conda-forge sip 4.18.1 py37h0a44026_1000 conda-forge six 1.12.0 py37_1000 conda-forge sqlite 3.26.0 h1765d9f_1001 conda-forge terminado 0.8.1 py37_1001 conda-forge testpath 0.4.2 py37_1000 conda-forge tk 8.6.9 ha441bb4_1000 conda-forge tornado 6.0.1 py37h1de35cc_0 conda-forge tqdm 4.31.1 py_0 conda-forge traitlets 4.3.2 py37_1000 conda-forge wcwidth 0.1.7 py_1 conda-forge webencodings 0.5.1 py_1 conda-forge widgetsnbextension 3.4.2 py37_1000 conda-forge xz 5.2.4 h1de35cc_1001 conda-forge zeromq 4.2.5 h0a44026_1006 conda-forge zlib 1.2.11 h1de35cc_1004 conda-forge ```

conda info

``` active environment : dlmmc active env location : /Users/psommer/miniconda/envs/dlmmc shell level : 2 user config file : /Users/psommer/.condarc populated config files : /Users/psommer/.condarc conda version : 4.6.7 conda-build version : 3.17.8 python version : 3.7.1.final.0 base environment : /Users/psommer/miniconda (writable) channel URLs : https://conda.anaconda.org/conda-forge/osx-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/osx-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/osx-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/osx-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /Users/psommer/miniconda/pkgs /Users/psommer/.conda/pkgs envs directories : /Users/psommer/miniconda/envs /Users/psommer/.conda/envs platform : osx-64 user-agent : conda/4.6.7 requests/2.21.0 CPython/3.7.1 Darwin/17.7.0 OSX/10.13.6 UID:GID : 502:20 netrc file : /Users/psommer/.netrc offline mode : False ```

[justinalsing]

Hi @Chilipp - sorry about that (I forgot to add the updated stan models on my last commit). I've added and pushed it, and tested end-to-end by re-cloning, re-compiling the models and running through the dlm validation notebook. Apologies for not doing this check beforehand - as far as I am aware it is fully functional now, just make sure you re-compile the models after you pull the latest, thanks!

[taqtiqa-mark]

Hi @justinalsing, I'll likely get time some evening to look at this in the coming week. Do I understand correctly that you've done something like:

1. Started a new VM (or container)
2. Followed your instructions to setup the software stack.
3. Followed your instructions to run your examples.
4. Observed no errors or missing steps.

Correct?

[justinalsing]

@taqtiqa-mark great, thanks! Yes that's correct

[Chilipp]

Dear @justinalsing,

sorry for being so slow in this. I ran through your tests and they run through. However, there was another issue with the filenames of your regressors in the notebooks. Please note that UNIX filesystems are case-sensitive, there either please change the filenames of your regressors or change them in the notebook.

As an example: your file name is regressors/ENSO_MEI_1950_201802.txt but in your notebook you use regressors/enso_mei_1950_201802.txt. The same happened in the dlm_tutorial notebook.

But I have to say that I like your tests and they are really useful to judge the quality of dlmmc.

[justinalsing]

Hi @Chilipp - not a problem at all, and thanks for the comment. I've now made the regressor filenames case consistent with the actual filenames in the notebook examples (and pushed the changes)

Thanks

[taqtiqa-mark]

CI builds are not a blocking issue for me.

[Chilipp]

Hi @justinalsing. Strictly speaking, this issue is not yet solved (although it does not affect the acceptance of https://github.com/openjournals/joss-reviews/issues/1157). But if you close it, I assume this means that you do not plan to add CI builds within the near future?

[justinalsing]

Hi @Chilipp - that's right, I don't envisage adding CI builds in the near future (although it is on my to do list)