search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
707 stars 37 forks source link

[REVIEW]: dlmmc: Dynamical linear model regression for atmospheric time-series analysis #1157

Closed whedon closed 5 years ago

whedon commented 5 years ago

Submitting author: @justinalsing (Justin Alsing) Repository: https://github.com/justinalsing/dlmmc Version: 1.1 Editor: @danielskatz Reviewer: @Chilipp, @taqtiqa-mark Archive: 10.5281/zenodo.2660704

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/af268dfacc4bff0fd37a70602d9e74b6"><img src="http://joss.theoj.org/papers/af268dfacc4bff0fd37a70602d9e74b6/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/af268dfacc4bff0fd37a70602d9e74b6/status.svg)](http://joss.theoj.org/papers/af268dfacc4bff0fd37a70602d9e74b6)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@Chilipp & @taqtiqa-mark, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @danielskatz know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @Chilipp

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @taqtiqa-mark

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 5 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @Chilipp, it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

danielskatz commented 5 years ago

@Chilipp, @taqtiqa-mark - thanks for agreeing to review. (Note that @Chilipp will not be able to start for a week.)

If you have any questions, please ask. Otherwise, I'll expect that you will work through the checklists above and raise issues as you find items you cannot check off.

danielskatz commented 5 years ago

👋 @Chilipp, @taqtiqa-mark - I hope you can start reviewing this soon - at least, the code of conduct and conflict of interest boxes for each of you...

Chilipp commented 5 years ago

Yes, @danielskatz, I will start tomorrow. My apologies for the delay

danielskatz commented 5 years ago

👋 @taqtiqa-mark - are you able to get started too?

Chilipp commented 5 years ago

Hello @justinalsing! I am so far done with my first look into your software and I would like to have your feedback on some of the following points:

General checks

Functionality

Documentation

Software paper

danielskatz commented 5 years ago

👋 @justinalsing - note that we're waiting for your inputs here

justinalsing commented 5 years ago

Hi @Chillip, @danielskatz, @taqtiqa-mark

Thanks for all the feedback and sorry for the delay. Let me try to address your points:

Version thanks for pointing this out! I’ve just made a release, v1.0

Installation I saw you had problems installing with pip. The dependency that typically causes problems (as in your case) is pyston, since it needs to play nicely with the C++ compilers on your machine. For some it has worked with pip, for others it seems to work with conda but not pip. I’ve added a note to the README that says you can use “conda install (recommended) or pip” and another note, “Note: conda seems to be more stable for getting pystan to work than pip - if you installed with pip and compiling the stan models throws errors, try again with with conda.” Do you have any other suggestions for making the install instructions more solid?

I’ve also added a point in the README that prior knowledge of python/jupyter is needed to be able to fully follow the tutorial, as suggested. Agree this was a good idea.

Statement of need yes, I intend that the HMC-Kalman filtering framework can be used to expand the suite of models beyond those initially provided. Essentially the space of DLM models are characterized by a few model matrices; more-or-less all that needs modifying in the stan model code to implement a new one is to replace the model matrices and add any additional hyper-parameters. I’ve now provided a CONTRIBUTING file as you suggested giving details of how to contribute extended models to the suite. It does assume familiarity with stan, which I think is needed to contribute meaningfully. I’d be happy to give further details here if you feel they are needed and am very open to suggestions on how to improve this.

Automated tests In my mind the tutorial also serves this purpose: if the models compile and the tutorial runs through without error, you’re in good shape. I’ve added a note to the README that the tutorial also serves this purpose. However, if you feel additional tests are required I am open to suggestions..

Community guidelines see above re added CONTRIBUTING file.

References I’ve updated the references in the model_descriptions.pdf as suggested (thanks for pointing those out)

I hope this addresses your points and look forward to hearing your further thoughts



Thanks again!

taqtiqa-mark commented 5 years ago

My apologies for the delay. It took a while to find my first notes. which follow:

I've had a quick look at the model PDF and some of the repository and can say, while its fresh in my mind:

The use-case appears to be reasonably well motivated - implementing a known/published model, and a non-trivial software stack (e.g. parallel libraries).

My working assumption is the question of originality is out-of-scope - it suffices that the library implements a model already published. Consequently, while non-linear AR/(G)ARCH models are wide spread, I haven't investigated the question of whether this improves on them. However I would suggest that data sets used thought to have those properties from the R-project, would be used in the test suite.

Given the JOSS is software focused I'm placing all emphasis on the software engineering aspects, and almost non on the contribution to domain knowledge.

That non-trivial software stack nature suggests weight be given to the following observations:

I'll update this with links to the issues opened against the upstream project, as I open them.

@danielskatz, for me blocking issues are:

  1. Automated test suite, that at a minimum:
    • Fits null and alternate specifications to simulated and empirical data. The tests would be simply checking parameters vales, and goodness of fit metrics. The 'true' or expected values should come from a reference implementation that is clearly identified, e.g. some R-project, etc. library result set, and the script used to generate the reference values would be part of the project.
    • Parallel test suite that exercises the parallel calculations. This would likely be slow (i.e. while processing large data or models is faster than serial runs the parallel run would be expected to be 'slow' in calendar time - unit test times are generally aimed to be in the sub-seconds range) so would likely have its own test harness to prevent the TDD/BDD cycle from being slowed down.
  2. Installation instructions for the platforms that are supported. My assumption is it is not the purpose of the reviewers to resolve all these issues across the various platforms? The warning note (should be more prominent) about conda instability and switching between pip/conda is really something that the package should resolve and provide the user with a painless path?
  3. Documentation. Each function a user is expected to interact with should be documented. Ack. that many functions are likely to be upstream, but if the user is expected to interact with them, the package document should, at a minimum, provide links to the upstream documentation of the version in use.
danielskatz commented 5 years ago

Thanks @taqtiqa-mark - these details are very helpful.

In JOSS, we don't really have results like

Right now I'd be inclined to conclude: Major Revisions, resubmit.

We just say what's needed to be done, and then let the author decide to do this or not. In other words, we keep going until the submission is acceptable or we decide that it cannot be made so.

justinalsing commented 5 years ago

Hi @taqtiqa-mark ,

Thanks for your feedback.

I think it would help it I reiterate/clarify the scope of this code. The code is a suite of statistical models, written in stan, than can then be ran on user’s data using the pystan interface (provided by the stan team).

Along with the models, I provide a tutorial covering how to run the stan models using pystan — this is really to ease the way for users who are not familiar with pystan. For users who want to run many DLMs quickly but are not used to setting up MPI runs with python, I also provide a template for how to run models in parallel this way. The tutorial and MPI-run template are really examples of how to use the code to help users get up and running quickly, rather than central to the package. That is why, for example, I do not attempt to give openmpi install instructions that will work robustly for everyone — as you point out that is a difficult thing to do in practice.

Since the package is a suite of models implemented in a probabilistic-programming language, the model code “in of itself” does not have any “functionality” to be documented, ie., the model codes in stan_dlm_models.py are not a set of functions that can be called by the user. Rather, they are specifications of probabilistic models that are then passed to the pystan interface, which then runs them on user input data. I am intentionally relying on the pystan interface to run the models, since it already exists and is well documented. I could have implemented a higher level interface so that users would not have to interact with pystan at all, but this seemed unnecessary and would make maintaining the code more cumbersome.

There is a question of whether a suite of stan models constitutes a valid open source project. I believe it does for the following reasons: — The (stratospheric science) community has explicitly asked for this, since the expertise/time to implement such models within the community is scarce, and the momentum to replace the industry-standard MLR with DLM is rapidly growing. The code is already being used by members of the community. — The code is extendable in a meaningful way: people can implement new models based on the dlmmc code with relatively little effort, since it essentially just involves modifying the lines of code that specify the model matrices that define the space of DLM models. The bulk of the work (setting up the Kalman-filtering and smoothing, etc) has been coded up in a general-purpose way so does not need modifying, massively reducing the barrier to entry for people to implement new models.

I hope this focuses the discussion on what I should do to make this code ready for public consumption.

@taqtiqa-mark and @Chillip, you both raised automated tests and more robust install instructions. If you have some concrete ideas/specifications for what you think is needed here I’d greatly appreciate it.

The other related point you raised @taqtiqa-mark is validation. Validation of Bayesian codes is non-trivial, requiring either comparison to a known analytical posterior (hard in this case), comparing to another code, or running on a large ensemble of mock data vectors and looking at the statistical properties of the results (expensive). If you really feel this is a requirement I’d love to hear what you think would be most worthwhile here.

taqtiqa-mark commented 5 years ago

@danielskatz,

I've amended my comment o remove that observation.

@justinalsing makes the following observation:

There is a question of whether a suite of stan models constitutes a valid open source project

My assumption is that this question has been resolved in the affirmative by the (sub-)editor(s) - otherwise this would not have progressed to the review stage.
Please indicate if that assumption is correct.

If incorrect, are there JOSS guidelines to on what is a "valid open source project" (for the purposes of JOSS) to guide reviewers to address this issue in a way that results in consistent outcomes?

I'll address @justinalsing response separately - by editing my comments above to add links to issue in his repository - to prevent this thread from becoming spaghetti.

danielskatz commented 5 years ago

There is a question of whether a suite of stan models constitutes a valid open source project

My assumption is that this question has been resolved in the affirmative by the (sub-)editor(s) - otherwise this would not have progressed to the review stage. Please indicate if that assumption is correct.

Yes, this is correct.

danielskatz commented 5 years ago

Also, @taqtiqa-mark - I think that rather than editing comments to address responses, it's better to just write a new comment, and, as you've said, to open issues in the source repository with links to them here. And so, to get an up-to-date status, can you (and also @Chilipp) please add new comments below that tell us what your current concerns are (linking to issues as needed)?

Chilipp commented 5 years ago

thanks @justinalsing for your detailed responses to my comments and the ones from @taqtiqa-mark ! I especially appreciate your work on the contribution guidelines.

There are still some open points and I would like to outsource them to other issues in your repository:

If you work on these issues, please directly comment in the issues in your repository as this is also recommended in the review process guidelines above

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

danielskatz commented 5 years ago

Re: automated tests - The JOSS requirement is just "Are there automated tests or manual steps described so that the function of the software can be verified?" While automated tests are generally better for the software, manual steps are sufficient for JOSS acceptance. Specifically, adding good automated tests can be a longer-term activity than JOSS acceptance.

Chilipp commented 5 years ago

Thanks @danielskatz for your clarification on this topic. @justinalsing states above (https://github.com/openjournals/joss-reviews/issues/1157#issuecomment-459128351)

In my mind the tutorial also serves this purpose: if the models compile and the tutorial runs through without error, you’re in good shape. I’ve added a note to the README that the tutorial also serves this purpose.

Would this be sufficient for an acceptance in JOSS? Because the feeling of @taqtiqa-mark and me is that there should be some more tests implemented (see https://github.com/justinalsing/dlmmc/issues/5)

justinalsing commented 5 years ago

Hi @Chilipp , @danielskatz , @taqtiqa-mark ,

Thanks for your detailed comments and for solidifying the action-points into concrete issues, I'll get to these ASAP.

Thanks for re-raising exactly what is sufficient for JOSS acceptance re tests. Given Daniel's comment about manual vs automated tests, I was going to suggest that the path of least resistance for me would be to beef up the tutorial to (a) have sections to run all of the implemented models rather than just the "vanilla" one, and (b) run each model on mock data generated from the same model so that the examples serve as explicit validation tests in addition to testing whether the code runs. If that sounds sufficient I'll go ahead and put those in there - otherwise it'll take a good while longer for me to implement an automated test suite.

Thanks!

danielskatz commented 5 years ago

I believe this would be reasonable.

justinalsing commented 5 years ago

Great - I'll wait for confirmation from the reviewers @Chilipp and @taqtiqa-mark that they would be satisfied with this and then go ahead and implement

danielskatz commented 5 years ago

👋 @Chilipp & @taqtiqa-mark - note that we're waiting on your inputs here...

Chilipp commented 5 years ago

Hey @danielskatz and @justinalsing. Sorry. Sure, if that is sufficient for the criteria of JOSS, I won't make a stand against it

danielskatz commented 5 years ago

👋 @taqtiqa-mark - can you let us know if this is acceptable to you?

... to beef up the tutorial to (a) have sections to run all of the implemented models rather than just the "vanilla" one, and (b) run each model on mock data generated from the same model so that the examples serve as explicit validation tests in addition to testing whether the code runs.

In my opinion, this would meet the JOSS criteria.

Also, how is your review coming otherwise?

justinalsing commented 5 years ago

I plan on proceeding with the above plan anyway (and can add more later if needed), just need to find time to sit down and do this, hopefully next week (maybe sooner)

danielskatz commented 5 years ago

Hi - I'm going to be on vacation for 2 1/2 weeks, so I'm shifting the editor of this submission to be @arfon, during that period - thanks @arfon

danielskatz commented 5 years ago

@whedon assign @arfon as editor

justinalsing commented 5 years ago

Hi All,

Following the discussion above about automated tests, I've included an additional notebook with validation tests for each of the DLM models implemented in the package - I've given some more details in https://github.com/justinalsing/dlmmc/issues/5

Hope this does the trick!

Thanks, Justin

PS @afron, above it looks like the two reviewers were unassigned when Daniel transferred this to you as editor? Does this need fixing? (thanks!)

arfon commented 5 years ago

PS @afron, above it looks like the two reviewers were unassigned when Daniel transferred this to you as editor? Does this need fixing? (thanks!)

Yes, I think that might be a bug.

@Chilipp, @taqtiqa-mark - please come and take another look at this submission following @justinalsing's updates.

danielskatz commented 5 years ago

@whedon assign @danielskatz as editor

danielskatz commented 5 years ago

Thanks @arfon for the help while I was on holiday

danielskatz commented 5 years ago

👋 @taqtiqa-mark - where are you on your review?

taqtiqa-mark commented 5 years ago

@danielskatz, In light of the feedback I'd say I'm substantively finished the review. I'm happy with the open tickets tracking the outstanding issues.

I'm about to work through the installation instructions and run the examples-as-test-suite... which raise a question: Rather than installation instructions etc, would JOSS accept a link(s) VM image(s) that is publicly available? e.g. https://app.vagrantup.com/boxes/search?utf8=%E2%9C%93&sort=downloads&provider=&q=python3

The reason I ask is, the author has indicated he had done something like start with a vanilla VM and tested his that his installation was able to run the examples.
My experience tells me that the current installation/configuration instructions are no where near enough to get a new user up and running without considerable trial and error and likely opening issues - not ideal given the audience for this publication are domain SMEs.

If it was acceptable for the author to provide his VM I expect we are all going to be saved considerable time and effort - particularly w.r.t. to the MPI setup.

justinalsing commented 5 years ago

Hi @taqtiqa-mark - super, thanks for the message. Regarding the VM - looking back at the thread, maybe I answered in haste earlier. All I did at that point was re-clone the repo and run everything through (including re-compiling the models, but not re-installing dependencies), I didn't do this in a VM.

I could definitely try what you suggest, though - I guess I'd have to do this for a bunch of different operating systems?

I see the two installation hurdles that have the potential to cause issues as pystan and mpi4py (although I reiterate that the MPI template example I included is really intended as a "bonus" to help people scale-up their runs, rather than an integral part of the package/suite of DLM models). I guess I'm not sure I'll be able to (realistically) provide more comprehensive support for stan and mpi4py than is already provided by the vendors of those packages? But if you think I can, I'll all ears

taqtiqa-mark commented 5 years ago

I guess I'd have to do this for a bunch of different operating systems?

This is up to you. What OS users will you support? What OS's have you tested this with?

I think it is reasonable to expect that you have tried your instructions starting from a "clean" system.

A VM is the most natural and easiest way to get a clean system. Please record or link to the VM image you start from.

You could try to nuke your bare-metal installation, but in my experience you tend to forget what you'd done in the past to make things work - I think this will end up being more work for you.

Please copy and paste all commands used, this will be your install script/instructions. Once you get it working for one OS anyone can adapt/extend your script for their environment.

... the MPI template example I included is really intended as a "bonus" to help people scale-up their runs, rather than an integral part of the package/suite of DLM models).

Okay, I have no horse in this race, so you need to decide: Are you willing to support MPI users or not?

danielskatz commented 5 years ago

👋 @justinalsing - I think this is waiting on you at this point

danielskatz commented 5 years ago

👋 @taqtiqa-mark - I wonder if there are some more items in your review checklist that you can check off? This would help make clear what the blocking items are.

taqtiqa-mark commented 5 years ago

Checklist updated.
Blocking items (Given the last comment I'm assuming the MPI/parallel use cases are out of scope).

justinalsing commented 5 years ago

Hi @danielskatz @taqtiqa-mark -

Yep sorry for the delay, I'm not well versed in using VMs or containers so there is some overhead for me to do this, I will try to find time to get this done asap

Best, Justin

justinalsing commented 5 years ago

Hi @taqtiqa-mark -

I haven't had time to set up the VM etc yet (I had a go but had some trouble getting VirtualBox working on my machine and ran out of time). However, I am going to help some new users get installed and working with the code this week or next - is there anything I could provide you from this process that would satisfy your last checkpoint re validating the installation instructions?

Best, Justin

justinalsing commented 5 years ago

Hi @taqtiqa-mark

OK so I've now done the following: created an isolated python environment using virtualenv, then followed my install instructions:

pip install numpy scipy matplotlib netCDF4 pystan git clone https://github.com/justinalsing/dlmmc.git cd dlmmc python compile_stan_models.py

and then tested that the jupyter notebook dlm_tutorial.ipynb runs through without errors by doing

jupyter-nbconvert --to notebook --execute --ExecutePreprocessor.timeout=100000 dlm_tutorial.ipynb

This all worked fine - I attach a text file with the terminal session for all of the above (from creating the virtual environment through to running the test).

One point of contention though: I used pip to install dependencies (which works fine for me and other colleagues who use pystan extensively). However, @Chilipp did not get a working pystan set up when he did pip install, but he did with conda, so suggested that I change the install instructions to conda (which I duly did). But I don't use conda personally or have it set up

So, is this satisfactory for you, and any ideas for resolving the pip/conda issue? I'll note that any issues are isolated to the dependency pystan (which is well documented at https://pystan.readthedocs.io/en/latest/). I feel like providing the existing install instructions, and pointing users to the pystan docs in the README in case they have issues with that particular package, is appropriate for this?

Thanks, Justin

DLMMC install:test terminal output.txt

danielskatz commented 5 years ago

Hi @taqtiqa-mark and @justinalsing,

Just to make sure I understand where things are, it looks like Mark has created three issues

that are blocking his acceptance of the installation part of the submission, which is the last part that needs to be resolved.

And this is now in Justin's court to respond to them.

Is this correct?

I also see https://github.com/justinalsing/dlmmc/issues/5 is still open, but apparently this is not blocking anything for Mark.

justinalsing commented 5 years ago

Hi @danielskatz @taqtiqa-mark

I've responded to the remaining three issues. As far as I can see the install instructions seem to be working fine, having already been changed from pip to conda following @Chilipp's suggestion, and the MPI example script provided in the Git repo is not an integral part of this package (as has been discussed multiple times) so I personally don't feel that this should be a blocking issue for JOSS acceptance (could you weigh in on this @danielskatz?)

justinalsing commented 5 years ago

Hi @taqtiqa-mark - I've done some more work on the install instructions and responded to your outstanding issues. I hope we are getting close now!

taqtiqa-mark commented 5 years ago

@danielskatz: LGTM

danielskatz commented 5 years ago

👋 @justinalsing - can you make a new archive of the software in zenodo, figshare, etc.? And paste the DOI here? Also, if the version has changed, please let me know the new version number.

danielskatz commented 5 years ago

Thanks very much @taqtiqa-mark and @Chilipp for your work on this review!

justinalsing commented 5 years ago

Yes, thanks @taqtiqa-mark @Chilipp @danielskatz for your work and patience!

@danielskatz I've archived with Zenodo as advised - here's the DOI

DOI

and the new version number is v1.1

danielskatz commented 5 years ago

@whedon set archive 10.5281/zenodo.2660704