[REVIEW]: Calliope: a multi-scale energy systems modelling framework

[whedon]

Submitting author: @sjpfenninger (Stefan Pfenninger) Repository: https://github.com/calliope-project/calliope Version: v0.6.2 Editor: @jedbrown Reviewer: @mdoucet, @gonsie, @ecotillasanchez Archive: 10.5281/zenodo.1262406

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/5db25929acbe2dd51e7e59078c000346"><img src="http://joss.theoj.org/papers/5db25929acbe2dd51e7e59078c000346/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/5db25929acbe2dd51e7e59078c000346/status.svg)](http://joss.theoj.org/papers/5db25929acbe2dd51e7e59078c000346)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@mdoucet & @gonsie & @ecotillasanchez, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @jedbrown know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @mdoucet

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: Does the release version given match the GitHub release (v0.6.2)?
• [x] Authorship: Has the submitting author (@sjpfenninger) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @gonsie

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: Does the release version given match the GitHub release (v0.6.2)?
• [x] Authorship: Has the submitting author (@sjpfenninger) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @ecotillasanchez

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: Does the release version given match the GitHub release (v0.6.2)?
• [x] Authorship: Has the submitting author (@sjpfenninger) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @mdoucet, it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

--> Check article proof :page_facing_up: <--

[mdoucet]

I've completed my review. While I'm not a subject matter expert of energy systems modeling, I found the documentation quite clear. I was able to go through examples without any issue. The code itself is well organized, with great use of continuous integration and automated testing.

[ecotillasanchez]

It was a pleasure to review this work by Pfenninger and Pickering. Excellent software and documentation. Please find below some suggestions:

• It is clear from the paper and the examples that the analysis scales well from urban to national networks. For a power systems user, however, it would be good to clarify that building a transmission model with Calliope is different than a traditional power flow analysis (Ybus, generation, demand -> dc or ac power flow equations -> voltage magnitude and angles).

• Similarly, perhaps note that the focus is power system planning (as opposed to operations).

• The tutorials cover a great amount of content, options, and optimization constraints. For a given network size, I would suggest a performance comparison between some of the external solvers, or simply some pointers to help users decide before finding possible roadblocks.

[gonsie]

Should the version number be updated to the latest release (v0.6.2)?

[sjpfenninger]

@gonsie Yes that would make sense. @jedbrown, should we modify the submission now or wait for all reviews to be completed?

[jedbrown]

I updated this thread to name v0.6.2. @gonsie It looks like the remaining items are in your review of Functionality.

[sjpfenninger]

I guess the paper also still says 0.6.0 - should I modify paper.md or do that at the very end of the review process?

[jedbrown]

We can do it at the end.

[gonsie]

I am unable the test the functionality of this due to issues with my work firewall (I am unable to install conda / dependencies). I may be able to review this on personal time (maybe another week or so), or I'm happy to defer to the other reviewers.

[jedbrown]

I notice the requirement ruamel.yaml<=0.15 which would allow 0.15.0, but not 0.15.1. Is that really intentional? I also notice that ruamel.yaml 0.15.0 is not compatible with Python-3.7 and that Calliope is not compatible with current ruamel.yaml (at least because it now returns a type that implements the dict interface, but is not an instance of dict).

I think the reviews above are sufficient, but wanted to give an opportunity to address version compatibility issues (if you so desire) because it would ease non-conda installation and coupling of Calliope with other software.

[sjpfenninger]

Thanks @jedbrown for raising the ruamel.yaml requirement issue. The version pinning is based on the ruamel.yaml README:

Starting with version 0.15.0 the way YAML files are loaded and dumped is changing. See the API doc for details. Currently existing functionality will throw a warning before being changed/removed. For production systems you should pin the version being used with ruamel.yaml<=0.15.

Not providing Python 3.7 compatibility is certainly issue. We will investigate this. I don't think there's a quick fix, as we'll have to make some changes to our internals.

[jedbrown]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[jedbrown]

I installed the software and followed one of the tutorials -- looks great.

@sjpfenninger Can you fix capitalization in the bib file? I think this is a result of a new JOSS style file, but proper names should be protected: "europe", "great britain", "ecos", "san diego, ca", "south africa", "python". When that is fixed, we'll be ready to archive. Thanks for your patience.

[sjpfenninger]

@jedbrown That should be fixed. I also bumped codemeta.json to v0.6.2 (the current version and the version that was online when the review started).

[jedbrown]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[jedbrown]

@sjpfenninger Looks good. Please archive your repository using Zenodo or similar and report the DOI here.

[sjpfenninger]

The generic Zenodo DOI is 10.5281/zenodo.593292 and the DOI for version 0.6.2 is 10.5281/zenodo.1262406 -- or do you need the specific commit that contains the accepted version of the JOSS paper to be archived?

[jedbrown]

Normally we ask to archive after the review issue is complete. In this case, I think the only review-relevant changes after your v0.6.2 tag is in the JOSS paper.bib. @arfon Do we need a fresh archive or is the v0.6.2 DOI sufficient?

[arfon]

@arfon Do we need a fresh archive or is the v0.6.2 DOI sufficient?

Should be sufficient as long as it represents the code associated with this submission (including any changes from the review).

[jedbrown]

@whedon set 10.5281/zenodo.1262406 as archive

[whedon]

OK. 10.5281/zenodo.1262406 is the archive.

[jedbrown]

@arfon Over to you. Thanks to @ecotillasanchez, @mdoucet, and @gonsie for your reviews.

[arfon]

@ecotillasanchez, @mdoucet, @gonsie many thanks for your reviews here and to @jedbrown for editing this submission ✨

@sjpfenninger - your paper is now accepted into JOSS and your DOI is https://doi.org/10.21105/joss.00825 :zap: :rocket: :boom:

[whedon]

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.00825/status.svg)](https://doi.org/10.21105/joss.00825)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.00825">
  <img src="http://joss.theoj.org/papers/10.21105/joss.00825/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.00825/status.svg
   :target: https://doi.org/10.21105/joss.00825

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://www.flipcause.com/secure/cause_pdetails/Mjk3Nzk=

[sjpfenninger]

Thanks @jedbrown, @ecotillasanchez, @mdoucet, @gonsie for your reviews/comments/editing!