Closed whedon closed 2 years ago
Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @noah80, @robinroche it looks like you're currently assigned to review this paper :tada:.
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
:star: Important :star:
If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿
To fix this do the following two things:
For a list of things I can do to help you, just type:
@whedon commands
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
@whedon generate pdf
PDF failed to compile for issue #3596 with the following error:
Can't find any papers to compile :-(
Software report (experimental):
github.com/AlDanial/cloc v 1.88 T=0.65 s (98.5 files/s, 179070.6 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
XML 2 4 0 93528
Python 46 3347 7224 5812
SVG 2 0 0 5223
HTML 1 91 0 365
Markdown 1 47 0 264
reStructuredText 8 98 92 201
YAML 1 0 2 51
DOS Batch 1 8 1 26
make 1 4 7 9
INI 1 0 0 3
-------------------------------------------------------------------------------
SUM: 64 3599 7326 105482
-------------------------------------------------------------------------------
Statistical information for the repository '6a58250476f57617e864f97d' was
gathered on 2021/08/10.
The following historical commit information, by author, was found:
Author Commits Insertions Deletions % of changes
Ben Elliston 1 10 39 0.10
Nicholas Gorman 2 459 150 1.23
Nick 26 8124 6965 30.38
Nick Gorman 2 356 505 1.73
nick-gorman 111 24034 9032 66.57
Below are the number of rows from each author that have survived and are still
intact in the current revision:
Author Rows Stability Age % in comments
Ben Elliston 3 30.0 1.2 0.00
nick-gorman 16380 68.2 3.1 6.53
@whedon generate pdf from branch joss-paper
Attempting PDF compilation from custom branch joss-paper. Reticulating splines etc...
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@noah80, @robinroche, I'll add a reminder for both of you to notify you when half of the six weeks has elapsed.
@whedon remind @noah80 in three weeks
Reminder set for @noah80 in three weeks
@whedon remind @robinroche in three weeks
Reminder set for @robinroche in three weeks
/ooo September 4 until September 26
@robinroche, @noah80, I will be out of office for the duration mentioned above. If there are any questions from your side about this review, it would be great if you could ask them before I leave.
:wave: @robinroche, please update us on how your review is going (this is an automated reminder).
:wave: @noah80, please update us on how your review is going (this is an automated reminder).
@robinroche, please update us on how your review is going (this is an automated reminder).
I have started the review, and will complete it in the coming weeks.
@noah80, can you please let us know how your review is going?
:wave: @noah80, please update us on how your review is going (this is an automated reminder).
:wave: @robinroche, please update us on how your review is going (this is an automated reminder).
@timtroendle @nick-gorman
Thanks for this interesting submission. I installed the package and ran the examples without trouble.
After reviewing the submission, here are some suggestions for improvement:
@noah80, can you please give us an indication about the progress of your review?
@timtroendle @nick-gorman
Sorry for the delay. After reviewing the software too, I reached similar conclusions to Robin.
Right now with the given software it is rather difficult to judge if the software delivers the promised functionality and who could use the software for what purpose.
@robinroche and @noah80 thank you both for your reviews, I really appreciate the feedback. I'll aim to address your comments and questions in the next couple of weeks.
Hi @robinroche and @noah, thanks again for reviewing and your help comments. I've drafted a new version of the paper, made some changes to the documentation, and released a new version of the software with a minor update to try and address them.
Bellow I've tried highlight specifically where and how I've address each of your comments, please ask me to clarify if needed.
I've added significantly to the statement of need section, lines 60 - 117. I've outlined 4 distinct use cases, and provided a simple example analysis that briefly demonstrates one of the use cases.
I've edited the summary section of the paper to address this, lines 10 - 13. I've introduced electricity markets, the way the NEM spot market works simply, and how it needs more complex features like FCAS in real life.
I have edited the examples in the documentation to address this. For examples 1 - 5 I have added additional inline comments to explain the results. For examples 6 and 7 I've added graphs to make interpreting the results easier.
https://nempy.readthedocs.io/en/latest/examples.html#examples
For the 2nd example in the read me and examples 6 - 8 in the documentation I've added in text warnings about the volume of data to be downloaded.
I've also published a new version of the software that allows the historical data xml cache to be built on a daily rather than monthly basis. This reduces the data downloaded for the examples from 65GB to 8.5GB.
https://github.com/UNSW-CEEM/nempy/releases/tag/v1.1.0
I've added a section to the documentation to outline the steps of the dispatch procedure.
https://nempy.readthedocs.io/en/latest/intro.html#dispatch-procedure-outline
I've added additional detail:
"As part of the ongoing reform process significant changes to the dispatch process have also been proposed, for example, the introduction of an operating reserve market is being considered, as is the adoption of a dynamic intra-regional loss model."
https://nempy.readthedocs.io/en/latest/intro.html#introduction
This is standard semantic versioning, it's still the major version 1, the second and third numbers just get updated to indicate minor updates (but backwards compatible) and patches respectively.
I've also tried to update the text to make this clearer without being verbose in the intro.
"Version 1 will be a stable release and ongoing minor updates or patches will remain backwards compatible. Currently the latest minor release is v1.1.0."
I've added links to a description of the NEM in both the readme and the readthedocs page:
https://github.com/UNSW-CEEM/nempy#introduction https://nempy.readthedocs.io/en/latest/intro.html#introduction
Also, the fourth use case in the paper now explains that there is no complete description of the dispatch process provided by AEMO, and references several documents addressing aspects of the dispatch process.
The Run-time section (https://nempy.readthedocs.io/en/latest/intro.html#run-time) now links to the reference documentation on solvers.
The Ongoing Work section (updated) provides this information
https://nempy.readthedocs.io/en/latest/intro.html#ongoing-work
Not sure exactly what you mean by moving the examples, there was an error where they were showing up in the top level menu, is this what you meant?
Some more detail has been added to the descriptions.
A brief explanation has been added.
"The total runtime was calculated using the python time module and measuring the time taken from the loading of inputs to the extraction of results from the model. The runtime of different sub-process, i.e. loading of the XML file, was measured by inserting timing code into the Nempy source code where required."
https://nempy.readthedocs.io/en/latest/intro.html#run-time
This is explained in the readme on GitHub, I've added links to the the issues register and discussion board to make this clearer.
https://github.com/UNSW-CEEM/nempy#community
Added this section to the documentation
https://nempy.readthedocs.io/en/latest/intro.html#dependencies
Lines 10 - 17 of the paper should now provide a simpler intro for the wider audience. Lines 17 - 26 aim to softly introduce some of the additional complexity. Lines 60 - 117 deal specifically with use cases and example question. Lines 61 and 62 say who the target audience is.
I've also added a section to the documentation that talks about target users and uses cases: https://nempy.readthedocs.io/en/latest/intro.html#example-use-cases
The code is well-documented and readable.
Even after reading the documentation and the paper I still don't know what the software could be used for. Why would anyone want to model the dispatch?
I've added significantly to the statement of need section, lines 60 - 117. I've outlined 4 distinct use cases, and provided a simple example analysis that briefly demonstrates one of the use cases.
I have edited the examples in the documentation to address this. For examples 1 - 5 I have added additional inline comments to explain the results. For examples 6 and 7 I've added graphs to make interpreting the results easier.
https://nempy.readthedocs.io/en/latest/examples.html#examples
As for comment two I think lines 60 - 177 of the paper should now address this. I've also added a section on use cases to the documentation
https://nempy.readthedocs.io/en/latest/intro.html#example-use-cases
@whedon generate pdf from branch joss-paper
Attempting PDF compilation from custom branch joss-paper. Reticulating splines etc...
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@nick-gorman My comments have been addressed. Thank you for your contribution!
@robinroche comments:
- While I understand the general purpose of the code, additional information on how it could be used could be useful. For example: provide use cases, questions it could help answer, etc.
I've added significantly to the statement of need section, lines 60 - 117. I've outlined 4 distinct use cases, and provided a simple example analysis that briefly demonstrates one of the use cases.
- Briefly introduce the basics of electricity markets, e.g., a very simplified version of a market clearing problem, its objectives and constraints. Similarly, introduce acronyms such as FCAS
I've edited the summary section of the paper to address this, lines 10 - 13. I've introduced electricity markets, the way the NEM spot market works simply, and how it needs more complex features like FCAS in real life.
- Explain how to interpret example results. What should be expected and how to know it worked? Could graphs be added to have a more visual result?
I have edited the examples in the documentation to address this. For examples 1 - 5 I have added additional inline comments to explain the results. For examples 6 and 7 I've added graphs to make interpreting the results easier.
https://nempy.readthedocs.io/en/latest/examples.html#examples
- Specify the size of the data to download in the description of example 2 (this is only given in the comments). As this is really a lot of data, could it be possible to design another example with less data requirements?
For the 2nd example in the read me and examples 6 - 8 in the documentation I've added in text warnings about the volume of data to be downloaded.
I've also published a new version of the software that allows the historical data xml cache to be built on a daily rather than monthly basis. This reduces the data downloaded for the examples from 65GB to 8.5GB.
https://github.com/UNSW-CEEM/nempy/releases/tag/v1.1.0
- Briefly explain the main steps of the dispatch procedure.
I've added a section to the documentation to outline the steps of the dispatch procedure.
https://nempy.readthedocs.io/en/latest/intro.html#dispatch-procedure-outline
- Be more specific when you mention: "Significant changes to the dispatch process are also likely to occur soon"
I've added additional detail:
"As part of the ongoing reform process significant changes to the dispatch process have also been proposed, for example, the introduction of an operating reserve market is being considered, as is the adoption of a dynamic intra-regional loss model."
https://nempy.readthedocs.io/en/latest/intro.html#introduction
- The text mentions "Version 1 will be a stable release and ongoing minor updates or patches will remain backwards compatible." The current version is 1.0.1
This is standard semantic versioning, it's still the major version 1, the second and third numbers just get updated to indicate minor updates (but backwards compatible) and patches respectively.
I've also tried to update the text to make this clearer without being verbose in the intro.
"Version 1 will be a stable release and ongoing minor updates or patches will remain backwards compatible. Currently the latest minor release is v1.1.0."
- Link to a document explaining how the actual NEM works.
I've added links to a description of the NEM in both the readme and the readthedocs page:
https://github.com/UNSW-CEEM/nempy#introduction https://nempy.readthedocs.io/en/latest/intro.html#introduction
Also, the fourth use case in the paper now explains that there is no complete description of the dispatch process provided by AEMO, and references several documents addressing aspects of the dispatch process.
- Explain how to use solvers and what criteria they should meet.
The Run-time section (https://nempy.readthedocs.io/en/latest/intro.html#run-time) now links to the reference documentation on solvers.
- List "todos" you intend to implement in the future, possibly with priorities.
The Ongoing Work section (updated) provides this information
https://nempy.readthedocs.io/en/latest/intro.html#ongoing-work
- Move the long list of examples closer to where they would make sense, add more details in the description
Not sure exactly what you mean by moving the examples, there was an error where they were showing up in the top level menu, is this what you meant?
Some more detail has been added to the descriptions.
- Explain how to check the computation time.
A brief explanation has been added.
"The total runtime was calculated using the python time module and measuring the time taken from the loading of inputs to the extraction of results from the model. The runtime of different sub-process, i.e. loading of the XML file, was measured by inserting timing code into the Nempy source code where required."
https://nempy.readthedocs.io/en/latest/intro.html#run-time
- Explain how to report an issue or get support.
This is explained in the readme on GitHub, I've added links to the the issues register and discussion board to make this clearer.
https://github.com/UNSW-CEEM/nempy#community
- List the dependencies.
Added this section to the documentation
https://nempy.readthedocs.io/en/latest/intro.html#dependencies
- Regarding the paper abstract, make it simpler for a wider audience and clarify the problems it can help solve and/or the questions it can help answer. Who is the target audience?
Lines 10 - 17 of the paper should now provide a simpler intro for the wider audience. Lines 17 - 26 aim to softly introduce some of the additional complexity. Lines 60 - 117 deal specifically with use cases and example question. Lines 61 and 62 say who the target audience is.
I've also added a section to the documentation that talks about target users and uses cases: https://nempy.readthedocs.io/en/latest/intro.html#example-use-cases
Thanks @robinroche!
@robinroche @noah80 I've just had some grammar issues pointed out to me. I've edited the paper, you can see the very minor changes by viewing the commit below:
https://github.com/UNSW-CEEM/nempy/commit/192dce45d43fe30541ffd090c1bfabf105692d56
@robinroche there is one remaining open point on your checklist. Did you leave that open intentionally?
@noah80, could you have a look at the changes and tell us which of your points have been addressed?
@timtroendle No, I missed it. Everything has been addressed.
👋 @noah80 we are waiting for your update on this review. Can you let us know by when you will be able to update this?
@noah80, this review is currently blocked by the unfinished second review. Can you please give us an update?
The PDF update is excellent and fulfills now all criteria. I couldn't get the tests running so far though.
@nick-gorman : Could you contact me? Maybe I'm doing something wrong.
The PDF update is excellent and fulfills now all criteria. I couldn't get the tests running so far though.
@nick-gorman : Could you contact me? Maybe I'm doing something wrong.
@noah80 Good to hear I've been able to address your comments.
I've opened an issue on the nempy repo regarding the unit tests, see https://github.com/UNSW-CEEM/nempy/issues/4 . Can you reply on this issue and provide details of your problem running the unit tests?
I don't have your email, but you can contact me at n.gorman - at - unsw.edu.au
@noah80 please let me know if UNSW-CEEM/nempy#4 and UNSW-CEEM/nempy#5 have resolved your problems.
Still having some issues with running the examples in a clean enviroment due to missing dependencies in the package. This is not critical for the functionality but bad for the usability of your software in my opinion and I would suggest fixing this.
Still having some issues with running the examples in a clean enviroment due to missing dependencies in the package. This is not critical for the functionality but bad for the usability of your software in my opinion and I would suggest fixing this.
@noah80, good point, I've tried to address as per UNSW-CEEM/nempy#5
I was able to run all examples now after manually installing the missing dependencies.
While it would be nice to automate that part or surround make the code sections in the examples with try/except, it all works and looks usable.
@timtroendle All my issues have been adressed.
@nick-gorman Thanks for the cooperation!
@whedon check references
@whedon generate pdf from branch joss-paper
Attempting PDF compilation from custom branch joss-paper. Reticulating splines etc...
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@whedon check references from branch joss-paper
I'm sorry human, I don't understand that. You can see what commands I support by typing:
@whedon commands
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.5281/zenodo.3509134 is OK
- 10.25080/Majora-92bf1922-00a is OK
- 10.1038/s41586-020-2649-2 is OK
MISSING DOIs
- 10.1016/j.rser.2018.08.002 may be a valid DOI for title: A review of modelling tools for energy and electricity systems with large shares of variable renewables
- 10.1016/j.apenergy.2021.116731 may be a valid DOI for title: Trends in tools and approaches for modelling the energy transition
- 10.1016/j.rser.2020.110195 may be a valid DOI for title: A systemic approach to analyze integrated energy system modeling tools: A review of national models
- 10.1038/sdata.2018.203 may be a valid DOI for title: Open grid model of Australia’s National Electricity Market allowing backtesting against historic data
- 10.1016/j.enpol.2013.01.052 may be a valid DOI for title: Retrospective modeling of the merit-order effect on wholesale electricity prices from distributed photovoltaic generation in the Australian National Electricity Market
INVALID DOIs
- None
Thanks a lot @noah80 and @robinroche for your reviews!
We can now move on with the acceptance process. I had a quick read of the paper and found three small issues:
@nick-gorman , could you at this point please:
I can then move forward with accepting the submission.
/ooo December 24 until January 2
Thanks @timtroendle, @noah80 and @robinroche
I'm away from my work computer, which has the chart files that need correcting. @timtroendle I'll do the final tasks in a couple of weeks when I'm back at work.
@timtroendle
I've been able to fix the issues you found.
The release tag of the software is v1.1.2
And the zendo doi is: 10.5281/zenodo.5989170
Submitting author: @nick-gorman (Nicholas Gorman) Repository: https://github.com/UNSW-CEEM/nempy Version: v1.1.2 Editor: @timtroendle Reviewer: @noah80, @robinroche Archive: 10.5281/zenodo.5989170
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@noah80 & @robinroche, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @timtroendle know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Review checklist for @noah80
✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @robinroche
✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper