[REVIEW]: ETHOS.FINE: A Framework for Integrated Energy System Assessment

[editorialbot]

Submitting author: !--author-handle-->@t-gross<!--end-author-handle-- (Theresa Gross) Repository: https://github.com/FZJ-IEK3-VSA/FINE Branch with paper.md (empty if default branch): paper Version: v2.3.0 Editor: !--editor-->@arfon<!--end-editor-- Reviewers: @GregorBecker, @olejandro, @leonardgoeke Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c4c62e14f2273e97606b1eb96d58bec6"><img src="https://joss.theoj.org/papers/c4c62e14f2273e97606b1eb96d58bec6/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c4c62e14f2273e97606b1eb96d58bec6/status.svg)](https://joss.theoj.org/papers/c4c62e14f2273e97606b1eb96d58bec6)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@GregorBecker & @olejandro & @leonardgoeke, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @arfon know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @GregorBecker

📝 Checklist for @leonardgoeke

📝 Checklist for @olejandro

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.39 s (406.1 files/s, 143796.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          92           4996           8228          24874
Jupyter Notebook                12              0          12442           2506
HTML                             2            165              4            605
TeX                              1             31              0            482
reStructuredText                34            317            265            378
YAML                             6             21             37            205
Markdown                         5             70              0            167
DOS Batch                        3             21              1             46
TOML                             1              5              8             38
make                             1              4              7             10
Dockerfile                       1              5              1              8
-------------------------------------------------------------------------------
SUM:                           158           5635          20993          29319
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 1532

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.ijhydene.2019.08.032 is OK
- 10.1016/j.ijhydene.2020.12.197 is OK
- 10.3390/en12142825 is OK
- 10.1016/j.energy.2018.05.059 is OK
- 10.1016/j.ijhydene.2018.11.194 is OK
- 10.1007/s12532-011-0026-8 is OK
- 10.3390/en13030641 is OK
- 10.1016/j.apenergy.2021.117825 is OK
- 10.1016/j.apenergy.2022.119029 is OK
- 10.1016/j.ijhydene.2021.04.036 is OK
- 10.1016/j.apenergy.2018.01.023 is OK
- 10.1016/j.rser.2018.11.020 is OK
- 10.1016/j.enbuild.2022.112480 is OK
- 10.1016/j.ijhydene.2021.04.036 is OK
- 10.3390/en15103801 is OK
- 10.3390/en15249517 is OK
- 10.1007/s11590-021-01826-w is OK
- 10.1016/j.adapen.2021.100063 is OK
- 10.1016/j.ijhydene.2023.05.308 is OK
- 10.1016/j.adapen.2023.100148 is OK
- 10.1016/j.ijhydene.2023.05.012 is OK
- 10.1016/j.rset.2023.100059 is OK
- 10.18154/RWTH-2023-01485 is OK
- 10.18154/RWTH-2023-01465 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[arfon]

@GregorBecker, @olejandro, @leonardgoeke – This is the review thread for the paper. All of our communications will happen here from now on.

Please read the "Reviewer instructions & questions" in the first comment above. Please create your checklist typing:

@editorialbot generate my checklist

As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention https://github.com/openjournals/joss-reviews/issues/6274 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.

Also, noting that multiple reviewers here said they would not be able to complete their reviews until towards the middle-end of February.

[arfon]

:wave: @GregorBecker, @olejandro, @leonardgoeke – just checking in here to see if there's anything you need from me at this stage?

[GregorBecker]

Good morning @arfon, at the moment I don't need any further help from you, as soon as I find the time to review I will open my checklist. Kind Regards Gregor

[GregorBecker]

Review checklist for @GregorBecker

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/FZJ-IEK3-VSA/FINE?
• [x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@t-gross) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [ ] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[leonardgoeke]

Review checklist for @leonardgoeke

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/FZJ-IEK3-VSA/FINE?
• [x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@t-gross) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [ ] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[leonardgoeke]

Overall, the introduction of ETHOS.FINE is clear, comprehensible, and relevant. I have a few points listed below, following the guidelines above.

Functionality

The installation guide was clear, but mamba as a package installer was new to me and something I had to install separately. Therefore, it is worth briefly mentioning and explaining this choice in the paper. The long list of peer-reviewed publications applying the tool proves its functionality. Although this could be interesting in this context, there are no claims regarding performance.

General checks

All good here.

Documentation

The repository names the package's dependencies but only specifies the version in one case. In my experience with Python, packages are often not compatible with the newest version of a dependency. As a result, the user must invest a lot of time reconstructing a functioning configuration if the dependencies do not specify a version.

The documentation shows how to cite the package but gives no details on contributing or reporting problems.

Software Paper

The current state-of-the-art and need for the software is not discussed in the paper. The one paper that is supposed to prove the need is outdated and the tool is not differentiated from others in the field (Calliope, GenX, PyPSA, AnyMOD, etc.).

Earlier, the paper mentioned complexity reduction as a key capability of FINE. Therefore, some indicative information on performance is sensible, for instance, how long it takes to generate and solve a model with a certain number of time-steps, regions, and technologies. In this context, I would also like to know if the tools use pyomo as a backend for historical reasons or if it has an advantage over Linopy in the specific case.

[olejandro]

Review checklist for @olejandro

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [ ] Repository: Is the source code for this software available at the https://github.com/FZJ-IEK3-VSA/FINE?
• [ ] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• [ ] Contribution and authorship: Has the submitting author (@t-gross) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [ ] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [ ] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [ ] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [ ] Installation: Does installation proceed as outlined in the documentation?
• [ ] Functionality: Have the functional claims of the software been confirmed?
• [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [ ] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[arfon]

@t-gross – looks like there's a bunch of feedback from the reviewers here at this point. Have you started working on that already? If not, perhaps you could and respond here?

[t-gross]

I apologize for my late response. First, thank you very much for all the feedback! We have started to implement it but it will still take some time.

[arfon]

@t-gross – just checking in here on your updates?

[t-gross]

@t-gross – just checking in here on your updates?

Unfortunately, it will still take some time to get everything updated. We implemented some improvements to the code and readme. I hope to finish everything by end of June.

[arfon]

:wave: @t-gross – it's now the end of August. How are you getting on?

[t-gross]

👋 @t-gross – it's now the end of August. How are you getting on?

Thanks for the question, and again my apologies that it took so long. I just uploaded the latest version of the paper (https://github.com/FZJ-IEK3-VSA/FINE/tree/paper). We tried to address most of the comments of our reviewers. I also wanted to thank them to read and comment on our paper draft and our software package.

As an update on this:

Repository:

• We published a new version of our package with some new major features (adding investment periods as additional time dimension). This enables the modeler to optimize the whole transition pathway of an energy system.
• We reorganized the examples and added the description to the readme.
• We added a new section in the readme about how to contribute and get support. Additionally, we added a discussion area and the code of conduct.
• Installation description was updated; mamba is still the recommended way to install the package.
• Added specifiation of versions for package's dependencies

Paper:

• Update of the paper regarding the new features.
• Due to this new features, two additional authors have been added who are primarily responsible for the development of these features.
• Paragraph about ETHOS model suite has been moved to another section (summary -> examples of previous usage).
• Paragraph about built-in aggregation methods has been revised.
• Examples of other framework tools have been added.
• Added sentence about the usage of pyomo (historical reason; might be changed in future to improve performance).
• Update references (correct links)

Not added:

• As the performance of the model setup and optimization process is highly dependent on the computational resources, we have not added performance indicators to the publication.

Ongoing:

• Describing the mathematical background of the model is currently in progress and will be published on ReadTheDocs (https://vsa-fine.readthedocs.io/en/latest/index.html) (connected to https://github.com/FZJ-IEK3-VSA/FINE/issues/51). As the software package is further developed, new boundary conditions, parameters and variables must be added. We are working on this issue and hope to be able to put a first version online soon.

[arfon]

@GregorBecker, @olejandro, @leonardgoeke – apologies for the incredibly slow bump here but it looks like this submission is ready for your attention again.