search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
702 stars 36 forks source link

[REVIEW]: FHI-vibes: Ab initio Vibrational Simulations #2671

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @flokno (Florian Knoop) Repository: https://gitlab.com/vibes-developers/vibes Version: v1.0.2 Editor: @jgostick Reviewers: @keipertk, @ajjackson Archive: 10.5281/zenodo.4300415

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/54c29a971700c09ff80b82f47429a36d"><img src="https://joss.theoj.org/papers/54c29a971700c09ff80b82f47429a36d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/54c29a971700c09ff80b82f47429a36d/status.svg)](https://joss.theoj.org/papers/54c29a971700c09ff80b82f47429a36d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@keipertk, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @jgostick know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @keipertk

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @ajjackson

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @keipertk it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1088/1361-648X/aa680e is OK
- 10.1016/j.scriptamat.2015.07.021 is OK
- 10.1103/PhysRevB.91.094306 is OK
- 10.1016/j.cpc.2009.06.022 is OK
- 10.1016/j.cpc.2018.09.020 is OK
- 10.5334/jors.148 is OK
- 10.1109/MCSE.2011.37 is OK
- 10.1002/cpe.3505 is OK
- 10.1038/nmat3568 is OK
- 10.1103/PhysRevLett.78.4063 is OK
- 10.1103/PhysRevLett.118.175901 is OK
- 10.1002/adts.201800184 is OK
- 10.1002/anie.201812112 is OK
- 10.1021/acs.jpcc.5b11115 is OK
- 10.1038/nmat2090 is OK
- 10.1016/j.jeurceramsoc.2007.12.023 is OK
- 10.1103/PhysRevLett.96.115504 is OK
- 10.1103/physrevb.79.064301 is OK
- 10.1557/mrs.2018.208 is OK

MISSING DOIs

- 10.1103/physrevmaterials.4.083809 may be a valid DOI for title: Anharmonicity Measure for Materials
- 10.1038/s41597-020-00638-4 may be a valid DOI for title: AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance

INVALID DOIs

- None
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

keipertk commented 3 years ago

I was unable to find community guidleines for third parties wishing to contribute to the software, outside of the standard MIT license agreement. Everything else gets a check.

jgostick commented 3 years ago

@keipertk Thanks for this! So fast too! I still haven't even found a second reviewer yet!

flokno commented 3 years ago

@jgostick in case you don't find a second reviewer: I talked to [at]ajjackson at some earlier point and he was interested to review this.

@keipertk thanks for the feedback. Indeed we didn't formally include contribution guidelines, we'll add them.

jgostick commented 3 years ago

Thanks @flokno I will ask them.

Hi @ajjackson, your name has been suggested as a reviewer for this submission by @flokno. I am in need of a second reviewer, so your time would be much appreciated.

ajjackson commented 3 years ago

Hi @jgostick I will be happy to review this.

I have no COI as defined by JOSS policy. In interest of transparency I would state that I met Florian at a workshop last year for ASE developers/users and recommended submitting the code to JOSS based on presentation/discussion at that event. I have not used the code, examined it closely or otherwise been involved in its development.

jgostick commented 3 years ago

That sounds like a fine relationship. The review process is transparent, so if you didn't know each other before, you would afterwards!

jgostick commented 3 years ago

@whedon add @ajjackson as reviewer

whedon commented 3 years ago

OK, @ajjackson is now a reviewer

flokno commented 3 years ago

Hi @keipertk , would you be happy with such a contribution guideline: https://gitlab.com/vibes-developers/vibes/-/merge_requests/23 ?

keipertk commented 3 years ago

Yep that looks great, nice work!

flokno commented 3 years ago

Hi, I was waiting to merge https://gitlab.com/vibes-developers/vibes/-/merge_requests/23 until @ajjackson has made additional comments, any news there?

ajjackson commented 3 years ago

This is a sufficient set of contribution guidelines in my opinion, feel free to merge

ajjackson commented 3 years ago

@jgostick I am not able to click the check boxes in the review - I think I missed the notification to get the required permissions. Can you activate this?

ajjackson commented 3 years ago

Some initial comments on the manuscript:

I think this can be an exciting and useful contribution to the atomistic vibrations ecosystem. I especially welcome that the code interfaces with other active projects rather than reinventing the wheel.

The paper makes a reasonable survey of related tools, putting FHI-vibes in context. It does seem unfair to omit any mention of the MIT-licensed TDEP code which does firmly sit at the interface between lattice dynamics and MD. I don't think TDEP undermines the scholarly effort or community need for this work; the statement of need already clearly argues that the wide scope and integrated features set FHI-vibes apart from existing codes.

I am confused about the relationship between the statement of need and the actual existing feature set of FHI-vibes. The SoN gives several examples of methods that might be accessed using an integrated LD/MD toolkit: efficient MD initialisation; harmonic analysis of MD trajectories; anharmonicity analysis (Knoop et al); ab initio GK (Carbogno et al). Of these, it is not clear from the manuscript which of these is implemented as a workflow in FHI-vibes, and which are aspirational examples of things that a user might build on top of FHI-vibes. From the documentation, I can find information about MD initialisation and anharmonicity analysis but not the other two techniques. I see green_kubo and harmonic_analysis modules in the source code so presumably these are WIP?

I will have a crack at the tutorials next.

flokno commented 3 years ago

Hi Adam, thanks for the initial comments!

The paper makes a reasonable survey of related tools, putting FHI-vibes in context. It does seem unfair to omit any mention of the MIT-licensed TDEP code which does firmly sit at the interface between lattice dynamics and MD. I don't think TDEP undermines the scholarly effort or community need for this work; the statement of need already clearly argues that the wide scope and integrated features set FHI-vibes apart from existing codes.

Of course we are aware of TDEP (and actually big fans of it), and we actually support dumping MD trajectories to TDEP input files, although this is kind of a hidden feature. However, since TDEP is essentially unrelated to our officially supported features (run calculations and in particular MD, compute finite-differences harmonic forceconstants, postprocessing like anharmonicity quantification) and it is outside of the python environment, I omitted it in the survey.

I am confused about the relationship between the statement of need and the actual existing feature set of FHI-vibes. The SoN gives several examples of methods that might be accessed using an integrated LD/MD toolkit: efficient MD initialisation; harmonic analysis of MD trajectories; anharmonicity analysis (Knoop et al); ab initio GK (Carbogno et al). Of these, it is not clear from the manuscript which of these is implemented as a workflow in FHI-vibes, and which are aspirational examples of things that a user might build on top of FHI-vibes. From the documentation, I can find information about MD initialisation and anharmonicity analysis but not the other two techniques. I see green_kubo and harmonic_analysis modules in the source code so presumably these are WIP?

Yes, Green Kubo and more sophisticated harmonic analysis are WIP. These things should be finalized sometime next year. We plan a separate release (vibes 1.1.?) when GK is ready. Our plan with the paper was to write it such that it covers the envisaged scope of FHI-vibes and not just the features available in 1.0. It is stated in the paper that the officially supported features are listed in the docs/the webpage. If you wish to disentangle this more clearly in the paper, we can do so. I could for example write sth. like "In the initial release, FHI-vibes supports XYZ. We plan to add features A and B in upcoming releases. The currently supported list of features and a use guide etc. are available at vibes....de". Would you prefer that?

I will have a crack at the tutorials next.

Looking forward to that, thanks again for the good comments.

arfon commented 3 years ago

@whedon re-invite @ajjackson as reviewer

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@ajjackson please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

jgostick commented 3 years ago

Thanks @arfon ... I should have been able to figure that out, sorry

ajjackson commented 3 years ago

If you wish to disentangle this more clearly in the paper, we can do so. I could for example write sth. like "In the initial release, FHI-vibes supports XYZ. We plan to add features A and B in upcoming releases. The currently supported list of features and a use guide etc. are available at vibes....de". Would you prefer that?

I think that would be ideal. @jgostick is that reasonably in line with JOSS conventions?

jgostick commented 3 years ago

I would suggest to leave out any reference to future functionality. It makes the paper feel like a temporary sign post rather than a destination when you talk about all the things you want to add, more like wishful thinking. If these future functionalities are important enough, then the software is incomplete without it. That's not official JOSS policy, just my opinion.

flokno commented 3 years ago

Thanks for these comments. I will discuss this with my coauthors as soon as the rest of the review is in.

flokno commented 3 years ago

Hi guys, did you already have a look at the tutorials? Looking forward to your comments.

jgostick commented 3 years ago

HI @ajjackson How is the review coming? Don't forget to check off the tick-boxes at the top of this thread for each aspect of the code as you look through it.

ajjackson commented 3 years ago

I have been working through the tutorial today. So far so good, will pick it up again once the molecular dynamics part has run. This is with a view to checking the functionality/doc quality.

There are probably some more checkboxes that I can already tick based on the manuscript, will get on those soon.

I have posted one documentation-related issue on the project board here
https://gitlab.com/vibes-developers/vibes/-/issues/21

jgostick commented 3 years ago

Thanks for the update.

ajjackson commented 3 years ago

Re:

Documentation

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

I find that the introductory paragraph in the README and landing page of the documentation does a good job of outlining the scope and problems. However, it may not be so obvious who the target audience is. The tutorials do repeatedly reiterate that users are expected to have some experience of lattice dynamics and/or molecular dynamics simulations. I worry that a naive user (or, say, a supervisor directing a student) could see this introduction and think that such matters will be taken care of for them. I think this will benefit from a line clarifying that it is intended for computational materials researchers who already use atomistic simulations.

ajjackson commented 3 years ago

I have completed tutorials up to the Fireworks part. This will require a bit more configuration to check, I'm afraid.

Re:

Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

There are quite a few aspects to the documentation here.

As such I can say that the core (command-line) functionality is documented, but there is room for improvement on the Python API. As this is only promised "for advanced analysis", I don't find the paper to be misleading on this point. And the tutorials provide a decent entrypoint.

I should be able to tick off

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

and

State of the field: Do the authors describe how this software compares to other commonly-used packages?

once the manuscript has been revised to make a clearer separation between things that are currently possible and things that we would like to do in future.

flokno commented 3 years ago

Hi Adam, thanks for the further comments, we're awaiting your issues on gitlab and comments on fireworks.

flokno commented 3 years ago

@ajjackson I understand your comments on the manuscript are complete and mainly focus on the "state of the field" section with a clearer separation between examples/potential future directions and implemented featuers? If yes I'll go back to my coauthors and revise the script to progess on that end.

ajjackson commented 3 years ago

That's correct. While there is truth in @jgostick 's comment about "wishful thinking" future features, the situation is a little more muddled with packages like this that present a toolkit. One can make accurate statements about how the current state provides a capability for users and developers to build workflows, even if those workflows are not yet provided with the package. So I can see why unimplemented workflows belong in the "statement of need" - they are intended to illustrate the need for a good framework.

So I would like to see a clearer separation between:

and to avoid a long "wishlist" the bar for significance should be set fairly high for that last category. I think the examples given (harmonic analysis of MD and Green-Kubo calculations) are good, but maybe a few words are needed to explain how they benefit from the tools provided by vibes.

I still need to do the Fireworks tutorials, which hopefully will be aided by the installation instructions provided in the docs. I have played with Fireworks before, but it was some time ago on a machine I no longer have access to.

ajjackson commented 3 years ago

Have not had much luck today trying to follow the mongodb instructions with a Podman container. Will have to try a native installation.

tpurcell90 commented 3 years ago

@ajjackson I might be able to help out with the mongodb installation, I never used Podman before, but I have set up the mongodb stuff for FHI-vibes.

ajjackson commented 3 years ago

I got it working and have completed the Fireworks tutorials. I'm pleased to report that the Vibes docs were more helpful than the Fireworks ones when it came to the commands to set up the DB, as the Mongo syntax is more up-to-date. The main issues were around the instructions for shutdown/restart/reset of the DB; this didn't quite play nicely with the container setup. But in the end it was ok and I don't have any specific recommendations for the Vibes docs on that point. The tutorials themselves were enlightening, easy to follow and not too long. There is some exciting functionality in there which should assist some good science (especially the automated convergence testing.)

I reported some minor things here but they do not impact this review https://gitlab.com/vibes-developers/vibes/-/issues/32

I have checked off the functionality documentation as satisfactory; the tutorials give a good run-down of the options and I think at this stage they are fairly essential to understand the package. I would recommend that the authors do also consider the accessibility of their reference/API documentation, especially as the package grows. An auto-generated tree from the docstrings would make this information more navigable/searchable for little development effort.

I now await:

flokno commented 3 years ago

Thank you very much Adam, we'll discuss the changes and come back to you soon.

flokno commented 3 years ago

Hi @ajjackson , thank you very much again for going through the tutorials in this great detail, this was very helpful.

We're addressing your remaining comments in this merge request: https://gitlab.com/vibes-developers/vibes/-/merge_requests/25

In particular, we add the statement

In the documentation and tutorials, knowledge of first-principles electronic-structure theory as well as proficiency with ab initio codes such as FHI-aims and high-performance computing are assumed. Additional experience with Python, the Atomic Simulation Environment (ASE), or Phonopy is helpful, but not needed.

to the README to clarify the prerequisites for running the tutorials.

In the paper, we added a dedicated subsection Features to explicitly list the available features of the 1.0 release in order to avoid possible confusion with methods mentioned in Statement of need which summarizes the scope of FHI-vibes as a framework for vibrational simulations.

Is there anything left to do from your side?

flokno commented 3 years ago

Hi @keipertk , also many thanks to your for reviewing the code.

We added community guidelines as discussed in this merge request: https://gitlab.com/vibes-developers/vibes/-/merge_requests/23

Is there anything else left to do from your side?

flokno commented 3 years ago

@whedon generate pdf from branch joss_paper_review

whedon commented 3 years ago 
Attempting PDF compilation from custom branch joss_paper_review. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

ajjackson commented 3 years ago

Hi @ajjackson , thank you very much again for going through the tutorials in this great detail, this was very helpful.

We're addressing your remaining comments in this merge request: https://gitlab.com/vibes-developers/vibes/-/merge_requests/25

This is good, I will be able to complete the review checklist when it has been merged.

flokno commented 3 years ago

Hi @ajjackson , thank you a lot. I merged the MR and updated the PyPi release acocordingly.

@keipertk if you are on board as well I would ask you to tick the remaining checkbox.

Many thanks to all of you for this very detailed and productive review process!

keipertk commented 3 years ago

Checked it off. Thank you for contributing this excellent work!

flokno commented 3 years ago

Thank you!

@jgostick the boxes are ticked, what next?

jgostick commented 3 years ago

Nice work everyone! The next steps are up to me and the EIC. I will get started on it later today.

jgostick commented 3 years ago

@flokno

Here is a list of what to do next:

flokno commented 3 years ago

Hi @jgostick ,

flokno commented 3 years ago

@whedon generate pdf