search

IPCC-WG1 / Atlas

Repository supporting the implementation of FAIR principles in the IPCC-WGI Atlas
Other
228 stars 51 forks source link

ATLAS review #29

Closed kthyng closed 3 years ago

kthyng commented 3 years ago

Hi, I've gone through all the items in this repo and I will give a high level review because I am neither an R user nor familiar with the pieces that go into global scale climate work (I am a coastal oceanographer). However, I am an Associate Editor in Chief with the Journal for Open Source Software (JOSS) and I have related experience through that (as well as my own research).

Generally I really like the approach you are taking. It is clear you are working very hard to be clear, open, and reproducible. This is excellent! Using GitHub as a tool for this is great since it is all tracked and open, and additionally there are a number of tools built in to help, for example displaying notebooks is really helpful. I think that Python has a larger user base for this sort of work, but mine is an outside perspective and maybe isn't true for climate modeling. The readme's are nice for guidance through the directories, and the notebooks are helpful for more controlled explanation. A full readthedocs sort of comprehensive documentation would be better.

All this said, I would like to suggest that for future, more formal reviews, you could consider the JOSS review approach. Or, perhaps it would be appropriate to initiate a formal JOSS-IPCC partnership for this (I am happy to initiate a conversation about this between groups if there is interest on your part). For a review, an editor recruits at least 2 reviewers to actually install, test, and use the code, using the available software installation instructions and other documentation. The review process is done in github, but in a specific issue (in the JOSS repo) in which the reviewers summarize their reviews and then link to other issues in the software repo for more details. This helps to keep the reviews organized and have a basic train of thought and conversation from the reviewers while still allowing for lots of details in the separate issues. I didn't try out your JupyterHub, but in your case it might make more sense to use that environment since probably all the model output is available there which is important, though if by the time of the next IPCC the model output and data is easily available in the cloud, it would be best for reviewers to test everything totally independently. The goal in your case might be that the reviewers can reasonably reproduce the final plots, products, or analysis, but that would need to be specified.
You can get a good feel for the review approach from the review checklist.

jesusff commented 3 years ago

Thank you for your review. I agree this would be a good path to follow in future IPCC cycles. We relied on Github as a mean for open review but, definitely, the JOSS review approach is much more systematic, providing clearer guidance for the reviewers. The checklist could be adapted for this purpose. We will prepare a paper summarizing this experience and the challenges found, counting on the reviewers who participated in this FAIR review. It would be great to include your proposal for the review and documentation as a future perspective.

Regarding the JupyterHub, we finally opted for a binder environment, which is a standard and open tool to provide a ready-to-go environment to run the scripts and notebooks. The Hub needs further development which will be tackled during next year.