search

openjournals / jose-reviews

Reviews for the Journal of Open Source Education (JOSE)
http://jose.theoj.org
Creative Commons Zero v1.0 Universal
33 stars 4 forks source link

[REVIEW]: Hyper2D: a finite-volume solver for hyperbolic PDEs and hypersonic flows #163

Closed whedon closed 2 years ago

whedon commented 2 years ago

Submitting author: @sbocce (Stefano Boccelli) Repository: https://github.com/sbocce/hyper2D Version: v1.0.0 Editor: @labarba Reviewers: @mandli Archive: Pending

:warning: JOSE reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSE is currently operating in a "reduced service mode".

Status

status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/bfae4c46b559c6edea1e47b89f37e08d"><img src="https://jose.theoj.org/papers/bfae4c46b559c6edea1e47b89f37e08d/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/bfae4c46b559c6edea1e47b89f37e08d/status.svg)](https://jose.theoj.org/papers/bfae4c46b559c6edea1e47b89f37e08d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ketch & @mandli, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @labarba know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @ketch

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @mandli

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 2 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @ketch, @mandli it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 2 years ago

Wordcount for paper.md is 1257

whedon commented 2 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.05 s (252.6 files/s, 27802.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
MATLAB                           5            203            254            391
Markdown                         1             40              0            139
make                             5             76             43            112
TeX                              1              7              0             56
-------------------------------------------------------------------------------
SUM:                            12            326            297            698
-------------------------------------------------------------------------------

Statistical information for the repository '4ddeee7314d2208fdf358ed2' was
gathered on 2022/02/26.
No commited files with the specified extensions were found.
whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1063/5.0006258 is OK
- 10.1111/j.1365-2966.2005.09546.x is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

labarba commented 2 years ago

@ketch, @mandli — Thank you for agreeing to review for JOSE! This is where the action happens: work your way through the review checklist, feel free to ask questions or post comments here, and also open issues in the submission repository as needed. Godspeed! 🚀

whedon commented 2 years ago

:wave: @ketch, please update us on how your review is going (this is an automated reminder).

whedon commented 2 years ago

:wave: @mandli, please update us on how your review is going (this is an automated reminder).

labarba commented 2 years ago

hiya, @ketch, @mandli – Would you pop in this review thread and see if you can push it forward a bit? We'd like to get this done within the next month, if you could. Thank you! 🙏

ketch commented 2 years ago

I have to say respectfully that I don't understand how this package is intended to be an educational resource. How does one use it to learn?

It consists of examples of implementations of various algorithms for various problems. But those problems and algorithms are not documented within the package. In fact, there aren't even citations to other resources where the user could find out what equations are being solved and what discretizations are being used. Perhaps I am missing something?

labarba commented 2 years ago

From the submission cover message, I understand that this package was originally submitted to JOSS. I suspect that the editors there assessed that its research application is limited and suggested that probably JOSE was a better venue. The author's cover message says about the package that

It targets (1) students learning numerical methods or needing a software for running computations for their MSc thesis, and (2) researchers in need of a simple code that is yet general enough for implementing complex PDEs.

If reuse of the package in teaching contexts needs additional documentation, that is something the authors should add in revision.

ketch commented 2 years ago

Well, after reading the JOSE guidelines and looking at the checklist here, I'm completely confused. As far as I can tell, there are no "lessons" or pedagogical explanations of any kind in this package, but the reviewer checklist also doesn't mention anything of that kind. What I can say is that I teach a course where this package would be relevant, but I don't see how it would be helpful to a student, given the lack of any pedagogical material.

The reviewer guidelines do ask me to look for tests, while the author claims that JOSS sent him here so he could get his paper accepted without needing to write tests. I think there are some inconsistencies and ambiguities regarding the journal, and I don't feel capable of refereeing papers for it in light of that. I have provided as much feedback as I could (here and in issues in the repository). I hope that is helpful, and that you'll understand that I respectfully decline to play a further role in the evaluation of this submission.

labarba commented 2 years ago

JOSE admits two types of submissions: learning modules, and software submissions. Depending on the article type chosen by the author on submission, the Review issue will display one of two checklists. In this case, the checklist is for a software submission.

Our documentation does mention the Specific requirements for software submissions, including testing.

The paper should "describe the functionality of the software, usage and recent experience of use in teaching and learning situations." The journal does seek accepted submissions to be geared for reuse of the software (or lessons, in the case of learning modules), so the author should certainly address this.

labarba commented 2 years ago

the author claims that JOSS sent him here so he could get his paper accepted without needing to write tests

Where did you see that? Certainly that would be the wrong reason to transfer a submission from JOSS to JOSE. The right reason is that the software is not for research but for pedagogical uses.

ketch commented 2 years ago

https://github.com/sbocce/hyper2D/issues/3#issuecomment-1055067770

Tests -> I have prepared no automated tests. Actually, I had previously submitted Hyper2D to JOSS and the reviewers kindly suggested me to submit to JOSE instead for this very reason, that there are no tests and the software is mostly educational.

mandli commented 2 years ago

I am in general agreement with @ketch. At the very least the educational aspect/use of the software is lacking.

labarba commented 2 years ago

@ketch I see. I will add a link here to the issues you opened in the target repo, to get the breadcrumbs back.

https://github.com/sbocce/hyper2D/issues/1

https://github.com/sbocce/hyper2D/issues/2

https://github.com/sbocce/hyper2D/issues/3

labarba commented 2 years ago

@sbocce — I'm seeing from the reviewers' comments (and exploring your repo) that this submission needs quite some work before it can be considered for publication. For the software to be reusable in teaching and learning contexts, it is essential that you provide documentation beyond raw-text README files in each folder. As it is, you seem to be expecting of users to "dig through" the repository to find out what they could do with these codes.

Most JOSE submissions are of the learning module category, but here is one in the software category:

Ogbureke, K. U., (2021). mLEARn: An Implementation of Multi-layer Perceptron in C++. Journal of Open Source Education, 4(41), 59, https://doi.org/10.21105/jose.00059

The paper describes some use cases in the context of teaching and learning, and the software repository offers a well-formatted README with user instructions, examples, and an API doc in PDF. Users need these entry points to adopt a new open-source software package.

At this juncture, we have two options: we can add a paused label and you can come back and ping us when you have done more work in response to our input here, or we can withdraw the submission and you can resubmit when the software and documentation is in better shape to target reuse.

sbocce commented 2 years ago

Hello @labarba, hello @ketch & @mandli, thank you for the review & observations.

Where did you see that? Certainly that would be the wrong reason to transfer a submission from JOSS to JOSE. The right reason is that the software is not for research but for pedagogical uses.

Just to clarify: once I submitted to JOSS some months ago, the reviewers pointed to me that rather than being a "full software package aimed at production use" aka JOSS target (and thus being shipped with automated google tests and all fancy things), it is instead a software more aimed towards educational use. So, they pointed me to JOSE, that I didn't know back then, and we thought that your venue is closer to our aims. So we came here.

At this juncture, we have two options: we can add a paused label and you can come back and ping us when you have done more work in response to our input here, or we can withdraw the submission and you can resubmit when the software and documentation is in better shape to target reuse.

I would be happy to implement the said changes to the repository. I see that @ketch requested to get out of the review loop though. Anyway, if you want to add a paused label, I'll ping you once I have the submission ready for further review.

labarba commented 2 years ago

I'm adding the link here to the conversation over at JOSS, for future reference:

https://github.com/openjournals/joss-reviews/issues/4049#issuecomment-1013249736

labarba commented 2 years ago

☝️ note that nowhere did the JOSS editors imply that lack of testing was OK with us

sbocce commented 2 years ago

Hi @labarba thanks for your help.

Quick question about the type of tests admitted by JOSE:

Tests: Are there automated tests or manual steps described so that the function of the software can be verified?

I do have the "manual steps described" for running the code. Do you think that adding a folder with the expected results in form of dat files and/or VTK and/or images will do?

labarba commented 2 years ago

@whedon remove @ketch as reviewer

whedon commented 2 years ago

OK, @ketch is no longer a reviewer

labarba commented 2 years ago

@sbocce — My assessment is that your software submission is not ready for publication, for a number of reasons. If the reviewer(s) when looking through the submission realize that it would take an inordinate amount of work on their part to shepherd the necessary revisions to bring it up to publishable standards, it is not surprising that they would renege on their commitment to review. I believe this may be the case here.

It looks to me like you wrote this code as part of your PhD work and you are now looking to get a publication from it. This is what JOSS and JOSE were founded to provide: a publication venue for software written as part of a scholarly endeavor. But to be eligible for publication, the software needs to adhere to minimum standards that enable reusability: the code should be immediately useful to others.

For JOSE, the software should be useful in teaching and learning settings. If you did not write the software with these uses in mind (you wrote it for your own purposes during your PhD), then you would have to invest the work to make it usable for education.

In open-source software, users expect documentation, ways to check for correctness (tests), and good design (modularity, separation of concerns, low code redundancy, etc.). Our submission guidelines are not exhaustive, because they cannot be, and we do rely on community standards. The JOSS editors and the reviewers here saw that your software is not up to standards, and that the effort required on their part to help you get there are substantial.

Your question above also leads me to the assessment that it could take a lot of effort on our part if you need us to tell you how to provide checks for code correctness. (Certainly, images of expected results are not enough. Data files are not enough: there should be at least scripts that compare the golden files to the user's output.)

If you are still interested in pursuing a publication, I suggest you first work on the documentation (a wiki is not what users expect), then look at your code design for reusability (modify and edit docs as needed), and develop a couple of examples with testable expected output (perhaps read about code testing: you may not want to include unit tests, but you at least want regression tests).

sbocce commented 2 years ago

@labarba hi, thanks for your detailed reply.

Right, I see your point. Indeed, it seems pointless to keep working on the present submission, given the situation. If you could please just go ahead and withdraw my application, it would be amazing.

Once again thank you to both yourself and @ketch and @mandli for taking the time to check out my submission.

labarba commented 2 years ago

@whedon withdraw

whedon commented 2 years ago

Paper withdrawn.