search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
694 stars 36 forks source link

[REVIEW]: mathlib: A Scala package for readable, verifiable and sustainable simulations of formal theory #6049

Open editorialbot opened 8 months ago

editorialbot commented 8 months ago

Submitting author: !--author-handle-->@markblokpoel<!--end-author-handle-- (Mark Blokpoel) Repository: https://github.com/markblokpoel/mathlib Branch with paper.md (empty if default branch): joss Version: v0.9.0 Editor: !--editor-->@danielskatz<!--end-editor-- Reviewers: @larkz, @stephenfmann, @drussellmrichie Archive: Pending

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/713c6a7eef58b64966588b2b87d948c4"><img src="https://joss.theoj.org/papers/713c6a7eef58b64966588b2b87d948c4/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/713c6a7eef58b64966588b2b87d948c4/status.svg)](https://joss.theoj.org/papers/713c6a7eef58b64966588b2b87d948c4)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@bzz & @larkz, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danielskatz know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @larkz

📝 Checklist for @stephenfmann

📝 Checklist for @drussellmrichie

editorialbot commented 8 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 8 months ago 
Software report:

github.com/AlDanial/cloc v 1.88  T=0.01 s (516.2 files/s, 29422.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                         1             22              0             84
TeX                              1              4              0             38
YAML                             1              1              4             18
-------------------------------------------------------------------------------
SUM:                             3             27              4            140
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 8 months ago

Wordcount for paper.md is 711

editorialbot commented 8 months ago

Failed to discover a valid open source license

editorialbot commented 8 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1177/1745691620970585 is OK
- 10.1177/1745691620970604 is OK

MISSING DOIs

- None

INVALID DOIs

- None
editorialbot commented 8 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 8 months ago

@bzz and @larkz - Thanks for agreeing to review this submission. This is the review thread for the paper. All of our communications will happen here from now on.

As you can see above, you each should use the command @editorialbot generate my checklist to create your review checklist. @editorialbot commands need to be the first thing in a new comment.

As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#6049 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if either of you require some more time. We can also use editorialbot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@danielskatz) if you have any questions/concerns.

danielskatz commented 7 months ago

👋 @bzz and @larkz - Just a brief reminder that you will need to use the command @editorialbot generate my checklist to create your review checklist before you can do anything else. @editorialbot commands need to be the first thing in a new comment.

danielskatz commented 7 months ago

👋 @bzz and @larkz - I would appreciate it if you would use the command @editorialbot generate my checklist to create your review checklist, and then check off the first two items to make sure the process works correctly. @editorialbot commands need to be the first thing in a new comment.

bzz commented 7 months ago

Review checklist for @bzz

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

larkz commented 7 months ago

Review checklist for @larkz

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

larkz commented 6 months ago

@editorialbot generate pdf

editorialbot commented 6 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

larkz commented 6 months ago

Thank you so much for your effort in providing this very interesting piece of code.

Disclaimer: This review is provided by an individual fairly experienced in Scala and JVM development, along with having a good grasp of probability theory. While I have rudimentary knowledge of set theory and graph theory, I am not super-familiar with 'formal theory' or prior work in Blokpoel, M., & van Rooij, I. (2021).

Lines 6-9:

Regarding the phrases, "These methods allow scientists to ‘conceptually analyze, specify, and formalize intuitions that otherwise remain unexamined…" and "…They make otherwise underspecified theories precise and open for critical reflection."

This wording is rather vague, and it could apply to anything. For instance:

It would be beneficial to provide a more specific explanation of what constitutes formal theory and offer concrete examples.

Line 14:

It is recommended to omit "cannot (easily)" for precision. If a problem cannot be solved, the predictions are not "derived" but rather "approximated." Additionally, defining the term "easily" is essential – does it refer to computational time savings, memory efficiency, or ease of problem formulation for coders?

Lines 15-20:

The author contends that Scala is an ideal language for modelling mathematical expressions. Concrete examples, especially for those unfamiliar with Scala, would enhance the clarity of this assertion.

Line 25:

While the claim about backwards compatibility is generally valid, it would be beneficial to elaborate on how this constitutes a tremendous advantage over other programming languages. Backwards compatibility is a common goal for many languages/frameworks; is mathlib notably more robust in this aspect compared to others?

Lines 28-27:

The argument seems to be that Scala's functional programming nature, resembling mathematical language, is its primary advantage. Is this the sole advantage, and are there any trade-offs in terms of runtime? A comparison with a non-functional language like Java, highlighting concrete improvements in Scala, would provide a stronger stance.

Lines 46-47:

Again, similar to Line 25, a more nuanced discussion about the uniqueness of backwards compatibility in Scala compared to other languages/frameworks would be beneficial.

Line 51:

Resources: There seems to be an issue accessing the editorial notebooks in all the provided links.

And for the ones that are “not yet available”, I would expect to see more of them become available, especially for the purposes of this review, so that I (and other users) can get a quicker grasp of how to use the framework, and its potential advantages. The repository should be complete work, with core examples, not work-in-progress.

Additional Comments:

Even though the paper is supposed to be concise, there is room for improvement in the clarity of the exposition. Enhancing the presentation would allow readers, potentially lacking expertise in formal theory, to quickly understand the novelty and technical contributions of the software, which currently remain unclear and challenging to ascertain.

danielskatz commented 6 months ago

@bzz - how is your review coming?

@larkz - thanks for your work so far.

markblokpoel commented 6 months ago

@larkz thank you very much for the invaluable feedback. I'm out of office for the next two weeks, but will reply to your suggestions when I get back as soon as possible.

@danielskatz regarding the linked repository with the examples and tutorial: @larkz rightly points out these are still under development. I can fix the bugs with running the notebooks, but I perceived these as a separate academic contribution, next to the mathlib library (potentialy suitable when finished for publication in the sibling Journal of Open Source Education. Your advise as editor would be most welcome.

danielskatz commented 6 months ago

👋 @bzz and @larkz - I'm just checking in post-holidays on how your reviews are going. I look forward to seeing progress, or some discussion about what's blocking you.

danielskatz commented 6 months ago

👋 @bzz and @larkz - I'm just checking in again, when it's more likely we are post-holidays, on how your reviews are going. I look forward to seeing progress, or some discussion about what's blocking you.

danielskatz commented 6 months ago

👋 @markblokpoel - sorry I missed this somehow...

@danielskatz regarding the linked repository with the examples and tutorial: @larkz rightly points out these are still under development. I can fix the bugs with running the notebooks, but I perceived these as a separate academic contribution, next to the mathlib library (potentialy suitable when finished for publication in the sibling Journal of Open Source Education. Your advise as editor would be most welcome.

I think there are two potentially separable things:

  1. As you suggest, examples and tutorials can be separate academic contributions
  2. However, there is likely some minimum stable contribution needed as part of this submission.

So, perhaps you can separate and mark the work that is part of this submission, and that reviewers (and users) can rely on today to help them understand the software, and what is work-in-progress that will provide more support but isn't part of the minimum needed to start using the software.

danielskatz commented 5 months ago

@markblokpoel - did my comment above make sense? Does this seem reasonable?

danielskatz commented 5 months ago

👋 @bzz and @larkz - I'm just checking in again on how your reviews are going. I look forward to seeing progress, or some discussion about what's blocking you.

larkz commented 5 months ago

👋🏼 @danielskatz, I'm waiting for @markblokpoel to address my previous comments in the paper. And I look forward to seeing the updated code with examples 😃 .

markblokpoel commented 5 months ago

Dear @danielskatz and @larkz,

Thanks so much for your feedback!

@danielskatz your suggestions are reasonable, I'll try to clarify this in my revision.

@larkz I still need to find time to answer your feedback, I'm wrapping up my teaching for this semester and expect to have time for this early February.

I appreciate your patience!

danielskatz commented 4 months ago

👋 @markblokpoel - any news on your updates?

danielskatz commented 4 months ago

👋 @markblokpoel - any news on your updates?

danielskatz commented 4 months ago

@markblokpoel - it's now early March...

markblokpoel commented 4 months ago

@danielskatz I'm working on my revisions and replies, I apologize for the prolonged radio silence.

markblokpoel commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

markblokpoel commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

markblokpoel commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

markblokpoel commented 3 months ago

Dear @danielskatz and @larkz, thank you for your patience. I've completed my revisions. Please find the updated manuscript in the next post. Below I've replied to your points of feedback.

Lines 6-9: Regarding the phrases, "These methods allow scientists to ‘conceptually analyze, specify, and formalize intuitions that otherwise remain unexamined…" and "…They make otherwise underspecified theories precise and open for critical reflection." This wording is rather vague, and it could apply to anything. For instance:

Reply: Thank you for pointing out this ambiguity. I have added clarification in the summary by giving examples of the formal systems. The approach in Blokpoel and van Rooij is agnostic about the specific formal systems that theoreticians can use, which is why I prefer to give examples here and not to be concrete. In addition to the subset choice illustration, I have also included an additional illustration (viz., Coherence) to hopefully give the reader a better idea what the method is about.

Line 14: It is recommended to omit "cannot (easily)" for precision. If a problem cannot be solved, the predictions are not "derived" but rather "approximated." Additionally, defining the term "easily" is essential – does it refer to computational time savings, memory efficiency, or ease of problem formulation for coders?

Reply: I agree that the word 'easily' is confusing. I did not intend approximation. In fact approximation has in our field of computational cognitive science problems in and of itself (see e.g., van Rooij & Wareham, 2012 and Kwisthout, Wareham & van Rooij, 2011). I intended the latter meaning, which I have hopefully clarified in Footnote 1.

Lines 15-20: The author contends that Scala is an ideal language for modelling mathematical expressions. Concrete examples, especially for those unfamiliar with Scala, would enhance the clarity of this assertion.

Reply: I have extended the illustration of Subset choice and added a second illustration on Coherence theory. I have also clarified which expressions in the code have been implemented in mathlib. (Section Illustrations)

Line 25: While the claim about backwards compatibility is generally valid, it would be beneficial to elaborate on how this constitutes a tremendous advantage over other programming languages. Backwards compatibility is a common goal for many languages/frameworks; is mathlib notably more robust in this aspect compared to others? Lines 46-47: Again, similar to Line 25, a more nuanced discussion about the uniqueness of backwards compatibility in Scala compared to other languages/frameworks would be beneficial.

Reply: True backwards compatibility is of course dependent on future iterations of the software. What is important for cognitive science, is that older code can easily be run and critically analyzed, courtesy of Maven and the Scala Built Tool. In cognitive science, it is very common that old simulations cannot be run or verified, because of the complications arising in running old software. I realize this is not backwards compatibility. I still belief that Scala has an edge on other programming languages, but it seems that this argument is outside the scope of this paper. I have limited the scope on my claims on backwards compatibility, which I've hopefully addressed in the paper (Lines 25; 40--50).

Lines 28-27: The argument seems to be that Scala's functional programming nature, resembling mathematical language, is its primary advantage. Is this the sole advantage, and are there any trade-offs in terms of runtime? A comparison with a non-functional language like Java, highlighting concrete improvements in Scala, would provide a stronger stance.

Reply: I agree that ideally a comparison might make the need for mathlib clearer, but at the same time I worry that such a comparison may lead to a discusion on how one can also do functional programming in language X, which I actually think is true. Also, I do not wish to argue that cognitive sciences should use mathlib, but the principles that it encourages are relevant for any theoretical simulation. This discussion, however, is outside the scope of the article. I did try to summarize the primary advantage as you so kindly summarized (lines 51-52), and I would like to point out that line 55-57 expresses the tradeoff we considered: usability and transparency versus computational expressiveness and efficiency.

Line 51: / Resources: There seems to be an issue accessing the editorial notebooks in all the provided links.

Reply: The tutorials are work-in-progress to be submitted separately to the Journal of Open Source Education. I've removed the reference to them in the paper, as they are related to, but not part of this submission. I did update the demos in the mathlib repository for the two illustrations (Subset Choice and Coherence), and these should be runnable after checking out the mathlib repository and by using sbt to compile / run the project. References to these files have been added to the paper (lines 73-74 and lines 94-95).

Additional Comments: Even though the paper is supposed to be concise, there is room for improvement in the clarity of the exposition. Enhancing the presentation would allow readers, potentially lacking expertise in formal theory, to quickly understand the novelty and technical contributions of the software, which currently remain unclear and challenging to ascertain.

Reply: Thanks again for the review @larkz . I took the liberty of expanding the paper, I hope that still fits the journal constraints, @danielskatz ? If not, I'm happy to compress or otherwise update the paper.

markblokpoel commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 3 months ago

Thanks @markblokpoel 👋 @larkz - I look forward to seeing what you think now

danielskatz commented 3 months ago

👋 @bzz - please see if you can make progress on your review.

markblokpoel commented 3 months ago

Found a typo (see updated pdf). Also, I want to let everyone know that I'm out of office from April 2nd until May 1st. I will not be able to reply to your messages in that period.

markblokpoel commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 3 months ago

👋 @larkz and @bzz - it would be great if we could make some progress on this before @markblokpoel goes out of the office. I hope you will have time to work on your reviews in this next ~2 weeks before April 2nd.

danielskatz commented 3 months ago

👋 @larkz - please let me know if you are still willing to work on this review

danielskatz commented 3 months ago

👋 @bzz - please let me know if you are still willing to work on this review

danielskatz commented 3 months ago

👋 @markblokpoel - unfortunately, it looks like we need to find new reviewers.

danielskatz commented 3 months ago

Can you make any additional suggestions for reviewers? Again, this could be from the list of people who have already agreed to review, or from outside this list.

larkz commented 2 months ago

Hello gentlemen, I'm ready to review the revised work, though it may take some time; if @danielskatz prefers to move forward quickly with other reviewers, I'm open to that. I kindly ask for your patience as I aim to complete it before the end of May.

danielskatz commented 2 months ago

@larkz - that timing is ok; thanks for being willing to come back to this.

danielskatz commented 2 months ago

@markblokpoel - can you suggest a few more potential reviewers to use in place of @bzz, who seems to have gone away?

markblokpoel commented 2 months ago

Hi @danielskatz I'm still traveling and out of office. Will get back to this when I get back in the office.