search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
714 stars 38 forks source link

[PRE REVIEW]: π-Base: Cyberinfrastructure for Formalized Results in Mathematics #6594

Closed editorialbot closed 3 months ago

editorialbot commented 5 months ago

Submitting author: !--author-handle-->@StevenClontz<!--end-author-handle-- (Steven Clontz) Repository: https://github.com/pi-base/web/ Branch with paper.md (empty if default branch): joss2024 Version: unversioned Editor: Pending Reviewers: Pending Managing EiC: Daniel S. Katz

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/037b58f14e87926171ff980f37283000"><img src="https://joss.theoj.org/papers/037b58f14e87926171ff980f37283000/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/037b58f14e87926171ff980f37283000/status.svg)](https://joss.theoj.org/papers/037b58f14e87926171ff980f37283000)

Author instructions

Thanks for submitting your paper to JOSS @StevenClontz. Currently, there isn't a JOSS editor assigned to your paper.

@StevenClontz if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands
editorialbot commented 5 months ago

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 5 months ago

:warning: An error happened when generating the pdf.

editorialbot commented 5 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.14 s (1841.1 files/s, 292333.5 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
JSON                             13            542              0          23718
TypeScript                      118            982            208           5456
YAML                              3            837              4           5409
Svelte                           85            260              1           1864
Markdown                         22            157              0            568
SVG                               2              2              2            146
JavaScript                        4              5              8             80
TeX                               1              6              0             53
CSS                               1             10              0             38
Bourne Again Shell                2             10              7             33
Dockerfile                        2             16              2             31
HTML                              1              5              0             27
Bourne Shell                      1              1              0              2
--------------------------------------------------------------------------------
SUM:                            255           2833            232          37425
--------------------------------------------------------------------------------

Commit count by author:

   250  James Dabbs
    55  Steven Clontz
     1  Marshall Williams
editorialbot commented 5 months ago

Paper file info:

📄 Wordcount for paper.md is 1601

✅ The paper includes a Statement of need section

editorialbot commented 5 months ago

License info:

✅ License found: MIT License (Valid open source OSI approved license)

danielskatz commented 5 months ago

👋 @StevenClontz

Is this is a software paper? I'm a bit confused by the programming language info, and the fact that the title seems to say this is a community database.

StevenClontz commented 5 months ago

It's custom software running at https://topology.pi-base.org and various GitHub actions that gives researchers access to the database.

StevenClontz commented 5 months ago

It's not clear why the PDF didn't build here. It's building on my branch: https://github.com/pi-base/web/actions/runs/8573939776

danielskatz commented 5 months ago

@xuanxu - can you see what's going on with the paper? it doesn't compile, and doesn't give a useful error message. Also, I don't see any information about checking the references.

StevenClontz commented 5 months ago

@editorialbot generate pdf

editorialbot commented 5 months ago

:warning: An error happened when generating the pdf.

xuanxu commented 5 months ago

The format of the date in the paper's metadata is invalid, this PR should fix it.

danielskatz commented 5 months ago

thanks @xuanxu - is this something we can flag in future processing so that authors can figure it out?

xuanxu commented 5 months ago

Currently the only clue is that the compilation message mentions a Date error. We could catch that an provide a better message here.

StevenClontz commented 5 months ago

I've merged https://github.com/pi-base/web/pull/143 although I find it surprising that your CI is choking on our use of https://en.m.wikipedia.org/wiki/ISO_8601

StevenClontz commented 5 months ago

@editorialbot generate pdf

editorialbot commented 5 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 5 months ago

@StevenClontz - thanks for working with us on getting the paper to compile in our system.

At this point, I'm still not sure that I fully understand the software aspect of this, though your message helps. I don't think reviewers will have an easy time with this, as the paper is more about things around the software than the software itself. I think the software aspects are in the paragraphs at lines 69-95. Is this correct? If so, perhaps a heading/section might be useful here.

Similarly, the README in the repo could be a bit more descriptive. Note the JOSS review criteria in the Documentation section. While it's possible that everything needed exists, it's not obvious how to get to it, and an expanded README would help.

Future steps in the JOSS process might involve a scope review and peer reviews, and I want to make sure that these can succeed by being able to access enough information to understand the submission as software.

So please consider what you can do to make this aspect more clear in the paper and README, and let me know when you have so, and then I'll see which next steps we should take.

StevenClontz commented 5 months ago

Thanks for the feedback! I'll reach out once I've had time to implement.

StevenClontz commented 5 months ago

Thinking a beat further, is a distinction between this submission and other JoSS submissions is that pi-Base is RSaaS (Research Software as a Service), utilized through the topology.pi-base.org website and GitHub infrastructure on the pi-base/data repository, rather than software installed by individual users? This is why we don't have "Installation instructions" in the Readme: none of our researchers need to actually install the software. (Similarly, we have a devcontainer for development, so installing for devs is just clicking a green button.)

danielskatz commented 5 months ago

I guess so, but what if someone else wanted to copy this whole setup for a slightly different purpose? Would they be able to modify and install this for a new server/database?

StevenClontz commented 5 months ago

A good point, and it's not so theoretical; I have a call set up for next week to explore how to set pi-Base up for another math discipline. So I'm happy to work through that exercise and return to this thread afterwards.

danielskatz commented 5 months ago

As a note, @boisgera has volunteered to edit this once we restart it

danielskatz commented 4 months ago

👋 @StevenClontz - How are things going? Any news?

StevenClontz commented 4 months ago

Hey! 👋 Sorry, I hadn't budgeted for the extra time it turned out I needed to follow up on this. We did end up supporting a fork for topos theory without providing much hands-on support, so I imagine fixing up docs to align with your requirements shouldn't be too much work, but I need to carve out the time.

StevenClontz commented 3 months ago

I have an updated submission at https://github.com/pi-base/web/actions/runs/9540630329?pr=141 that I think addresses the raised concerns. Note the new title: π-Base: Cyberinfrastructure for Formalized Results in Mathematics

Let me know next steps, and thanks for your patience.

danielskatz commented 3 months ago

@editorialbot generate pdf

danielskatz commented 3 months ago

@arfon - Is there something we need to change in a database to reflect the new title, which I changed in this issue?

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 3 months ago

Thanks @StevenClontz - I'm going to start a scope review of the editors now to make sure this is in-scope.

Copying from some of the comments above (for other editors who are reviewing the scope question):

@StevenClontz:

Thinking a beat further, is a distinction between this submission and other JoSS submissions is that pi-Base is RSaaS (Research Software as a Service), utilized through the topology.pi-base.org website and GitHub infrastructure on the pi-base/data repository, rather than software installed by individual users? This is why we don't have "Installation instructions" in the Readme: none of our researchers need to actually install the software. (Similarly, we have a devcontainer for development, so installing for devs is just clicking a green button.)

@danielskatz:

I guess so, but what if someone else wanted to copy this whole setup for a slightly different purpose? Would they be able to modify and install this for a new server/database?

@StevenClontz:

A good point, and it's not so theoretical; I have a call set up for next week to explore how to set pi-Base up for another math discipline. So I'm happy to work through that exercise and return to this thread afterwards.

Feel free to add anything else that you think is relevant.

I think this will take a couple of weeks to determine.

danielskatz commented 3 months ago

@editorialbot query scope

editorialbot commented 3 months ago

Submission flagged for editorial review.

danielskatz commented 3 months ago

👋 @StevenClontz - after quite a bit of discussion, we have decided that this submission does not fit JOSS well enough to be reviewed as research software as defined by JOSS. This does not mean that it is not software that is useful in research, but just that JOSS does not consider it in scope for review as research software. In fact, the editors who considered this all appreciate the work that was done in it, and how it serves its community. Please see https://joss.readthedocs.io/en/latest/submitting.html#other-venues-for-reviewing-and-publishing-software-packages for other suggestions for how you might receive credit for your work.

danielskatz commented 3 months ago

@editorialbot reject

editorialbot commented 3 months ago

Paper rejected.

StevenClontz commented 3 months ago

Thanks for the update. I have to admit I'm a bit perplexed that this software, which is specifically designed by mathematics researchers to support research in mathematics, isn't in scope to be considered research software by JOSS.

Your alternative suggestions aren't super helpful, as a Git DOI doesn't provide a scholarly artifact necessary for academic advancement, and the other suggestions are specific to Python and R.

I'm soapboxing a bit, as I think a lot about how mathematicians can be better credited for contributions to mathematics software infrastructure. Thanks anyway for the time your team spent on this submission.

danielskatz commented 3 months ago

@StevenClontz - I appreciate your thoughts. A key issue might be that JOSS does not consider this research software that JOSS can review, not that it's not software that's useful in research. See https://joss.readthedocs.io/en/latest/submitting.html#a-note-on-web-based-software specifically, which I probably should have pointed to in my message earlier today.

And I apologize if our suggestions aren't helpful to you. I agree that credit for software in academia is a problematic issue, and while JOSS is trying to solve a part of it, we can't solve all of it.

StevenClontz commented 3 months ago

Ah, that does more directly address the issue at play, though I felt that we clearly satisified "the web application demonstrates a high-level of rigor with respect to domain modeling and testing (e.g., adopts and implements a design pattern such as MVC using a framework such as Django)."

Are you aware of any avenue for peer review of "research software as a service", where the goal is explicitly to deliver web-based solutions to research problems? Honestly, most researchers in my field are not going to pip install anything; if I want to deliver tools that improve research, they need to be web tools. If such a thing doesn't exist, but there are others interested in addressing that vacuum, I'd be happy to chat (maybe in the US RSE Slack?).

danielskatz commented 3 months ago

Are you aware of any avenue for peer review of "research software as a service", where the goal is explicitly to deliver web-based solutions to research problems?

No.

If such a thing doesn't exist, but there are others interested in addressing that vacuum, I'd be happy to chat (maybe in the US RSE Slack?).

The US-RSE and SocRSE slacks, along with potentially social media (e.g., mastodon) might be good places to see if there are others interested in this.