[PRE REVIEW]: CiteLang: Modeling the Research Software Ecosystem

editorialbot commented 2 years ago

Submitting author: !--author-handle-->@vsoch@faroit<!--end-editor-- Reviewers: @gflofst, @rmmilewi Managing EiC: Kevin M. Moerman

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/3e3f4e819e6f9400ba204dbae4b9c3d5"><img src="https://joss.theoj.org/papers/3e3f4e819e6f9400ba204dbae4b9c3d5/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/3e3f4e819e6f9400ba204dbae4b9c3d5/status.svg)](https://joss.theoj.org/papers/3e3f4e819e6f9400ba204dbae4b9c3d5)

Author instructions

Thanks for submitting your paper to JOSS @vsoch. Currently, there isn't an JOSS editor assigned to your paper.

@vsoch if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

editorialbot commented 2 years ago

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 2 years ago

Software report:

github.com/AlDanial/cloc v 1.88  T=0.09 s (1299.8 files/s, 120382.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          61           1156           1166           4207
Markdown                        19            137              0            608
HTML                             4            106              0            532
YAML                            12             79             16            464
reStructuredText                 8            511            532            411
SVG                              2              1             21            239
make                             1             28              6            143
TeX                              1             19              0            123
Bourne Shell                     4             29             20            105
CSS                              2             17              8             82
XML                              1              0              0             30
JSON                             1              0              0             22
Dockerfile                       1              3              1             14
-------------------------------------------------------------------------------
SUM:                           117           2086           1770           6980
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

editorialbot commented 2 years ago

Wordcount for paper.md is 1398

editorialbot commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5334/jors.by is OK
- 10.1109/MS.2020.2973362 is OK
- 10.48550/ARXIV.1905.08674 is OK
- 10.1109/MIC.2014.88 is OK
- 10.1109/CHASE.2019.00039 is OK
- 10.7717/peerj-cs.86 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.1016/S0950-5849(02)00049-6 is INVALID because of 'https://doi.org/' prefix

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

vsoch commented 2 years ago

@editorialbot commands

editorialbot commented 2 years ago

Hello @vsoch, here are the things you can ask me to do:


# List all available commands
@editorialbot commands

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Check the references of the paper for missing DOIs
@editorialbot check references

# Perform checks on the repository
@editorialbot check repository

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for branch
@editorialbot set joss-paper as branch

# Generates the pdf paper
@editorialbot generate pdf

# Get a link to the complete list of reviewers
@editorialbot list reviewers

vsoch commented 2 years ago

@editorialbot check references

editorialbot commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5334/jors.by is OK
- 10.1016/S0950-5849(02)00049-6 is OK
- 10.1109/MS.2020.2973362 is OK
- 10.48550/ARXIV.1905.08674 is OK
- 10.1109/MIC.2014.88 is OK
- 10.1109/CHASE.2019.00039 is OK
- 10.7717/peerj-cs.86 is OK

MISSING DOIs

- None

INVALID DOIs

- None

Kevin-Mattheus-Moerman commented 2 years ago

@arfon do you have the bandwidth to handle this one as editor?

Kevin-Mattheus-Moerman commented 2 years ago

@editorialbot invite @faroit as editor

editorialbot commented 2 years ago

Invitation to edit this submission sent!

Kevin-Mattheus-Moerman commented 2 years ago

@faroit can you handle this work as editor?

Kevin-Mattheus-Moerman commented 2 years ago

@editorialbot invite @faroit as editor

editorialbot commented 2 years ago

Invitation to edit this submission sent!

Kevin-Mattheus-Moerman commented 2 years ago

@faroit :wave:

faroit commented 2 years ago

@Kevin-Mattheus-Moerman hi, sorry for not replying and delaying the process. I was on vacation and forgot to set my status. Let me have a look at the paper today and then come back to this issue.

Kevin-Mattheus-Moerman commented 2 years ago

@faroit any news, can you help edit this? It would be great if you could, even if it is outside your domain a bit.

faroit commented 2 years ago

@Kevin-Mattheus-Moerman I spent some time getting into this now and - yes, while its clearly outside my core domain – I find the contribution interesting and relevant. Happy to accept

Kevin-Mattheus-Moerman commented 2 years ago

@editorialbot assign @faroit as editor

editorialbot commented 2 years ago

Assigned! @faroit is now the editor

faroit commented 2 years ago

👋 Hi @vsoch, thanks again for your submission to JOSS and sorry for the delay. I will be serving as the editor for this now. We'll use this issue to identify reviewers and resolve technical issues.

This tool implements a broad (and alternative) vision of how research software can be cited outside classical DOIs. I would like to diversify the reviewers in order to accommodate this vision but also assess the implementation details at the same time. E.g. at least one very experienced senior researcher that can comment on the broader vision of citelang. But also one experienced python expert.

I think the paper is already in a good shape. Looking at the repository however, one thing I clearly miss is more polished examples of typical outputs from popular repositories in order to better understand the vision of citelang and how it is recommended to be used by research software engineers.

I feel that potential reviewers would like to quickly assess what citelang is addressing for their citation/research software workflow. Therefore I would suggest - before we find potential reviewers - to update the repository to accommodate:

[x] more examples outputs, ideally in a separate html docs gallery.
[x] more detailed "getting started" guide that describes the vision and how citelang addresses the issues raised in the JOSS paper. Ideally an example workflow. I would also think that part of your blog post can be used as material.
[x] clarification of why the software is subtitled "markdown syntax for citing software" - what is the relation to markdown?

@vsoch let me know if you think this sounds appropriate to you

vsoch commented 2 years ago

hey @faroit totally no worries about the timing - it's an amazing effort that y'all do here to do this in your free time (and it's still pretty fast imho).

I think the paper is already in a good shape. Looking at the repository however, one thing I clearly miss is more polished examples of typical outputs from popular repositories in order to better understand the vision of citelang and how it is recommended to be used by research software engineers.

I'm happy to add this, but did you not see the docs? https://vsoch.github.io/citelang/getting_started/user-guide.html. CiteLang is a fairly diverse tool and there is no "here is the one use case" or "this is exactly how you are supposed to us it" for it. It's main purpose is that it's an API for getting data about contributions, turning that into a graph structure, and the intention (and why it is research software) is that you can use these graphs to understand an ecosystem as I've done with the rsepedia-analysis. That entire thing is derived from CiteLang data, and akin to how a researcher would use numpy in their project to do something higher level, I used citelang. The render/graph/badge provided with the software are extra bells and whistles, or how I chose to use these graphs (and provide fun tools for the interested user). So let's chat about what you think is missing - granted that you saw that documentation that has links to:

badges that show the entire credit tree (including deps). That is shown in the README and in the docs here for multiple variations.
summarizing contributions, described here
interactive badges and credit trees linked from docs
The functionality that originally derived the name - a "markdown syntax for citations" there is a pre and post rendering, where citelang reads in the pre, parses the citation and generates those same credit trees, and makes a nice table. That is also done automated and present in the citelang repository.

And the content of the paper is somewhat represented here and at the top of here.

I feel that potential reviewers would like to quickly assess what citelang is addressing for their citation/research software workflow. Therefore I would suggest - before we find potential reviewers - to update the repository to accommodate:

I don't think I'm saying this is yet a part of any research software citation workflow? I think right now our workflow is "make bibtex or a DOI with Zenodo and copy paste into a bibliography" and CiteLang is saying "this could be different." CiteLang is intended to be provided to empower people to model the space and think more about these ideas. To go back to the numpy example, numpy itself isn't explicitly stated in some workflow for a domain of analysis, but it's an underlying library that forms components that turn out to be. Let me know if that makes sense.

more examples outputs, ideally in a separate html docs gallery.

So I'm happy to add what you think is missing, but I don't see that any of the functionality that is provided is missing. Could you tell me what you think is missing? Do you want some sort of a "quick summary" bullet list of things that link to the same examples? Specifically what output are you looking for? I thought the requirement for JoSS was robust documentation, which I think I have with every example that the software can do, so I apologize if I missed an extra requirement (and I'd like to ask for clarity or a link to what you are looking for).

more detailed "getting started" guide that describes the vision and how citelang addresses the issues raised in the JOSS paper. Ideally an example workflow. I would also think that part of your blog post can be used as material.

Would this also be addressed by this summary listing? I think I present the tool as one idea for how we can influence the future of credit for software, but I don't make claims that there is some single command here that you run and "that's the answer." CiteLang has a lot of cool functionality and commands, but those are more examples of what I've built using these APIs and graphs. I'm hoping other researchers interested in software credit might do similar and make other cool ideas.

clarification of why the software is subtitled "markdown syntax for citing software" - what is the relation to markdown?

This is the syntax shown in the links above that is shown with render.

And there are GitHub actions for all of those things so if someone just wants to "generate a cool badge or a software credit graph from a paper they have written in markdown" that's probably the most direct usage, along with summarizing contributions. E.g., the singularity project is using CIteLang to be able to say thank you to contributions for each release, here is an example I was just checking with one of their developers:

Thank you for your comments!

faroit commented 2 years ago

@vsoch thanks for your detailed explanations.

I'm happy to add this, but did you not see the docs? https://vsoch.github.io/citelang/getting_started/user-guide.html. CiteLang is a fairly diverse tool and there is no "here is the one use case" or "this is exactly how you are supposed to us it" for it.

Thanks for the clarification. In the submitted paper you state:

"On a cultural level, CiteLang is the first tool of its kind to suggest an alternative way to value software beyond the traditional academic practice of publication. It suggests a model where RSEng are not required to generate separate DOIs (digital object identifiers) associated with papers..."

That is why, when looking only at the repo, I was a bit overwhelmed by the possibilities of citelang - in a very positive way :-) As I am not doing research on research software engineering, I was directly looking for an (opinionated), best-practice guide of how to use citelang in a research software package. I understand now that this wasn't the software's intention and you believe that this wouldn't be helpful for users – That's okay for me then and we can still come back to this later during the review process if reviewers ask for it.

So I'm happy to add what you think is missing, but I don't see that any of the functionality that is provided is missing. Could you tell me what you think is missing? Do you want some sort of a "quick summary" bullet list of things that link to the same examples? Specifically what output are you looking for?

Sorry, i wasn't clear. The example outputs are extensive enough. Then, a very quick summary at the top of #usage would be indeed helpful. E.g. what would a research software engineer typically use Credit, Graph and Badges for and how would one usually present them in a paper or a website -> example links.

I thought the requirement for JoSS was robust documentation, which I think I have with every example that the software can do, so I apologize if I missed an extra requirement (and I'd like to ask for clarity or a link to what you are looking for).

From what I see, the documentation meets all requirements!

This is the syntax shown in the links above that is shown with render.

:+1: Maybe a bit of this could be added to the readme.md?

faroit commented 2 years ago

Proceeding with the review... @vsoch, I have one or two potential reviewers in mind but I am also interested in your input on potential reviewers, so any suggestions you have are also welcome.

For this, please check the JOSS COI policy and only mention them with a space between the @ and their handle (so you would refer to me as "@ faroit").

vsoch commented 2 years ago

That's okay for me then and we can still come back to this later during the review process if reviewers ask for it.

Yep that works perfectly! Sorry about my wording, we indeed can work on that. I have to be careful because I feel like I can easily offend someone when I talk about this (it already happened in a slack). I think that's what led to "suggests."

Sorry, i wasn't clear. The example outputs are extensive enough. Then, a very quick summary at the top of #usage would be indeed helpful. E.g. what would a research software engineer typically use Credit, Graph and Badges for and how would one usually present them in a paper or a website -> example links.

This sounds perfect! I'll put together a quick "why do I care!" bullet list and add to the README and to the top of the getting started guide.

Proceeding with the review... @vsoch, I have one or two potential reviewers in mind but I am also interested in your input on potential reviewers, so any suggestions you have are also welcome.

Yes of course! And here is the spreadsheet for others - I couldn't easily find it. I can't remember all my old reviews, but I can definitely pick people's names that I don't recognize, here is a random search for "reproducible".

@ gflofst @ daissi @ fabianomenegidio.

vsoch commented 2 years ago

All done @faroit - I've added more summary / pictures to the README https://github.com/vsoch/citelang#getting-started along with a bullet summary section: https://vsoch.github.io/citelang/getting_started/index.html and added more headers to the GitHub actions to more easily link readers directly to the one they want (it was just one section before).

vsoch commented 2 years ago

Quick reference to supplement my description above - here is how the Singularity project is using it - https://sylabs.io/2022/05/whats-in-a-singularityce-release/ and https://twitter.com/Singularity_CE/status/1528785462210469889

faroit commented 2 years ago

here is a random search for "reproducible". @ gflofst @ daissi @ fabianomenegidio.

@vsoch I did some research and contacted some people working in this area. I would also suggest: @ sdruskat @ hainesr @ yochannah - do you see any conflicts of interests with respect to the JOSS COI policy?

vsoch commented 2 years ago

I haven't worked with any of them - the only connection is that I interviewed the second on RSE Stories a few years ago: https://us-rse.org/rse-stories/2020/robert-haines/.

faroit commented 2 years ago

👋 @rmmilewi @gflofst @sdruskat @hainesr @yochannah - would any of you be willing to review this submission for JOSS? The JOSS review process takes place on GitHub and focuses on the software and a short paper. We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The software under review is CiteLang.

This issue is a "pre-review" issue in which reviewers are assigned. Once sufficient reviewers are recruited we will open a dedicated review issue where the review will take place.

hainesr commented 2 years ago

I could probably review this - my reticence is that I don't count Python in the set of ecosystems that I am confident in. I am on leave next week and then very busy in June, but so long as "sometime in June" is an acceptable timeframe, then I can do this.

gflofst commented 2 years ago

I can and would like to do this.

I know someone perfect we should add though. Reed Milewicz (rmilewi@sandia.gov). This is exactly his area and he is likely to generate citations as well.

faroit commented 2 years ago

@editorialbot add @gflofst as reviewer

editorialbot commented 2 years ago

@gflofst added to the reviewers list!

faroit commented 2 years ago

@gflofst thanks – great to have you on board here.

@vsoch do you see a COI with respect to @ rmmilewi?

vsoch commented 2 years ago

I know him from USRSE (he gives great talks!) but we’ve never worked together, so no I don’t think there is one.

gflofst commented 2 years ago

@vsoch I don't know all of his COI, but I believe this is safe.

sdruskat commented 2 years ago

I'm very interested in reviewing this, as this is extremely relevant for my own research. Would like to get a few days to clarify a potential COI before I confirm if that's okay?

faroit commented 2 years ago

@hainesr my reticence is that I don't count Python in the set of ecosystems that I am confident in.

I had a look at the code base and I think a deep knowledge of the python packaging system is likely to be useful. Thanks for your quick reply, though!

sdruskat commented 2 years ago

@faroit, sorry, I will have to pass on this opportunity to review.

rmmilewi commented 2 years ago

Sure! I'd be happy to do so!

On Wed, Jun 1, 2022, 11:34 PM Fabian-Robert Stöter @.***> wrote:

👋 @rmmilewi https://github.com/rmmilewi @gflofst https://github.com/gflofst @sdruskat https://github.com/sdruskat @hainesr https://github.com/hainesr @yochannah https://github.com/yochannah - would any of you be willing to review this submission for JOSS? The JOSS review process takes place on GitHub and focuses on the software and a short paper. We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The software under review is CiteLang https://github.com/vsoch/citelang.

This issue is a "pre-review" issue in which reviewers are assigned. Once sufficient reviewers are recruited we will open a dedicated review issue where the review will take place.

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/4352#issuecomment-1139626698, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD3OEVMFYCILQJTUKDSKVI3VNBBU7ANCNFSM5UOTXLMQ . You are receiving this because you were mentioned.Message ID: @.***>

faroit commented 2 years ago

@editorialbot add @rmmilewi as reviewer

editorialbot commented 2 years ago

@rmmilewi added to the reviewers list!

faroit commented 2 years ago

@editorialbot generate pdf

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

hainesr commented 2 years ago

I had a look at the code base and I think a deep knowledge of the python packaging system is likely to be useful. Thanks for your quick reply, though!

I thought as much! No worries

vsoch commented 2 years ago

@editorialbot generate pdf

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

vsoch commented 2 years ago

I'm not sure if the bot here is using a cache? I fixed the URLs at the bottom to address https://github.com/vsoch/citelang/issues/41 (see preview attached from Whedon preview service) but it doesn't appear to be reflected above. whedon-preview-service.pdf

openjournals / joss-reviews

[PRE REVIEW]: CiteLang: Modeling the Research Software Ecosystem #4352

Status