search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
720 stars 38 forks source link

[PRE REVIEW]: GGMnonreg: Non-Regularized Gaussian Graphical Models #3148

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @donaldRwilliams (Donald Williams ) Repository: https://github.com/donaldRwilliams/GGMnonreg Version: 1.0.0 Editor: @arfon Reviewers: @AlexChristensen, @GiulioCostantini Managing EiC: Daniel S. Katz

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Author instructions

Thanks for submitting your paper to JOSS @donaldRwilliams. Currently, there isn't an JOSS editor assigned to your paper.

@donaldRwilliams if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands
whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.11 s (228.8 files/s, 37370.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
TeX                              1            109              0            771
R                               19            315           1439            667
Markdown                         2            112              0            361
Rmd                              1             57            123             56
YAML                             2             15              6             52
-------------------------------------------------------------------------------
SUM:                            25            608           1568           1907
-------------------------------------------------------------------------------

Statistical information for the repository 'd84b6385247e0fe6b549dd44' was
gathered on 2021/04/05.
No commited files with the specified extensions were found.
whedon commented 3 years ago

Failed to discover a valid open source license.

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1371/journal.pone.0027407 is OK
- 10.3389/fncom.2015.00132 is OK
- 10.1016/j.brat.2016.06.006 is OK
- 10.1146/annurev-statistics-022513-115545 is OK
- 10.1080/00273171.2017.1379379 is OK
- 10.1080/00273171.2018.1454823 is OK
- 10.1111/bmsp.12173 is OK
- 10.3758/s13428-017-0910-x is OK
- 10.31234/osf.io/ad57p is OK
- 10.1080/00273171.2019.1575716 is OK
- 10.31234/osf.io/fb4sa is OK
- 10.3758/s13423-016-1221-4 is OK
- 10.1177/0013164416669201 is OK
- 10.31234/OSF.IO/X8DPR is OK
- 10.31234/osf.io/ypxd8 is OK
- 10.1080/00031305.2018.1549100 is OK
- 10.1016/j.csda.2007.09.008 is OK
- 10.1037/abn0000446 is OK
- 10.1080/00273171.2019.1640103 is OK
- 10.1214/009053606000000821 is OK
- 10.1037/met0000254 is OK
- 10.1214/17-BA1092 is OK
- 10.1214/07-AOAS107 is OK
- 10.3758/s13423-016-1221-4 is OK
- 10.1037/met0000201 is OK
- 10.1080/10618600.2012.679239 is OK
- 10.1080/01621459.1993.10476321 is OK
- 10.1007/bf00162520 is OK
- 10.1191/1471082x04st063oa is OK
- 10.1080/01621459.1988.10478640 is OK
- 10.21236/ada241408 is OK
- 10.1198/004017008000000064 is OK
- 10.21105/joss.00026 is OK
- 10.31234/osf.io/p69m7 is OK
- 10.1214/aos/1176346785 is OK
- 10.31234/osf.io/zw7pf is OK
- 10.3758/s13428-017-0910-x is OK
- 10.1080/00273171.2019.1614898 is OK
- 10.1214/13-ejs854 is OK
- 10.1109/vetecs.2005.1543265 is OK
- 10.18637/jss.v048.i04 is OK
- 10.18637/jss.v089.i03 is OK
- fpsyg.2019.01050 is OK
- 10.1080/00273171.2020.1716673 is OK
- 10.31234/osf.io/4kfjp is OK
- 10.1063/1.4870402 is OK
- 10.1186/1752-0509-3-55 is OK
- 10.1007/s41109-020-0251-z is OK
- 10.1093/biostatistics/kxt005 is OK

MISSING DOIs

- 10.1007/bf02985802 may be a valid DOI for title: The elements of statistical learning: data mining, inference, and prediction
- 10.1002/sta4.23 may be a valid DOI for title: A direct sampler for G-Wishart variates

INVALID DOIs

- 10.1201/b18401-1 is INVALID
whedon commented 3 years ago

PDF failed to compile for issue #3148 with the following error:

/app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:204:in block in parse_authors': Author (Donald R. Williams) is missing affiliation (RuntimeError) from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:202:ineach' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:202:in parse_authors' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:93:ininitialize' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/processor.rb:38:in new' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/processor.rb:38:inset_paper' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/bin/whedon:58:in prepare' from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:inrun' from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in invoke_command' from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:indispatch' from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in start' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/bin/whedon:131:in<top (required)>' from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in load' from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in

'

donaldRwilliams commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

donaldRwilliams commented 3 years ago

@danielskatz

Suggested reviewers: Payton Jones Sacha Epskamp

donaldRwilliams commented 3 years ago

@whedon commands

whedon commented 3 years ago

Here are some things you can ask me to do:

# List Whedon's capabilities
@whedon commands

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Ask Whedon to check the references for missing DOIs
@whedon check references

# Ask Whedon to check repository statistics for the submitted software
@whedon check repository
donaldRwilliams commented 3 years ago

@danielskatz

I want to address the number of line of code, as the package appears meager.

That said, the package includes many analytic solutions, that I (with colleagues) have introduced to the social sciences. For example, one novel method is used to compute the expected replication of edges in large networks. This is derived analytically (few lines), whereas, if pursuing a simulation based approach, this would take many (many) more lines of code (but not necessary).

Further, to date, our three peer-reviewed papers describing the methods in this package have been cited over 100 times.

(also, (1) it was submitted to CRAN a few days ago; and (2) I added the license)

danielskatz commented 3 years ago

@donaldRwilliams - thanks for your message - the editors will still go ahead and discuss it, and hopefully get back to you in a week or so

danielskatz commented 3 years ago

@whedon query scope

whedon commented 3 years ago

Submission flagged for editorial review.

donaldRwilliams commented 3 years ago

@danielskatz

Thanks !

Also note there are many other functions (e.g., 10 or so) that I took out for publishing version 1.0.0, but I can happily add them back in, if desired.

arfon commented 3 years ago

@donaldRwilliams - would it be possible to see the diff between this version submitted and the version with the additional functions? Currently I think this submission is too small to be considered.

donaldRwilliams commented 3 years ago

@arfon Sure, I can put those back in. How shall we do this ? I am currently out of town, but can certainly add them back in very very soon.

I wrote many of the functions that were removed in my first year of graduate school (more than 3 years ago), and I am now approaching my last year, so they were not concise and clearly written. That is why I removed them.

I can withdraw submission, and then resubmit ?

Or ?

PS I would also add that the length of code is but one criterion. Our papers describing the methods have been cited well over 100 times, with one in a flagship journal for psychometrics. Also, even without a proper citation, this package has been cited 5 times (obviously, this indicates it has been used more, as often mere R packages are not properly cited. And also that most have been citing the actual papers describing the methods). I say this because some other criteria is impact and whether others will use it. Here are two of the papers, doing quite well in terms of impact.

image

arfon commented 3 years ago

Sorry for the delayed responses here @donaldRwilliams.

Our papers describing the methods have been cited well over 100 times, with one in a flagship journal for psychometrics.

Could you say a little more about this please? Specifically, are descriptions of the methods implemented in this software already published in another journal?

donaldRwilliams commented 3 years ago

@arfon

Not sure what you are asking. Can you clarify ?

donaldRwilliams commented 3 years ago

@arfon

"descriptions of the methods implemented in this software already published in another journal"

Are you asking if this work has essentially been published elsewhere with this ? The answer is no.

(but again it is not entirely clear to me what you mean)

E.g., wouldn't we hope that most (if not all) statistical methods making their way into software are peer-reviewed themselves ? Isn't that often the case ?

In fact, I am just now reviewing a paper for JOSS that is a software implementation of a method that was published in a peer-reviewed journal (which would seem to be the norm).

arfon commented 3 years ago

Apologies, I should have been a little more explicit in my question here.

Our papers describing the methods have been cited well over 100 times, with one in a flagship journal for psychometrics.

I'm responding to this comment ☝️ that you made above about there being papers describing the methods already in the literature.

E.g., wouldn't we hope that most (if not all) statistical methods making their way into software are peer-reviewed themselves ? Isn't that often the case ?

In theory yes! But we're also sensitive to the idea of double-publishing for the same scholarly work, i.e., a paper describing the method, and a second (likely JOSS) paper describing the implementation of the method. In most cases, the work required for each of these is substantial, and so each paper in its own right represents a substantial scholarly contribution and so it's reasonable that there might be two papers.

In this situation, where the software contribution is definitely on the smaller side of what we consider allowable in JOSS (see https://blog.joss.theoj.org/2020/07/minimum-publishable-unit), whether or not there are already publications related to this one is a factor in our decision making here.

Hopefully this helps? I'm afraid I still don't have clarity on how this submission relates to this work: "Our papers describing the methods have been cited well over 100 times, with one in a flagship journal for psychometrics."

donaldRwilliams commented 3 years ago

@arfon I am sorry, but I have to be frank.

I am quite literally reviewing a JOSS work that is actually fewer lines of code and computes a summary of what my package would provide. I have even consulted with the other reviewer, as we are both in the field, and he agreed that this is quite frustrating given the scope of this package compared to that one

The package here does the following image

Note I understand you are busy, but clearly having a package with fewer lines of code that computes one summary compared to a package that does all this seems a bit of an awkward situation (seems to not make much sense).

Anyhow, I am in quantitative methods in psychology. I often develop statistical methods. For example, in one case, we "invented" an approach to test differences in large networks. The software implementation is in this package. The primary paper did not even discuss the software.

As I mentioned, I can happily withdraw the package and resubmit.

Thanks !

donaldRwilliams commented 3 years ago

@arfon

Note also I really much like the aims of this journal and respect very much the work you all do. And this will in no way compromise my review of that paper (https://github.com/openjournals/joss-reviews/issues/2944).

If you are going to reject the paper, please just do so with the option to resubmit. It is this back and forth that is frustrating for me (because meanwhile I am reviewing a JOSS paper in my field that does way less and is fewer lines of code).

I can just add the other functions in, when I have time, and we can start over :-)

Best, Donny

arfon commented 3 years ago

Hi @donaldRwilliams - I'm sorry that this is frustrating, and I understand why you are inclined to compare this submission with #2944. At this point though, the size of submissions (lines of code etc.) not at the core of my outstanding concern here.

My concern at this stage is getting clarity from you on the question I asked above which I will ask again for clarity:

How does this submission relate to any other published work about the same method and or software?

In the above comment you say "Our papers describing the methods have been cited well over 100 times, with one in a flagship journal for psychometrics.". This comment raises a concern of potential duplicate publishing of the same work which would mean we could not review this work.

donaldRwilliams commented 3 years ago

@arfon

This is still not entirely clear to me

"How does this submission relate to any other published work about the same method and or software?"

Maybe clarifying what the package does will help:

The package estimates 6 different kinds of graphical models, and it is the only R package to do so. And there is a need to do so in my field. The methods for estimating graphical models go back decades (not our work), but there is no software that implements them all. Perhaps if we searched hard, we can find implementations of some of them scattered about (honestly not sure, as my search came up empty). But, really, this is the only package that is a one-stop shop for graphical modeling.

Then there are some additional methods, e.g., for testing differences in those graphical models, maybe quantifying expected replicability, etc. These are the ones we introduced to the field (and I noted above), but are not the focus of the package at all. The focus is graphical modeling, which is becoming very, very popular in my field, but no software.

Does this help ?

Perhaps from my previous comment you may have thought the package only implements what is in our papers ?

arfon commented 3 years ago

Perhaps from my previous comment you may have thought the package only implements what is in our papers ?

That's exactly what I was concerned about but I realize that's not the case now. Thank you for your patience explaining this @donaldRwilliams.

arfon commented 3 years ago

@whedon assign me as editor

whedon commented 3 years ago

OK, the editor is @arfon

arfon commented 3 years ago

Suggested reviewers: Payton Jones Sacha Epskamp

@donaldRwilliams - do you know the GitHub handles for these individuals so I can invite them here?

donaldRwilliams commented 3 years ago

Perhaps from my previous comment you may have thought the package only implements what is in our papers ?

That's exactly what I was concerned about but I realize that's not the case now. Thank you for your patience explaining this @donaldRwilliams.

Thank you as well for working through this with me. I realize now that I was not clear about what the package did, etc.

Anyhow, here is another reviewer.

@AlexChristensen

And also

@paytonjjones

If those do not work out, I can provide some others.

arfon commented 3 years ago

:wave: @AlexChristensen and @paytonjjones - would you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The submission under review here is GGMnonreg: Non-Regularized Gaussian Graphical Models.

paytonjjones commented 3 years ago

Unfortunately no, I'm out of academia now and wouldn't be able to get a review done in a timely manner.

On Fri, May 7, 2021 at 11:44 AM Arfon Smith @.***> wrote:

πŸ‘‹ @AlexChristensen https://github.com/AlexChristensen and @paytonjjones https://github.com/paytonjjones - would you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The submission under review here is GGMnonreg: Non-Regularized Gaussian Graphical Models https://github.com/donaldRwilliams/GGMnonreg.

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/3148#issuecomment-834686181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFQWBJKD7BRDQ64W2ZEGR7LTMQYI3ANCNFSM42M6QKEA .

chrisbrunsdon commented 3 years ago

Is it possible to remove me from this? I now have 13000 or so emails in mail box and only around 300 not openjournals/jossreviews related...

Sent from my iPhone

On 7 May 2021, at 8:15 p.m., Payton J. Jones @.***> wrote:

ο»Ώ Unfortunately no, I'm out of academia now and wouldn't be able to get a review done in a timely manner.

On Fri, May 7, 2021 at 11:44 AM Arfon Smith @.***> wrote:

πŸ‘‹ @AlexChristensen https://github.com/AlexChristensen and @paytonjjones https://github.com/paytonjjones - would you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The submission under review here is GGMnonreg: Non-Regularized Gaussian Graphical Models https://github.com/donaldRwilliams/GGMnonreg.

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/3148#issuecomment-834686181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFQWBJKD7BRDQ64W2ZEGR7LTMQYI3ANCNFSM42M6QKEA .

β€” You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

danielskatz commented 3 years ago

@chrisbrunsdon - sorry - please see the instructions in https://github.com/openjournals/joss-reviews/issues/3265#issuecomment-834821098 for how to stop watching this...

donaldRwilliams commented 3 years ago

@arfon @danielskatz should I suggest more possible reviewers ?

arfon commented 3 years ago

@arfon @danielskatz should I suggest more possible reviewers ?

Sure, a few more can't hurt. Please list their GitHub handles in a way that doesn't cause a notification, i.e., arfon rather than @arfon.

AlexChristensen commented 3 years ago

@arfon and @donaldRwilliams,

I'd be happy to review this package

arfon commented 3 years ago

@whedon add @AlexChristensen as reviewer

whedon commented 3 years ago

OK, @AlexChristensen is now a reviewer

arfon commented 3 years ago

:zap: thanks so much @AlexChristensen! I'd like to find a second reviewer before starting the actual review thread (with checkboxes etc.) if that sounds OK?

donaldRwilliams commented 3 years ago

@arfon

Giulio Costantini sacha epskamp

arfon commented 3 years ago

:wave: @GiulioCostantini @SachaEpskamp - would you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The submission under review here is GGMnonreg: Non-Regularized Gaussian Graphical Models.

SachaEpskamp commented 3 years ago

Hi! I would love to review this package as it looks like an excellent package indeed. Unfortunately, however, I have paternity leave coming up very fast now and too many deadlines before to finish. Some people that come to mind perhaps in addition to Giulio are Hudson Golino, Jonas Haslbeck, Alex Millner, Jill de Ron & Myrthe Veenman.

arfon commented 3 years ago

:wave: @hfgolino @Jillderon – would you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

The submission under review here is GGMnonreg: Non-Regularized Gaussian Graphical Models.

GiulioCostantini commented 3 years ago

Hi all, Sorry, I missed the first email. I would like to review this package, when is the deadline for completing this task? Best, Giulio

Giulio Costantini, PhD Assistant professor Psychology Department University of Milano-Bicocca P.zza dell'Ateneo Nuovo, 1 (U6) - room 3127 20126 Milan (MI) Phone ++39 02.6448.3839

Il giorno ven 21 mag 2021 alle ore 19:47 Sacha Epskamp < @.***> ha scritto:

Hi! I would love to review this package as it looks like an excellent package indeed. Unfortunately, however, I have paternity leave coming up very fast now and too many deadlines before to finish. Some people that come to mind perhaps in addition to Giulio are Hudson Golino, Jonas Haslbeck, Alex Millner, Jill de Ron & Myrthe Veenman.

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/3148#issuecomment-846131855, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABR376MQ3ADNVKKIH32K5Z3TO2MERANCNFSM42M6QKEA .

arfon commented 3 years ago

I would like to review this package

⚑ thanks!

when is the deadline for completing this task?

@GiulioCostantini - ideally we'd want the review to be complete within 4-6 weeks but we ask reviewers to start before then as JOSS reviews are generally quite iterative (authors making updates as feedback is received).

I'm going to add you as a reviewer now and start the main review issue. If this timing doesn't work, please let me know, we can probably work with your schedule.

arfon commented 3 years ago

@whedon add @GiulioCostantini as reviewer

whedon commented 3 years ago

OK, @GiulioCostantini is now a reviewer

arfon commented 3 years ago

@whedon start review

whedon commented 3 years ago

OK, I've started the review over in https://github.com/openjournals/joss-reviews/issues/3308.