[PRE REVIEW]: wiserank: a platform for running pairwise comparison experiments

[editorialbot]

Submitting author: !--author-handle-->@ianvanbuskirk<!--end-author-handle-- (Ian Van Buskirk) Repository: https://github.com/LarremoreLab/wiserank/ Branch with paper.md (empty if default branch): Version: v0.1.0 Editor: Pending Reviewers: Pending Managing EiC: Samuel Forbes

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/f8c81960d92b41e27c4eea804341862c"><img src="https://joss.theoj.org/papers/f8c81960d92b41e27c4eea804341862c/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/f8c81960d92b41e27c4eea804341862c/status.svg)](https://joss.theoj.org/papers/f8c81960d92b41e27c4eea804341862c)

Author instructions

Thanks for submitting your paper to JOSS @ianvanbuskirk. Currently, there isn't a JOSS editor assigned to your paper.

@ianvanbuskirk if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=1.12 s (37.5 files/s, 128109.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
CSV                              4              0              0         129383
JSON                             6              0              0          12402
Vuejs Component                  9             45             15            633
Python                          12            179             81            551
Markdown                         4             49              0            119
TeX                              1              2              0             28
JavaScript                       3              1              2             20
YAML                             1              1              4             19
HTML                             1              0              1             16
Bourne Shell                     1              0              0              2
-------------------------------------------------------------------------------
SUM:                            42            277            103         143173
-------------------------------------------------------------------------------

Commit count by author:

    57  ianvanbuskirk
     3  Ian Van Buskirk

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 712

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: MIT License (Valid open source OSI approved license)

[editorialbot]

:warning: An error happened when generating the pdf. Problem with affiliations for Daniel Larremore, perhaps the affiliations index need quoting?.

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- None

🟡 SKIP DOIs

- No DOI given, and none found for title: SpringRank

❌ MISSING DOIs

- 10.1371/journal.pone.0123483 may be a valid DOI for title: Wiki surveys: Open and quantifiable social data co...
- 10.1126/sciadv.aar8260 may be a valid DOI for title: A physical model for efficient ranking in networks

❌ INVALID DOIs

- None

[samhforbes]

Hi @ianvanbuskirk thanks for submitting to JOSS. We typically will discuss whether a submission of this size is in scope for JOSS before commencing review, so I'm going to do that here. If you have thoughts as to why this is in scope given our requirements you can post that here. In the meantime you can see some issues flagged by editorialbot that can be addressed.

[samhforbes]

@editorialbot query scope

[editorialbot]

Submission flagged for editorial review.

[ianvanbuskirk]

Hi @samhforbes, thanks for your prompt response. I've made some edits based on the editorialbot's suggestions.

With respect to scope, my collaborators and I discussed JOSS's note on web-based software before deciding to submit. Ultimately, we came to think the wiserank platform could be a valuable tool for researchers as a web-based application, but above all else contributes a core set of functionalities for running a pairwise comparison experiment. These functionalities could be used to run a local, lab-based experiment or to simulate comparisons and study how different experimental designs impact the results. Overall, we hope wiserank can be a rigorous starting point for many kinds of projects that involve pairwise comparisons.

Please let me know if there are other scope related topics I should touch on!

[ianvanbuskirk]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[editorialbot]

Five most similar historical JOSS papers:

Autorank: A Python package for automated ranking of classifiers Submitting author: @sherbold Handling editor: @arfon (Active) Reviewers: @JonathanReardon, @ejhigson Similarity score: 0.6937

Multi-attribute task builder Submitting author: @Yury-Shevchenko Handling editor: @alexhanna (Retired) Reviewers: @u01ai11 Similarity score: 0.6870

psychTestR: An R package for designing and conducting behavioural psychological experiments Submitting author: @pmcharrison Handling editor: @oliviaguest (Active) Reviewers: @Cognitive-Modeller, @DivyaSeernani Similarity score: 0.6808

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing Submitting author: @stsievert Handling editor: @ajstewartlang (Active) Reviewers: @hoechenberger, @stain, @jorgedch Similarity score: 0.6790

PyExperimenter: Easily distribute experiments and track results Submitting author: @tornede Handling editor: @timtroendle (Retired) Reviewers: @ArsamAryandoust, @schnorr Similarity score: 0.6758

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[samhforbes]

Hi @ianvanbuskirk Thanks for submitting to JOSS. After editorial review unfortunately it was decided that wiserank does not fall into our scope. I should stress that this is no comment on the usefulness or quality of the package.

[samhforbes]

@editorialbot reject

[editorialbot]

Paper rejected.