[PRE REVIEW]: Jury: A Comprehensive Evaluation Toolkit

[editorialbot]

Submitting author: !--author-handle-->@devrimcavusoglu<!--end-author-handle-- (Devrim Çavuşoğlu) Repository: https://github.com/obss/jury Branch with paper.md (empty if default branch): joss-paper Version: v2.3 Editor: !--editor-->@crvernon<!--end-editor-- Reviewers: @evamaxfield, @KennethEnevoldsen Managing EiC: Arfon Smith

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/a41a840a782ead74b46ac05699ce6eed"><img src="https://joss.theoj.org/papers/a41a840a782ead74b46ac05699ce6eed/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/a41a840a782ead74b46ac05699ce6eed/status.svg)](https://joss.theoj.org/papers/a41a840a782ead74b46ac05699ce6eed)

Author instructions

Thanks for submitting your paper to JOSS @devrimcavusoglu. Currently, there isn't a JOSS editor assigned to your paper.

@devrimcavusoglu if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.14 s (992.0 files/s, 83781.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                         109           1402           2189           5368
JSON                            22              0              0           1713
Markdown                         4            101              0            279
TeX                              1              7              0            140
Jupyter Notebook                 1              0            399            111
YAML                             2             15              1             81
TOML                             1              1              0             17
-------------------------------------------------------------------------------
SUM:                           140           1526           2589           7709
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 771

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.18653/v1/W18-5446 is OK
- 10.18653/v1/N19-1423 is OK
- 10.18653/v1/2021.emnlp-demo.21 is OK
- 10.3115/1073083.1073135 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[arfon]

@devrimcavusoglu – thanks for your submission to JOSS. In your submission you said:

A detailed paper describing the framework and the library submitted to three ACL venues (e.g. system demonstration tracks of ACL'23, EMNLP'23, EACL'24) earlier.

Could you please share a link to these papers (or upload them here).

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[editorialbot]

Five most similar historical JOSS papers:

TRUNAJOD: A text complexity library to enhance natural language processing Submitting author: @dpalmasan Handling editor: @danielskatz (Active) Reviewers: @mbdemoraes, @apiad Similarity score: 0.8115

JASP for Audit: Bayesian Tools for the Auditing Practice Submitting author: @koenderks Handling editor: @drvinceknight (Active) Reviewers: @trstewart212, @fgeertman Similarity score: 0.8049

Multi-attribute task builder Submitting author: @Yury-Shevchenko Handling editor: @alexhanna (Retired) Reviewers: @u01ai11 Similarity score: 0.8046

gobbli: A uniform interface to deep learning for text in Python Submitting author: @jasonnance Handling editor: @arfon (Active) Reviewers: @w4ngatang, @ljvmiranda921, @sisco0 Similarity score: 0.7997

RSMTool: collection of tools building and evaluating automated scoring models Submitting author: @desilinguist Handling editor: @arfon (Active) Reviewers: @jkahn Similarity score: 0.7997

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[devrimcavusoglu]

@arfon Of course, I thought I also provided the paper in the note. Link to the paper.

[arfon]

@arfon Of course, I thought I also provided the paper in the note. Link to the paper.

Ah, sorry. This is the paper that has been submitted to those venues? If so, could you tell me more about what ACL'23, EMNLP'23, EACL'24 are? Are those submissions reviewed and published conference papers?

[devrimcavusoglu]

@arfon Yes, it is. All the submission/review phases are completed, the paper has not been published on those venues, and there will be no further other submissions.

[arfon]

@editorialbot query scope

Thanks for the additional background @devrimcavusoglu. My concern here is that we might (as a journal) consider this a repeat publication for the same work which we don't allow. I'm going to ask the wider editorial team for their input here.

[editorialbot]

Submission flagged for editorial review.

[devrimcavusoglu]

@arfon Thanks for the response, but arxiv is only an online preprint and it's not considered as a publication as in many venues (including journals, conferences, etc.) publishing the paper as an online preprint does not invalidate submissions to the venues. Having said that I don't throughly know your stance on this, so we'll be waiting for your response.

[danielskatz]

@devrimcavusoglu - I'm another track editor looking at this. Can you help me understand how I should interpret

A detailed paper describing the framework and the library submitted to three ACL venues (e.g. system demonstration tracks of ACL'23, EMNLP'23, EACL'24) earlier.

and

All the submission/review phases are completed, the paper has not been published on those venues, and there will be no further other submissions.

I don't think I understand the model of these conferences, if you submit work to them and they then don't publish it, unless you mean that the submissions to these conferences were not accepted?

[devrimcavusoglu]

@devrimcavusoglu - I'm another track editor looking at this. Can you help me understand how I should interpret

A detailed paper describing the framework and the library submitted to three ACL venues (e.g. system demonstration tracks of ACL'23, EMNLP'23, EACL'24) earlier.

and

All the submission/review phases are completed, the paper has not been published on those venues, and there will be no further other submissions.

I don't think I understand the model of these conferences, if you submit work to them and they then don't publish it, unless you mean that the submissions to these conferences were not accepted?

@danielskatz Of course, I mean I thought that was obvious enough and didn't mention it explicitly. The paper was refactored w.r.t. all comments and published on arxiv by us, marking the current version of the paper.

[danielskatz]

Thanks! This wasn't obvious to me. It was unclear what "submitted" meant, given that the work could still be under consideration. This also clarifies the lack of any publication duplication concern, in my opinion.

[arfon]

Thanks for the feedback @devrimcavusoglu. I've removed the scope review label, but now we're in a 'waitlisted' state. We're currently managing a large backlog of submissions and the editor most appropriate for your area is already rather busy.

For now, we will need to waitlist this paper and process it as the queue reduces. Thanks for your patience!

[arfon]

Tagging @crvernon as probably the best-placed editor to take this on once he has some capacity 😅

[crvernon]

No problem @arfon , I am happy to take this on. I'll leave it in waitlist for a few more days and then launch.

[crvernon]

@editorialbot assign me as editor

[editorialbot]

Assigned! @crvernon is now the editor

[crvernon]

👋 @evamaxfield - Would you be willing to review this submission to JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

[crvernon]

👋 @KennethEnevoldsen - Would you be willing to review this submission to JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

[crvernon]

👋 @cmorris2945 - Would you be willing to review this submission to JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

[evamaxfield]

👋 @evamaxfield - Would you be willing to review this submission to JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

Happy to review!

[crvernon]

Fantastic, thanks @evamaxfield !

[crvernon]

@editorialbot add @evamaxfield as reviewer

[editorialbot]

@evamaxfield added to the reviewers list!

[KennethEnevoldsen]

I can do a review as well.

[crvernon]

Excellent, thanks @KennethEnevoldsen !

[crvernon]

@editorialbot add @KennethEnevoldsen as reviewer

[editorialbot]

@KennethEnevoldsen added to the reviewers list!

[crvernon]

👋 - Alright @devrimcavusoglu , @evamaxfield , and @KennethEnevoldsen - I am going to close this Pre-Review and kick off the full review which you should receive a notification for. Thanks!

[crvernon]

@editorialbot start review

[editorialbot]

OK, I've started the review over in https://github.com/openjournals/joss-reviews/issues/6452.