[REVIEW]: Jury: A Comprehensive Evaluation Toolkit

editorialbot commented 8 months ago

Submitting author: !--author-handle-->@devrimcavusoglu@crvernon<!--end-editor-- Reviewers: @evamaxfield, @KennethEnevoldsen Archive: 10.5281/zenodo.11170894

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/a41a840a782ead74b46ac05699ce6eed"><img src="https://joss.theoj.org/papers/a41a840a782ead74b46ac05699ce6eed/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/a41a840a782ead74b46ac05699ce6eed/status.svg)](https://joss.theoj.org/papers/a41a840a782ead74b46ac05699ce6eed)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@evamaxfield & @KennethEnevoldsen, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @crvernon know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @evamaxfield

📝 Checklist for @KennethEnevoldsen

devrimcavusoglu commented 5 months ago

Hi @crvernon A new release is on, and associated zenodo version is up.

crvernon commented 5 months ago

@editorialbot generate pdf

editorialbot commented 5 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

crvernon commented 5 months ago

@editorialbot set v2.3.1 as version

editorialbot commented 5 months ago

Done! version is now v2.3.1

crvernon commented 5 months ago

@devrimcavusoglu no need to do another release, but please edit the authors in the Zenodo record metadata to match the authors and order of those who appear in your paper.

Also...please ensure that the title of the Zenodo record matches exactly with what is in the paper.

Please let me know when this is done so we can move forward.

devrimcavusoglu commented 5 months ago

@crvernon I updated the zenodo version, as you suggested under the same version, I updated the metadata.

crvernon commented 5 months ago

@devrimcavusoglu the titles still do not match.

[x] Please fix the title What is in your paper: Jury: A Comprehensive Evaluation Toolkit The title of your archive: Jury: A Comprehensive NLP Evaluation toolkit
- [x] Also, the author order is still incorrect in your archive as compared to what it is in the paper.

Please fix both issue to match what is in the paper.

devrimcavusoglu commented 5 months ago

@crvernon sorry for the inconvenience, I updated the zenodo version with the proper name and corrected author order.

crvernon commented 5 months ago

@editorialbot check references

editorialbot commented 5 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.18653/v1/W18-5446 is OK
- 10.18653/v1/N19-1423 is OK
- 10.18653/v1/2021.emnlp-demo.21 is OK
- 10.3115/1073083.1073135 is OK

MISSING DOIs

- No DOI given, and none found for title: Relevance of Unsupervised Metrics in Task-Oriented...
- No DOI given, and none found for title: Findings of the 2020 Conference on Machine Transla...
- No DOI given, and none found for title: XLNet: Generalized Autoregressive Pretraining for ...

INVALID DOIs

- None

crvernon commented 5 months ago

@editorialbot set 10.5281/zenodo.11170894 as archive

editorialbot commented 5 months ago

Done! archive is now 10.5281/zenodo.11170894

crvernon commented 5 months ago

@editorialbot recommend-accept

editorialbot commented 5 months ago

Attempting dry run of processing paper acceptance...

editorialbot commented 5 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.18653/v1/W18-5446 is OK
- 10.18653/v1/N19-1423 is OK
- 10.18653/v1/2021.emnlp-demo.21 is OK
- 10.3115/1073083.1073135 is OK

MISSING DOIs

- No DOI given, and none found for title: Relevance of Unsupervised Metrics in Task-Oriented...
- No DOI given, and none found for title: Findings of the 2020 Conference on Machine Transla...
- No DOI given, and none found for title: XLNet: Generalized Autoregressive Pretraining for ...

INVALID DOIs

- None

editorialbot commented 5 months ago

:wave: @openjournals/dsais-eics, this paper is ready to be accepted and published.

Check final proof :point_right::page_facing_up: Download article

If the paper PDF and the deposit XML files look good in https://github.com/openjournals/joss-papers/pull/5360, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

crvernon commented 5 months ago

Alright @devrimcavusoglu, take a final look at the proof above and make sure all is well from your side and let me know when you are satisfied with it. Then I'll accept this one for publication in full.

devrimcavusoglu commented 5 months ago

@crvernon I just checked the proof, LGTM 👍 I didn't want to provide a link to the arxiv, but I don't know we can do such thing. If joss allow, I can cite/provide our arxiv paper as well. If not, we can proceed with this latest proof.

crvernon commented 5 months ago

@devrimcavusoglu - We have no problems with the preprint. See https://joss.readthedocs.io/en/latest/submitting.html#preprint-policy

Just let me know if you alter the paper and I'll recheck. If not, just let me know and we can move forward. Thanks!

devrimcavusoglu commented 5 months ago

@crvernon I rechecked it and I think there's no need for any revision. We can proceed with this version.

crvernon commented 5 months ago

@editorialbot accept

editorialbot commented 5 months ago

Doing it live! Attempting automated processing of paper acceptance...

editorialbot commented 5 months ago

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

``` cff-version: "1.2.0" authors: - family-names: Cavusoglu given-names: Devrim - family-names: Sen given-names: Secil - family-names: Sert given-names: Ulas - family-names: Altinuc given-names: Sinan contact: - family-names: Cavusoglu given-names: Devrim doi: 10.5281/zenodo.11170894 message: If you use this software, please cite our article in the Journal of Open Source Software. preferred-citation: authors: - family-names: Cavusoglu given-names: Devrim - family-names: Sen given-names: Secil - family-names: Sert given-names: Ulas - family-names: Altinuc given-names: Sinan date-published: 2024-05-20 doi: 10.21105/joss.06452 issn: 2475-9066 issue: 97 journal: Journal of Open Source Software publisher: name: Open Journals start: 6452 title: "Jury: A Comprehensive Evaluation Toolkit" type: article url: "https://joss.theoj.org/papers/10.21105/joss.06452" volume: 9 title: "Jury: A Comprehensive Evaluation Toolkit" ```

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

editorialbot commented 5 months ago

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

editorialbot commented 5 months ago

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/5363
Wait five minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.06452
If everything looks good, then close this review issue.
Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

crvernon commented 5 months ago

🥳 Congratulations on your new publication @devrimcavusoglu! Many thanks to @evamaxfield and @KennethEnevoldsen for your time, hard work, and expertise in reviewing!! JOSS wouldn't be able to function nor succeed without your efforts.

Please consider becoming a reviewer for JOSS if you are not already: https://reviewers.joss.theoj.org/join

editorialbot commented 5 months ago

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.06452/status.svg)](https://doi.org/10.21105/joss.06452)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.06452">
  <img src="https://joss.theoj.org/papers/10.21105/joss.06452/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.06452/status.svg
   :target: https://doi.org/10.21105/joss.06452

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

openjournals / joss-reviews