search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
721 stars 38 forks source link

[REVIEW]: EvoMaster: A Search-Based System Test Generation Tool #2153

Closed whedon closed 3 years ago

whedon commented 4 years ago

Submitting author: @arcuri82 (Andrea Arcuri) Repository: https://github.com/EMResearch/EvoMaster Version: 1.1.0 Editor: @gkthiruvathukal Reviewers: @mado89, @s0nata, @UTH-Tuan Archive: 10.5281/zenodo.4300745

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c58344ed30444a648b60b5e534fa1fed"><img src="https://joss.theoj.org/papers/c58344ed30444a648b60b5e534fa1fed/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c58344ed30444a648b60b5e534fa1fed/status.svg)](https://joss.theoj.org/papers/c58344ed30444a648b60b5e534fa1fed)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@mado89, @UTH-Tuan, @s0nata, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @gkthiruvathukal know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @mado89

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @UTH-Tuan

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @s0nata

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 4 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @mado89, @jaynanavati it looks like you're currently assigned to review this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 4 years ago 
Reference check summary:

OK DOIs

- None

MISSING DOIs

- https://doi.org/10.1109/icst.2018.00046 may be missing for title: EvoMaster: Evolutionary Multi-context Automated System Test Generation
- https://doi.org/10.1007/978-3-319-66299-2_1 may be missing for title: Many Independent Objective (MIO) Algorithm for Test Suite Generation
- https://doi.org/10.1109/qrs.2017.11 may be missing for title: RESTful API Automated Test Case Generation
- https://doi.org/10.1145/3321707.3321815 may be missing for title: Resource-based test case generation for RESTful web services
- https://doi.org/10.1145/3321707.3321732 may be missing for title: SQL data generation to enhance Search-Based System Testing

INVALID DOIs

- None
whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

arfon commented 4 years ago

Dear authors and reviewers

We wanted to notify you that in light of the current COVID-19 pandemic, JOSS has decided to suspend submission of new manuscripts and to handle existing manuscripts (such as this one) on a "best efforts basis". We understand that you may need to attend to more pressing issues than completing a review or updating a repository in response to a review. If this is the case, a quick note indicating that you need to put a "pause" on your involvement with a review would be appreciated but is not required.

Thanks in advance for your understanding.

Arfon Smith, Editor in Chief, on behalf of the JOSS editorial team.

mado89 commented 4 years ago

@arfon Sorry to hear about the closing down. Hopefully, once my submission is ready everything is back to normal @arcuri82 just wanted to let you know I haven't forgotten about this. I am a bit behind schedule but I hope to start on the weekend or next week. Sorry for the delay

arfon commented 4 years ago

:wave: @mado89, @jaynanavati - just a friendly check-in to see how things are going with your reviews?

mado89 commented 4 years ago

I see that @arcuri82 is working on this repository. Which version should be reviewed? 1.0.0 or the snapshot for 1.0.2?

arfon commented 4 years ago

@arcuri82 - could you advise if there are any major changes between v1.0.0 and v1.0.2?

arcuri82 commented 4 years ago

the project is an active development (as it has been for the last 3.5 years). The latest release is v1.0.1 (see release page), and it was made 2 weeks ago. It is just bug fixes over v1.0.0.

The version 1.0.2-SNAPSHOT is simply the working version in the master branch.

Which one to review? Both would be equivalent, as all new features under development are deactivated by default. It all depends if you want to download an already packaged jar file from the release page, or build the jar directly

arcuri82 commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

arcuri82 commented 4 years ago

thanks @mado89 for your feedback. i made changes and updated the paper. Let me know if you require further changes

Question to @arfon: is @jaynanavati active? I was looking at his/her GitHub profile today, and it does not seem there is any activity whatsoever on GitHub: https://github.com/jaynanavati?tab=overview&from=2020-04-01&to=2020-04-24 Or did I miss something? (I apologize in that case)

arfon commented 4 years ago

Question to @arfon: is @jaynanavati active? I was looking at his/her GitHub profile today, and it does not seem there is any activity whatsoever on GitHub: https://github.com/jaynanavati?tab=overview&from=2020-04-01&to=2020-04-24

Not clear. I'd like us to wait longer before considering changing out reviewers, especially as so many people's lives are severely disrupted by COVID-19.

arcuri82 commented 4 years ago

COVID-19 is indeed a problem :( However, what is peculiar here is that @jaynanavati account was created in 2018, and there is absolutely no activity of any kind since then (unless all activities are in private repositories, and then those are not shown). Anyway, I am not in a hurry :)

arcuri82 commented 4 years ago

I would like to apologize: I forgot (or did not pay proper attention) of the "[PRE REVIEW]" #2001 , where @jaynanavati explicitly stated to be willing to be a reviewer. Somehow, I was worried that a wrong user was added by mistake to this review, and just wanted to double-check :)

jaynanavati commented 4 years ago

Dear Editor,

Kindly replace me with someone else as I don't have spare time to review the work nowadays.

Thank you.

Regards,

Dr. Jay Nanavati

On Mon, 13 Apr, 2020, 9:22 PM Arfon Smith, notifications@github.com wrote:

👋 @mado89 https://github.com/mado89, @jaynanavati https://github.com/jaynanavati - just a friendly check-in to see how things are going with your reviews?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2153#issuecomment-612958540, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJUTOL3DLZ5SIF3MKJ3UO4LRMMYL3ANCNFSM4LEJ45HQ .

arfon commented 4 years ago

@arcuri82 - looks like we're going to need to find another reviewer - could you take a look a this list of potential reviewers and identify a few people who would be good candidates to review this submission?

arcuri82 commented 4 years ago

@arfon - considering the ones that have been already contacted in #2001, then some other possibilities could be:

hainesr commented 4 years ago

Hi all,

I am interested in this work, but I am terribly sorry I can't see that I could review it in a timely manner at the moment. Home schooling is currently a large drain on my resources!

Rob

mado89 commented 4 years ago

thanks @mado89 for your feedback. i made changes and updated the paper. Let me know if you require further changes

I will take a look at it. Sorry once again for being so slow on this review. Hopefully, the second reviewer - once found - can do their job faster than me so that you won't be delayed for much longer

arfon commented 4 years ago

@whedon remove @jaynanavati as reviewer

whedon commented 4 years ago

OK, @jaynanavati is no longer a reviewer

arfon commented 4 years ago

👋 @feliperodri & @s0nata - we're looking for a second reviewer of the EvoMaster package for JOSS.

Would either of you be willing to review this submission for us? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

feliperodri commented 4 years ago

Hi @arfon, thanks for the invitation. I can take some time this week to review it and provide a feedback by Friday or Saturday. Would that be a reasonable time frame for you?

arfon commented 4 years ago

Hi @arfon, thanks for the invitation. I can take some time this week to review it and provide a feedback by Friday or Saturday. Would that be a reasonable time frame for you?

That would be fantastic, thanks! I'll go ahead and set up your review here now.

arfon commented 4 years ago

@whedon add @feliperodri as reviewer

whedon commented 4 years ago

OK, @feliperodri is now a reviewer

arfon commented 4 years ago

@feliperodri - please carry out your review in this issue by updating the checklist above and giving feedback in this issue. The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

Any questions/concerns please let me know.

s0nata commented 4 years ago

hello @arfon, thanks for the invitation, I should have some time over the weekend.

arfon commented 4 years ago

hello @arfon, thanks for the invitation, I should have some time over the weekend.

OK great! I'll set up your review now...

arfon commented 4 years ago

@whedon add @s0nata as reviewer

whedon commented 4 years ago

OK, @s0nata is now a reviewer

arfon commented 4 years ago

@s0nata - please carry out your review in this issue by updating the checklist above and giving feedback in this issue. The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

Any questions/concerns please let me know.

arfon commented 4 years ago

@gkthiruvathukal - feel free to take this over when you are ready.

s0nata commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

s0nata commented 4 years ago

Hi @arcuri82, I have just added my comments on the paper text to the issue #170

arcuri82 commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

arcuri82 commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

gkthiruvathukal commented 4 years ago

@arfon Just letting you know, I'm able to take over and see this through.

@mado89, @feliperodri, @s0nata, How are things coming along with the review? Let me know if you have any questions.

s0nata commented 4 years ago

@gkthiruvathukal almost ready on my end, just need to fix an issue with my OS that blocks tool evaluation, and I'm done

gkthiruvathukal commented 4 years ago

Thanks, @s0nata! Please signal me when ready!

arcuri82 commented 4 years ago

Sadly, today marks 6 months since submission (7th January)... :(

gkthiruvathukal commented 4 years ago

@arcuri82 So sorry about this. I am still hopeful we will be able to see this through.

@mado89, @s0nata, and @feliperodri, can you provide any update on when you will be able to complete your review? For some reason I thought we had reviews from at least one of you (@mado89) but we have none. I really don't want to keep @arcuri82 waiting much longer.

arcuri82 commented 4 years ago

@gkthiruvathukal thanks

Apart from the message of 29th April, it seems there has been no other communication from @feliperodri

@mado89 has been a reviewer since January, but unfortunately has been postponing, month after month

I guess that, besides @s0nata, there is a need for a second reviewer. From the list of the 26th of April, other possibilities could be:

gkthiruvathukal commented 4 years ago

@arcuri82, my pleasure. We will get there.

Apart from all of this, I also had to shut down for much of COVID-19 until the end of my academic year (May). So thank you for your patience.

Since I am not even getting a response from the two you mentioned, I think I'm going to proceed with adding a reviewer with the hope that @s0nata will eventually be able to complete a review for us.

gkthiruvathukal commented 4 years ago

@UTH-Tuan and @shanamatthews, would either or both of you be willing to contribute a review for this JOSS submission?

gkthiruvathukal commented 4 years ago

@whedon commands