[REVIEW]: EvoMaster: A Search-Based System Test Generation Tool

whedon commented 4 years ago

Submitting author: @arcuri82 (Andrea Arcuri) Repository: https://github.com/EMResearch/EvoMaster Version: 1.1.0 Editor: @gkthiruvathukal Reviewers: @mado89, @s0nata, @UTH-Tuan Archive: 10.5281/zenodo.4300745

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c58344ed30444a648b60b5e534fa1fed"><img src="https://joss.theoj.org/papers/c58344ed30444a648b60b5e534fa1fed/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c58344ed30444a648b60b5e534fa1fed/status.svg)](https://joss.theoj.org/papers/c58344ed30444a648b60b5e534fa1fed)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@mado89, @UTH-Tuan, @s0nata, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @gkthiruvathukal know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @mado89

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@arcuri82) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @UTH-Tuan

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@arcuri82) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @s0nata

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@arcuri82) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

whedon commented 4 years ago

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the sole reviewer of this submission
@whedon assign @username as reviewer

# Add a GitHub user to the reviewers of this submission
@whedon add @username as reviewer

# Re-invite a reviewer (if they can't update checklists)
@whedon re-invite @username as reviewer

# Remove a GitHub user from the reviewers of this submission
@whedon remove @username as reviewer

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Set the software archive DOI at the top of the issue e.g.
@whedon set 10.0000/zenodo.00000 as archive

# Set the software version at the top of the issue e.g.
@whedon set v1.0.1 as version

# Open the review issue
@whedon start review

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Remind an author or reviewer to return to a review after a
# certain period of time (supported units days and weeks)
@whedon remind @reviewer in 2 weeks

# Ask Whedon to do a  dry run of accepting the paper and depositing with Crossref
@whedon accept

# Ask Whedon to check the references for missing DOIs
@whedon check references

# Ask Whedon to check repository statistics for the submitted software
@whedon check repository

EiC TASKS

# Invite an editor to edit a submission (sending them an email)
@whedon invite @editor as editor

# Reject a paper
@whedon reject

# Withdraw a paper
@whedon withdraw

# Ask Whedon to actually accept the paper and deposit with Crossref
@whedon accept deposit=true

ProfTuan commented 4 years ago

@UTH-Tuan and @shanamatthews, would either or both of you be willing to contribute a review for this JOSS submission?

@gkthiruvathukal I guess I can step in light of the situation. I'll admit that this is a new way of doing reviews for manuscripts so I may need some direction and guidance.

gkthiruvathukal commented 4 years ago

@UTH-Tuan I will be able to answer any questions about the review process. Just so you know, we have reviewer guidelines that take you through the process at https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. I'm going to add you as a reviewer now!

gkthiruvathukal commented 4 years ago

@whedon add @UTH-Tuan as reviewer

whedon commented 4 years ago

OK, @UTH-Tuan is now a reviewer

mado89 commented 4 years ago

@arcuri82 So sorry about this. I am still hopeful we will be able to see this through.

@mado89, @s0nata, and @feliperodri, can you provide any update on when you will be able to complete your review? For some reason I thought we had reviews from at least one of you (@mado89) but we have none. I really don't want to keep @arcuri82 waiting much longer.

I am sorry for not posting my review more explicitily: In my opinion this tool needs some more work. Although investing quite some time into this review. I am still unable to perform a thorough review of this tool for the following reasons:

Lack of gradle support in documentation (I am aware that most people use Maven. Maybe my issues are related to some other aspects)
Lack of user documentation: For "white box" testing there is a document explaining a lot of technical details. However, in my opinion it lacks a "Step-by-step" like guide helping novice users of the tool performing their first test. Hence, I was unable to perform the test (also I just received a cryptic failure message, which I reported back to the author. There the review stopped).
Lack of OAuth support: I had a brief discussion with the author about this. However, we are in a disagreement about the importance of OAuth. If mentioned somewhere more prominent that OAuth is not supported than this could be ok for me. I discovered this draw back only after investing quite some time into the review and trying to set up a test for an application.

Unfortunately, this comments are somewhat hard to add to the standard review form. When these issues are somewhat addressed I am willing to again spend some time on this review (I spent several days trying to get the tool running).

arcuri82 commented 4 years ago

I am here going to reply to the 3 points raised by @mado89 in the previous post

In the 6 months that @mado89 has been a reviewer, 4 issues (https://github.com/EMResearch/EvoMaster/issues) were open: 169, 170, 177 and 178. Unless I missed it, nothing about Gradle was ever mentioned. Adding an example of Gradle import of a library is pretty trivial, and I ll do it after writing this response.
I guess this is referring to 177, where @mado89 wrote The instructions are much clearer now. If there are parts that are still not clear, this does not seem reported in 177 (besides OAuth, discussed next). Furthermore, regarding (also I just received a cryptic failure message, which I reported back to the author. There the review stopped): I assume @mado89 is referring to 178. If that is the case, I am still waiting for my questions of 21st of May. It is impossible to debug such issue if no more info is provided.
1. On this point, I am in disagreement with @mado89. As such communication was done via email, I will post here my answer:

On this, I do disagree:

of all open-source REST API that I found (eg, then ones in https://github.com/EMResearch/EMB), none use OAuth

none of the industrial partners I collaborate with use OAuth (and they have SpringBoot REST APIs)

on API directories (eg https://www.programmableweb.com/ and https://www.programmableweb.com/) there are literally hundreds (if not thousands) of APIs not using OAuth. If you want to look at some numbers, check > https://www.programmableweb.com/news/spotting-api-security-trends-programmablewebs-api-directory/research/2018/01/02

no other test generation tool support OAuth

We are going to support OAuth in the (hopefully not far) future, but it has not be a priority so far, as it is not so popular yet. There are other more important features to add first

arcuri82 commented 4 years ago

Gradle example has now been added to the documentation.

arcuri82 commented 4 years ago

To clarify regarding OAuth: If mentioned somewhere more prominent that OAuth is not supported than this could be ok for me. As I wrote in 177, the fact that OAuth is not supported has been added to the documentation, in the section regarding Security. As I did not get any reply in 177, I thought it was OK. But I can add it now to the main readme.

EDIT: it is now added to the main readme

gkthiruvathukal commented 4 years ago

@mado89 Thanks for responding with your feedback. We rely on our reviewers to ensure the quality of all JOSS submissions.

@arcuri82 I can understand the frustration. Rest assured, I see that you are making a diligent effort to address the issues. I am grateful that you are so responsive, despite the long wait. As long as you work to address most of the reviewer issues, we will be able to bring your submission to a graceful conclusion. It seems like there won't be a problem to resolve the issues. Please note that is is not a requirement to agree with reviewers on every issue raised, provided you have good rationale for why you see differently.

To make it clear, our goal is to get to a robust submission. I'm confident that we'll get there! We've all been through a lot with COVID-19, so please continue to be patient with the process. We're all #InThisTogether!

mado89 commented 4 years ago

@arcuri82 Perfect! I will try to have a look as soon as possible. Please ping me if you don't have a reply within 10 days

ProfTuan commented 4 years ago

@arcuri82 I have taken a look at the paper to understand the scope of the problem that the software addresses. I don't see any discussion relating to what others have done and how this tool differentiates itself.

arcuri82 commented 4 years ago

@UTH-Tuan, from the paper:

Related Work In the recent years, different techniques have been proposed for black-box testing of RESTful APIs (e.g., (Atlidakis, Godefroid, & Polishchuk, 2019)(Karlsson, Causevic, & Sundmark, 2020) (Viglianisi, Dallago, & Ceccato, 2020)(Ed-douibi, Cánovas Izquierdo, & Cabot, 2018)). However, at the time of this writing, EvoMaster appears to be the only one can do both black-box and white-box testing, and that is also released as open-source

Are you saying that you want to see that (very brief) section extended?

ProfTuan commented 4 years ago

@arcuri82 yes a brief extended version for those unfamiliar with the domain.

arcuri82 commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

arcuri82 commented 4 years ago

@UTH-Tuan thanks. I have added some more info in the Related Work section. Right now the paper.md file has 974 words, where JOSS has the limit of 250-1000 words.

ProfTuan commented 4 years ago

@arcuri82 Thanks. I am also encountering issues in running the example. Either that or is it not clear how to run the example (based off what was presented in the Example doc). I get the error "Cannot communicate with remote EvoMaster Driver for the system under test at localhost:40100 Make sure the EvoMaster Driver for the system under test is running correctly". Please clarify and/or make the example more straightforward.

arcuri82 commented 4 years ago

@UTH-Tuan in the main readme of the documentation: A short video (5 minutes) shows the use of EvoMaster on one of the case studies in EMB. Link. This should help better see how things are connected, besides the description in the documentation.

Btw, could be better to create new GitHub issues in the EvoMaster repository, instead of having the review here (see JOSS documentation).

gkthiruvathukal commented 4 years ago

@arcuri82 is right. The issue tracker is the best way to report issues on a submission, especially when it's on GitHub! Many reviewers will post the link to the issue in the review as well, so it is easy to click through.

Thanks to @UTH-Tuan for helping with this review! We really appreciate it.

gkthiruvathukal commented 4 years ago

@arcuri82 Based on the open issue raised by @mado89, I assume you are still working to bring it to closure.

@mado89 and @UTH-Tuan, please respond when you are ready to move forward with this submission. I will be nudging everyone every week or so.

mado89 commented 4 years ago

Yes we are trying to resolve some issues I have with the software

s0nata commented 4 years ago

Dear everybody, thank you for your patience with me. Got significantly affected at work due to covid situation, and only recently was able to return to a schedule that premits time for the review.

For my review the only missing part is the evaluation, and @arcuri82 , by any chance is there and example project I can set up and run EvoMaster on? I saw the links in the docs for black- and white-box testing setups, they do describe some possible use cases, but I would find it extremely helpful if there was a follow-up script (as in theater play script) for "Hey, to see how our tool works on a sample project, build it, then use this small toy project to evaluate it on, you should get following results for scenarios A, B, and C". Having a docker image with a demo would be probably too resource-consuming to produce, so not asking for it.

arcuri82 commented 4 years ago

Hi @s0nata,

having a demo project would indeed be useful. Although these weeks I am quite busy with the starting of teaching semester, so might take a while before I have time to prepare it.

Anyway, for the current time, you can look at the following: from the main readme, there is a link to the EMB repository, which contains a collection of REST APIs with also drivers for white-box testing. From the main readme, there is also a link to a YouTube video showing how to run EM on one of such projects.

gkthiruvathukal commented 4 years ago

@arcuri82, Given that I, too, am working intensely to be prepared for classes this week (with lots of video recording to do, being online) I am going to nudge you & the reviewers in a week or two. As you know, I want to bring this submission to closure, especially given how long you had to wait.

@s0nata and @UTH-Tuan, please do let me know if @arcuri82's suggestion helps you to move forward with ticking the remaining boxes.

arcuri82 commented 4 years ago

In the end, it took less than I thought. I created a small example of API, available at rest-api-example. I also made a video showing how to implement anEMDriver to enable white-box testing with EvoMaster. Video is now linked from the main Readme documentation.

mado89 commented 4 years ago

The video looks good. I took some notes while watchin it and will share them with you. Maybe they could be helpful for potential users of EvoMaster. Using the video I think I will be able to do my own testing using Evo Master.

Just one thing: Right at the end of the video either the audio cuts off or you stop speaking. Either way it is a bit hard to follow what you are doing (but I think I might be able to reproduce it using your other video about blackbox testing).

kthyng commented 3 years ago

Hi all! As the rotating associate editor in chief this week, I just wanted to check in on this issue. I see unchecked review boxes above. How is your review going @mado89, @feliperodri, @s0nata, @UTH-Tuan? Thanks all for your work!!

gkthiruvathukal commented 3 years ago

@kthyng I think we're really close on this one. I also notice there is no checklist for @UTH-Tuan, who was added as a reviewer. I don't think @feliperodri is still available for this review, which is one of the reasons we added @UTH-Tuan. So I think we only need @mado89 , @s0nata , and @UTH-Tuan to indicate whether they are waiting on any further changes.

kthyng commented 3 years ago

@gkthiruvathukal ah! I see. Ok, I will remove the one reviewer and edit the checklist to be for @UTH-Tuan instead. When reviewer changes happen after starting the review issue, some of this needs to happen by hand. Please let me know if anything remains incorrect after I'm done.

kthyng commented 3 years ago

@whedon remove @feliperodri as reviewer

whedon commented 3 years ago

OK, @feliperodri is no longer a reviewer

ProfTuan commented 3 years ago

@gkthiruvathukal @kthyng how do I generate a checklist? I am sorry but this my first time doing this type of review without any direction (I am more accustomed to the traditional paper review).

kthyng commented 3 years ago

@UTH-Tuan You now have a checklist above to use. So the review is checklist-driven, as above, but as issues come up, please open issues on the software repo to describe and discuss, and link to this review issue so that we can track related issues. Let me or @gkthiruvathukal know if you have any questions. Thanks!

ProfTuan commented 3 years ago

@kthyng @gkthiruvathukal I see it now. Earlier I didn't have a checklist. I initially thought I had generate it . Thanks!

ProfTuan commented 3 years ago

@kthyng @gkthiruvathukal how do i check off the items from the checklist?

kthyng commented 3 years ago

@UTH-Tuan If you cannot, you probably need to accept your reviewer invitation. I'll reinvite you.

kthyng commented 3 years ago

@whedon re-invite @UTH-Tuan as reviewer

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@uth-tuan please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

ProfTuan commented 3 years ago

@kthyng I get the following message "Sorry, we couldn't find that repository invitation. It is possible that the invitation was revoked or that you are not logged into the invited account."

kthyng commented 3 years ago

@whedon remove @UTH-Tuan as reviewer

whedon commented 3 years ago

OK, @UTH-Tuan is no longer a reviewer

kthyng commented 3 years ago

@whedon invite @UTH-Tuan as reviewer

whedon commented 3 years ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands

kthyng commented 3 years ago

@whedon add @UTH-Tuan as reviewer

whedon commented 3 years ago

OK, @UTH-Tuan is now a reviewer

kthyng commented 3 years ago

@UTH-Tuan please try again and see if it works for you

arfon commented 3 years ago

@whedon re-invite @UTH-Tuan as reviewer

whedon commented 3 years ago

@uth-tuan already has access.

mado89 commented 3 years ago

Just wanted to ask if there is still work left from my side. I've ticked of all my boxes some time ago. While I think there is room for improvement concerning usability I think this tool can be accepted for publication in JOSS.

openjournals / joss-reviews