search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: AST Toolbox: An Adaptive Stress Testing Framework for Validation of Autonomous Systems #3312

Closed whedon closed 2 years ago

whedon commented 3 years ago

Submitting author: !--author-handle-->@mark-koren<!--end-author-handle-- (Mark Koren) Repository: https://github.com/sisl/AdaptiveStressTestingToolbox/ Branch with paper.md (empty if default branch): Version: 2020.09.01.2 Editor: !--editor-->@arfon<!--end-editor-- Reviewers: !--reviewers-list-->@abhiramm7<!--end-reviewers-list-- Archive: Pending

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/a8b4805a525236d89aa9018eca84a407"><img src="https://joss.theoj.org/papers/a8b4805a525236d89aa9018eca84a407/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/a8b4805a525236d89aa9018eca84a407/status.svg)](https://joss.theoj.org/papers/a8b4805a525236d89aa9018eca84a407)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@abhiramm7, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @sjpfenninger know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @abhiramm7

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @giodegas, @abhiramm7 it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.21 s (771.5 files/s, 63726.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          77           1720           3458           5414
reStructuredText                65            491            781            568
YAML                             6              3             11            214
INI                              2             20              0            208
TeX                              3             23              7            204
Markdown                         1             17              0            117
DOS Batch                        3             15              2             50
Bourne Shell                     4              0              6             33
make                             1              4              6             10
-------------------------------------------------------------------------------
SUM:                           162           2293           4271           6818
-------------------------------------------------------------------------------

Statistical information for the repository '1177ce6c55bee72aa5221aa3' was
gathered on 2021/05/25.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Anthony Corso                    2             7              5            0.00
Mark Koren                      50         42252          11982            7.26
Xiaobai Ma                      24          2013           7253            1.24
mark-koren                      82        226776         312653           72.20
maxiaoba                       146        104140          30563           18.03
mkoren                          69          3500           5968            1.27

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Mark Koren                 2634            6.2         17.3               19.17
Xiaobai Ma                 1765           87.7          8.7               14.90
mark-koren                 4101            1.8         10.1                9.80
maxiaoba                   1667            1.6         19.1               10.50
mkoren                      425           12.1          6.2               33.88
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.1109/dasc.2015.7311613 may be a valid DOI for title: Adaptive stress testing of airborne collision avoidance systems
- 10.1613/jair.1.12190 may be a valid DOI for title: Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning
- 10.1201/9780429021121-19 may be a valid DOI for title: Decision Making Under Uncertainty
- 10.1109/itsc45102.2020.9294729 may be a valid DOI for title: Adaptive Stress Testing without Domain Heuristics using Go-Explore
- 10.1109/ivs.2018.8500400 may be a valid DOI for title: Adaptive Stress Testing for Autonomous Vehicles
- 10.1109/itsc.2019.8917242 may be a valid DOI for title: Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation
- 10.1109/itsc45102.2020.9294490 may be a valid DOI for title: Interpretable Safety Validation for Autonomous Vehicles
- 10.1109/itsc.2019.8917403 may be a valid DOI for title: Efficient autonomy validation in simulation with adaptive stress testing
- 10.1109/dasc50938.2020.9256730 may be a valid DOI for title: Adaptive Stress Testing of Trajectory Predictions in Flight Management Systems
- 10.1109/itsc45102.2020.9294549 may be a valid DOI for title: Validation of Image-Based Neural Network Controllers through Adaptive Stress Testing

INVALID DOIs

- None
sjpfenninger commented 3 years ago

@giodegas, @abhiramm7 - please carry out your review in this issue by updating the checklist above and giving feedback in this issue. The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

sjpfenninger commented 3 years ago

@mark-koren Can you ensure that the reference list includes DOIs or URLs wherever possible?

mark-koren commented 3 years ago

Sure thing, just updated the references to include the dois.

giodegas commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

giodegas commented 3 years ago

@whedon check references

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1109/DASC.2015.7311613 is OK
- 10.1613/jair.1.12190 is OK
- 10.7551/mitpress/10187.001.0001 is OK
- 10.1109/ITSC45102.2020.9294729 is OK
- 10.1109/IVS.2018.8500400 is OK
- 10.1109/ITSC.2019.8917242 is OK
- 10.1109/ITSC45102.2020.9294490 is OK
- 10.1109/ITSC.2019.8917403 is OK
- 10.1109/DASC50938.2020.9256730 is OK
- 10.1109/itsc45102.2020.9294549 is OK

MISSING DOIs

- None

INVALID DOIs

- None
giodegas commented 3 years ago

@mark-koren I tried to installi it, but got errors:

pip3 install ast-toolbox

ends with:

  Could not find a version that satisfies the requirement tensorflow<1.16,>=1.15.0 (from garage==2019.10.1->ast-toolbox) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1)
No matching distribution found for tensorflow<1.16,>=1.15.0 (from garage==2019.10.1->ast-toolbox)
pip3 install https://github.com/sisl/AdaptiveStressTestingToolbox.git

ends with:

Cannot unpack file /tmp/pip-unpack-79upke_s/AdaptiveStressTestingToolbox.git (downloaded from /tmp/pip-req-build-9pkfxc7f, content-type: text/html; charset=utf-8); cannot detect archive format Cannot determine archive format of /tmp/pip-req-build-9pkfxc7f

My Python3:

Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
mark-koren commented 3 years ago

Hmmm strange, lemme look into that, I may have broken something without noticing when updating the joss stuff.

mark-koren commented 3 years ago

I might not be able to get to it today but I will ping you when I have it fixed.

mark-koren commented 3 years ago

@giodegas Try using python 3.6*. If that is an issue, let me know -- I was able to successfully install the toolbox using the latest version of 3.7 (3.7.10), though using 3.7.3 I also get the error you did. I have updated the toolbox for now to enforce using 3.6.

*This is an inherited dependency from garage, but they have since released an update that will allow us to support higher versions of python. The next planned update for this toolbox will be to use the newer version of garage and support more python versions, but it is a lot of work and will be sometime this summer.

giodegas commented 3 years ago

@mark-koren in the paper this dependency should be noted, also in the installation docs. Eventually the best setup to have all software packages "frozen" at the time of the paper, to have a docker container description (Dockerfile) in which you can specify fixed version releases for each dependency, including pyhton. It would also facilitate the review actiity.

mark-koren commented 3 years ago

@mark-koren in the paper this dependency should be noted, also in the installation docs. Eventually the best setup to have all software packages "frozen" at the time of the paper, to have a docker container description (Dockerfile) in which you can specify fixed version releases for each dependency, including pyhton. It would also facilitate the review actiity.

Sorry, to clarify -- Is this something you need before you can continue the review? Or is this a longer-term add?

giodegas commented 3 years ago

Yes, please, I consider it to be my revision request. We are not just reviewing the paper content, but the accompaining software as well.

mark-koren commented 3 years ago

Sure thing, I'll ping you when I have that working.

whedon commented 3 years ago

:wave: @giodegas, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @abhiramm7, please update us on how your review is going (this is an automated reminder).

giodegas commented 3 years ago

@whedon I am waiting from the authors to setup a better software distribution as described in my previous comment.

whedon commented 3 years ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands
sjpfenninger commented 3 years ago

@mark-koren Can you update on whether you have been able to resolve the package version issues that prevented installation so that @giodegas can continue reviewing? I would be happy with a solution that specifies all dependencies with their versions, it does not have to be full blown Docker solution. Meanwhile I am also chasing up @abhiramm7 who is MIA.

abhiramm7 commented 3 years ago

@sjpfenninger sorry this feel though the cracks. I can review this by Monday. I apologize for the delay.

abhiramm7 commented 3 years ago

@mark-koren I am running into an issue with the version of torch when I use pip install ast-toolbox. 

ERROR: Could not find a version that satisfies the requirement torch==1.3.0 (from garage) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 1.7.1, 1.8.0, 1.8.1, 1.9.0)
ERROR: No matching distribution found for torch==1.3.0

I am using:

Python 3.9.5 (default, May  4 2021, 03:36:27)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin

This seems to coming from garage. I would recommend that you note these dependencies in the installation instructions.

mark-koren commented 3 years ago

Hey y'all, sorry for the long delay! I graduated and have been traveling since, but I am back now.

mark-koren commented 3 years ago

@mark-koren Can you update on whether you have been able to resolve the package version issues that prevented installation so that @giodegas can continue reviewing? I would be happy with a solution that specifies all dependencies with their versions, it does not have to be full blown Docker solution. Meanwhile I am also chasing up @abhiramm7 who is MIA.

I had a docker solution 90% done before I left, but I am digging deeper to find out best practices (e.g. stuff like should I still use venv too, or should docker user be root, etc). I know this level of detail probably isn't needed for the review itself, but I want to make sure the docker option is available to all our users, since this dependency issue seems to be cropping up more.

giodegas commented 3 years ago

This is great news! Looking forward to it.

mark-koren commented 3 years ago

Ok, there is now a docker install available! The instructions are on the README of the github page. Let me know if you run into any issues!

sjpfenninger commented 3 years ago

@abhiramm7 @giodegas Now that there is a docker install available, could you give us an update on your reviews?

abhiramm7 commented 3 years ago

I have reviewed most of the submission, I just need to go over the installation and use of the docker install. I'll try to wrap this today or latest by tomorrow. Thanks for the reminder.

abhiramm7 commented 3 years ago

@mark-koren I've opened an issue on repo about the installation issue. There seems an issue with torch

sjpfenninger commented 3 years ago

@giodegas I realise it's the start of the teaching period, but a quick update is appreciated!

arfon commented 2 years ago

@whedon assign me as editor

editorialbot commented 2 years ago

My name is now @editorialbot

arfon commented 2 years ago

@editorialbot assign me as editor

editorialbot commented 2 years ago

Assigned! @arfon is now the editor

arfon commented 2 years ago

:wave: all, I'm taking over this review as @sjpfenninger is stepping down from the JOSS editorial team. Thanks for all of your efforts @sjpfenninger!

@giodegas – my understanding was that you were waiting to start your review in earnest until there was a Docker solution in place which it looks like there now is (https://github.com/openjournals/joss-reviews/issues/3312#issuecomment-892157745).

mossr commented 2 years ago

Hey @mark-koren, it may make sense to cite my related JOSS paper (https://joss.theoj.org/papers/10.21105/joss.02749).

@article{moss2021pomdpstresstesting,
  title = {{POMDPStressTesting.jl}: Adaptive Stress Testing for Black-Box Systems},
  author = {Robert J. Moss},
  journal = {Journal of Open Source Software},
  year = {2021},
  volume = {6},
  number = {60},
  pages = {2749},
  doi = {10.21105/joss.02749}
}
arfon commented 2 years ago

Quick update: I just contacted @giodegas again for the final time over email to see if they were able to complete their review. If we don't hear back soon, I'll look for a different reviewer.

arfon commented 2 years ago

:wave: @mark-koren – I just heard back from @giodegas that they won't be able to complete their review. I will provide second review myself. Before I do so however I would like you to confirm that you're still interested in pursing this publication with JOSS?

I ask as it looks like you didn't respond to https://github.com/sisl/AdaptiveStressTestingToolbox/issues/53 or https://github.com/sisl/AdaptiveStressTestingToolbox/issues/52 from @abhiramm7's review.

arfon commented 2 years ago

@editorialbot remove @giodegas as reviewer

editorialbot commented 2 years ago

@giodegas removed from the reviewers list!

arfon commented 2 years ago

@mark-koren – it's a couple of months since we've heard anything from you on this submission. If we don't hear back from you in the next couple of weeks we will assume you're no-longer interested in publishing this paper and will proceed to reject.

mark-koren commented 2 years ago

Hi! Sorry, I completely lost track of this. There was supposed to be a student or two who were picking up this project, but it doesn't seem like that has happened. I'd say go ahead and reject -- if the transition ends up happening, we can always resubmit. Thanks for your consideration.

On Sat, Oct 1, 2022 at 4:55 AM Arfon Smith @.***> wrote:

@mark-koren https://github.com/mark-koren – it's a couple of months since we've heard anything from you on this submission. If we don't hear back from you in the next couple of weeks we will assume you're no-longer interested in publishing this paper and will proceed to reject.

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/3312#issuecomment-1264340709, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADC2KZN7WRJJFUNRHJVFMDDWBARCRANCNFSM45O6CNMQ . You are receiving this because you were mentioned.Message ID: @.***>

arfon commented 2 years ago

OK thanks for letting us know @mark-koren .

@abhiramm7 – thank you for all of your efforts here. I'm sorry this submission didn't work out.

arfon commented 2 years ago

@editorialbot reject

editorialbot commented 2 years ago

Paper rejected.