search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
694 stars 36 forks source link

[PRE REVIEW]: AST Toolbox: An Adaptive Stress Testing Framework for Validation of Autonomous Systems #3213

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @mark-koren (Mark Koren) Repository: https://github.com/sisl/AdaptiveStressTestingToolbox/ Version: 2020.09.01.2 Editor: @sjpfenninger Reviewers: @giodegas, @abhiramm7 Managing EiC: Arfon Smith

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Author instructions

Thanks for submitting your paper to JOSS @mark-koren. Currently, there isn't an JOSS editor assigned to your paper.

@mark-koren if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands
whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago

PDF failed to compile for issue #3213 with the following error: 

 /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/orcid_validator.rb:12:in `initialize': undefined method `strip' for nil:NilClass (NoMethodError)
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:155:in `new'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:155:in `block in check_orcids'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:153:in `each'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:153:in `check_orcids'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:90:in `initialize'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/processor.rb:38:in `new'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/processor.rb:38:in `set_paper'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/bin/whedon:58:in `prepare'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/bin/whedon:131:in `<top (required)>'
    from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in `load'
    from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in `<main>'
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.89 s (182.8 files/s, 15101.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          77           1720           3458           5414
reStructuredText                65            491            781            568
YAML                             6              3             11            214
INI                              2             20              0            208
TeX                              3             23              7            204
Markdown                         1             17              0            117
DOS Batch                        3             15              2             50
Bourne Shell                     4              0              6             33
make                             1              4              6             10
-------------------------------------------------------------------------------
SUM:                           162           2293           4271           6818
-------------------------------------------------------------------------------

Statistical information for the repository '7b26dcc626ec346ae0641421' was
gathered on 2021/04/23.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Anthony Corso                    2             7              5            0.00
Mark Koren                      50         42252          11982            7.26
Xiaobai Ma                      24          2013           7253            1.24
mark-koren                      82        226776         312653           72.20
maxiaoba                       146        104140          30563           18.03
mkoren                          66          3493           5961            1.27

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Mark Koren                 2634            6.2         17.1               19.17
Xiaobai Ma                 1765           87.7          8.5               14.90
mark-koren                 4101            1.8          9.9                9.80
maxiaoba                   1667            1.6         18.9               10.50
mkoren                      425           12.2          6.0               33.88
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.1109/dasc.2015.7311613 may be a valid DOI for title: Adaptive stress testing of airborne collision avoidance systems
- 10.1613/jair.1.12190 may be a valid DOI for title: Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning
- 10.1201/9780429021121-19 may be a valid DOI for title: Decision Making Under Uncertainty
- 10.1109/itsc45102.2020.9294729 may be a valid DOI for title: Adaptive Stress Testing without Domain Heuristics using Go-Explore
- 10.1109/ivs.2018.8500400 may be a valid DOI for title: Adaptive Stress Testing for Autonomous Vehicles
- 10.1109/itsc.2019.8917242 may be a valid DOI for title: Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation
- 10.1109/itsc45102.2020.9294490 may be a valid DOI for title: Interpretable Safety Validation for Autonomous Vehicles
- 10.1109/itsc.2019.8917403 may be a valid DOI for title: Efficient autonomy validation in simulation with adaptive stress testing
- 10.1109/dasc50938.2020.9256730 may be a valid DOI for title: Adaptive Stress Testing of Trajectory Predictions in Flight Management Systems
- 10.1109/itsc45102.2020.9294549 may be a valid DOI for title: Validation of Image-Based Neural Network Controllers through Adaptive Stress Testing

INVALID DOIs

- None
Kevin-Mattheus-Moerman commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

PDF failed to compile for issue #3213 with the following error: 

 /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/orcid_validator.rb:12:in `initialize': undefined method `strip' for nil:NilClass (NoMethodError)
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:155:in `new'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:155:in `block in check_orcids'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:153:in `each'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:153:in `check_orcids'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:90:in `initialize'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/processor.rb:38:in `new'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/processor.rb:38:in `set_paper'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/bin/whedon:58:in `prepare'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/bin/whedon:131:in `<top (required)>'
    from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in `load'
    from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in `<main>'
Kevin-Mattheus-Moerman commented 3 years ago

@openjournals/dev can you help trouble shoot this paper compilation error :point_up:

Kevin-Mattheus-Moerman commented 3 years ago

@mark-koren since the error message relates to the ORCID profiles, can you please double check these and remove any blank entries for them.

mark-koren commented 3 years ago

Oh strange, the preview system was able to compile the pdf. I'll look into the ORCID profiles asap.

mark-koren commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands
mark-koren commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Kevin-Mattheus-Moerman commented 3 years ago

@whedon invite @sjpfenninger as editor

whedon commented 3 years ago

@sjpfenninger has been invited to edit this submission.

sjpfenninger commented 3 years ago

@whedon set @sjpfenninger as editor

whedon commented 3 years ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands
sjpfenninger commented 3 years ago

@whedon assign @sjpfenninger as editor

whedon commented 3 years ago

OK, the editor is @sjpfenninger

mark-koren commented 3 years ago

Hello, just wanted to make sure ya'll weren't waiting on anything from me right now.

sjpfenninger commented 3 years ago

@mark-koren Nothing needed from you. I'm contacting potential reviewers (apologies - I should have stated that here). Do let me know if you have any suggestions for reviewers though.

mark-koren commented 3 years ago

Ok cool, thanks for the update! A labmate published a related (Julia) package through Joss and the reviewers were Jarvist Moore Frost (jarvist) and Sambit Kumar Dash (sambitdash), so maybe they could be good options, if they also do python.

sjpfenninger commented 3 years ago

@whedon add @giodegas as reviewer

whedon commented 3 years ago

OK, @giodegas is now a reviewer

sjpfenninger commented 3 years ago

@whedon add @abhiramm7 as reviewer

whedon commented 3 years ago

OK, @abhiramm7 is now a reviewer

sjpfenninger commented 3 years ago

@whedon start review

whedon commented 3 years ago

OK, I've started the review over in https://github.com/openjournals/joss-reviews/issues/3312.

sjpfenninger commented 3 years ago

@giodegas, @abhiramm7, many thanks for agreeing to review. Let's head over to #3312 for the actual review!