[PRE REVIEW]: JOAN: a framework for human-automated vehicle interaction experiments in a virtual reality driving simulator

[whedon]

Submitting author: !--author-handle-->@OlgerSiebinga<!--end-author-handle-- (Olger Siebinga) Repository: https://github.com/tud-hri/joan Branch with paper.md (empty if default branch): Version: V1.1.0 Editor: !--editor-->@taless474<!--end-editor-- Reviewers: @humanfactors, @bnriiitb Managing EiC: Daniel S. Katz

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/25b2e051dac33531e497c91e99e67d68"><img src="https://joss.theoj.org/papers/25b2e051dac33531e497c91e99e67d68/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/25b2e051dac33531e497c91e99e67d68/status.svg)](https://joss.theoj.org/papers/25b2e051dac33531e497c91e99e67d68)

Author instructions

Thanks for submitting your paper to JOSS @OlgerSiebinga. Currently, there isn't an JOSS editor assigned to your paper.

@OlgerSiebinga if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Wordcount for paper.md is 1126

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.23 s (694.3 files/s, 85704.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          95           1918           2048           7105
Qt                              34              0              0           6337
Markdown                        28            669              0           1364
TeX                              1             29              0            232
JSON                             1              0              0            104
YAML                             2             15              3             50
-------------------------------------------------------------------------------
SUM:                           161           2631           2051          15192
-------------------------------------------------------------------------------

Statistical information for the repository '59277480fc3bcab6d91f26a7' was
gathered on 2022/02/14.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Andre Kraan                      1             0            209            0.14
André van der Kraan             79         11191           6387           11.61
Joris                            2           779              1            0.52
Joris Giltay                   504         28973          24377           35.24
Niek Beckers                     4            45             54            0.07
O. Siebinga                      6           997            123            0.74
Olger Siebinga                 133          5607           4080            6.40
asevenster                       2           129            112            0.16
jnpgiltay                      114          6241           2381            5.70
joan-dev                         1             3              3            0.00
niek                           278         28305          31240           39.34
niekbeckers                      3            34             28            0.04
nwmbeckers                       1             6             12            0.01
timomelman                       1            42             14            0.04

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
André van der Kraan         175            1.6         16.4               16.57
Joris Giltay               6581           22.7         15.2               10.26
Olger Siebinga             2275           40.6         13.2                6.29
asevenster                   18           14.0         11.2                0.00
niek                       2022            7.1         15.4               12.71

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1109/THMS.2018.2791570 is OK
- 10.1016/j.apergo.2020.103196 is OK
- 10.1177/0018720812443984 is OK
- 10.31234/osf.io/p8dxn is OK
- 10.4324/9781315565712 is OK
- 10.2514/6.2000-4503 is OK
- 10.1016/j.aap.2016.10.016 is OK
- 10.1155/2021/4396401 is OK

MISSING DOIs

- 10.1007/978-3-319-67361-5_40 may be a valid DOI for title: AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles
- 10.1016/j.trc.2021.103008 may be a valid DOI for title: A Survey on Autonomous Vehicle Control in the Era of Mixed-Autonomy: From Physics-Based to AI-Guided Driving Policy Learning

INVALID DOIs

- None

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[danielskatz]

👋 @OlgerSiebinga - Thanks for your submission. While I look for an editor, you could work on the possibly missing DOIs that whedon suggests, but note that some may be incorrect. Please feel free to make changes to your .bib file, then use the command @whedon check references to check again, and the command @whedon generate pdf when the references are right to make a new PDF. Whedon commands need to be the first entry in a new comment.

[danielskatz]

👋 @gkthiruvathukal - Would you be willing to edit this submission?

[danielskatz]

@whedon invite @gkthiruvathukal as editor

[whedon]

@gkthiruvathukal has been invited to edit this submission.

[OlgerSiebinga]

@niekbeckers, @jorisgiltay, @avanderkraan, FYI

[OlgerSiebinga]

👋 @danielskatz - I will have a look at the DOIs.

[OlgerSiebinga]

We want to give a fair warning to all potential reviewers. Our software uses the open-source autonomous vehicle simulator CARLA, which is needed to properly test our work. Building and installing CARLA will take quite some time (4 hours according to their docs) and you'll need some reasonable hardware (a good graphics card is recommended).

[niekbeckers]

We want to give a fair warning to all potential reviewers. Our software uses the open-source autonomous vehicle simulator CARLA, which is needed to properly test our work. Building and installing CARLA will take quite some time (4 hours according to their docs) and you'll need some reasonable hardware (a good graphics card is recommended).

In addition, we only tested CARLA and our work on a machine running Windows 10; CARLA also has a Debian version, but JOAN has not been tested on Debian.

[OlgerSiebinga]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1109/THMS.2018.2791570 is OK
- 10.1007/978-3-319-67361-5_40 is OK
- 10.1016/j.apergo.2020.103196 is OK
- 10.1177/0018720812443984 is OK
- 10.31234/osf.io/p8dxn is OK
- 10.4324/9781315565712 is OK
- 10.2514/6.2000-4503 is OK
- 10.1016/j.aap.2016.10.016 is OK
- 10.1016/j.trc.2021.103008 is OK
- 10.1155/2021/4396401 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[OlgerSiebinga]

@whedon generate pdf

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[taless474]

@danielskatz I can edit this if possible

[danielskatz]

Thanks @taless474

[danielskatz]

@whedon assign @taless474 as editor

[whedon]

OK, the editor is @taless474

[taless474]

@OlgerSiebinga do you have any suggestions for who could be a reviewer from this list or in general?

[taless474]

@CameronDevine are you interested in reviewing this?

[CameronDevine]

@taless474 Unfortunately, I'm not able to at the moment. I am currently busy with other projects and this is an area where I don't really have any expertise. I'm also worried that my computer would be able to handle the simulator.

[niekbeckers]

@OlgerSiebinga do you have any suggestions for who could be a reviewer from this list or in general?

Hi @taless474, perhaps jackbrookes or humanfactors from the list could be reviewers.

[taless474]

@jackbrookes are you interested in reviewing this?

[taless474]

@humanfactors are you interested in reviewing this?

[humanfactors]

Happy to review this. I'm currently on sick leave, so I wouldn't be able to start this until at least next week. As it appears a more intensive workload in terms of setting up, I can't give a determinate date I can complete the review at this stage (though I obviously would get to it as soon as possible).

I also don't have VR, not sure if that is required for tests?

[taless474]

@humanfactors thank you. You can start in two weeks. @OlgerSiebinga I think the required hardware is only a good GPU setup not a VR. Is it right?

[OlgerSiebinga]

That's right! Our work does support experiments in VR, but everything can be used and tested without VR.

[taless474]

@editorialbot commands

[editorialbot]

Hello @taless474, here are the things you can ask me to do:

# List all available commands
@editorialbot commands

# Add to this issue's reviewers list
@editorialbot add @username as reviewer

# Remove from this issue's reviewers list
@editorialbot remove @username from reviewers

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Assign a user as the editor of this submission
@editorialbot assign @username as editor

# Remove the editor assigned to this submission
@editorialbot remove editor

# Remind an author or reviewer to return to a review after a 
# certain period of time (supported units days and weeks)
@editorialbot remind @reviewer in 2 weeks

# Check the references of the paper for missing DOIs
@editorialbot check references

# Perform checks on the repository
@editorialbot check repository

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for version
@editorialbot set v1.0.0 as version

# Set a value for archive
@editorialbot set 10.21105/zenodo.12345 as archive

# Set a value for branch
@editorialbot set joss-paper as branch

# Generates the pdf paper
@editorialbot generate pdf

# Recommends the submission for acceptance
@editorialbot recommend-accept

# Flag submission with questionable scope
@editorialbot query scope

# Get a link to the complete list of reviewers
@editorialbot list reviewers

# Open the review issue
@editorialbot start review

[editorialbot]

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@editorialbot commands

[taless474]

@editorialbot add @humanfactors as reviewer

[editorialbot]

@humanfactors added to the reviewers list!

[taless474]

@OlgerSiebinga any other potential reviewers?

[taless474]

@torressa are you interested in reviewing this?

[OlgerSiebinga]

@taless474, I had another look at the list, maybe one of these people would be interested in reviewing:

bnriiitb sea-bass destogl

[taless474]

@bnriiitb are you interested in reviewing this?

[humanfactors]

I begun preparing for my review this week and have spent quite a bit of time trying to validate the functionality.

Unfortunately, I am unable to get CARLA compiled on my computer. I followed the steps specified, but I am having some errors reported on their Github by other users too (related to -[install_rpclib]: - Make sure "CMake" is installed. during PythonAPI Make despite CMake being on path). I admit, I do have other versions of Visual Studio Installed (though I am using 2017), but unfortunately I am not able to uninstall these on my work computer. I really wanted to get this to work, but I also do need to balance the requisite debugging for the purpose of only a single test.

I have had no problems installing JOAN itself, but I will be unable to verify the claimed functionality in terms of it interfacing with a CARLA session. @taless474 I do have sufficient information to carry out a review on other criteria, and am happy to complete that. However, I would think we do need an additional reviewer to actually replicate the interface with Carla.

---

I should also flag this is a bit unique relative to the other work I have reviewed in JOSS. JOSS reviewer guidelines indicates generally the installation process should be as automated as possible (e.g., automated scripts) with an unambiguous set of tests for validating the functionality of the software. The fact that JOAN depends on the compilation of a secondary piece of software does make this criteria a little vague to interpret. To be clear, this issue is with Carla, not JOAN.

[bnriiitb]

@bnriiitb are you interested in reviewing this?

@taless474 sure I will do it.

[taless474]

@editorialbot add @bnriiitb as reviewer

[editorialbot]

@bnriiitb added to the reviewers list!

[taless474]

@editorialbot start review

[editorialbot]

OK, I've started the review over in https://github.com/openjournals/joss-reviews/issues/4250.