[REVIEW]: JOAN: a framework for human-automated vehicle interaction experiments in a virtual reality driving simulator

editorialbot commented 2 years ago

Submitting author: !--author-handle-->@OlgerSiebinga@diehlpk<!--end-editor-- Reviewers: @humanfactors, @imcatta Archive: 10.4121/21975335

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/25b2e051dac33531e497c91e99e67d68"><img src="https://joss.theoj.org/papers/25b2e051dac33531e497c91e99e67d68/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/25b2e051dac33531e497c91e99e67d68/status.svg)](https://joss.theoj.org/papers/25b2e051dac33531e497c91e99e67d68)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@humanfactors & @bnriiitb, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @taless474 know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @humanfactors

📝 Checklist for @imcatta

editorialbot commented 2 years ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 2 years ago

Software report:

github.com/AlDanial/cloc v 1.88  T=0.36 s (446.7 files/s, 55135.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          95           1918           2048           7105
Qt                              34              0              0           6337
Markdown                        28            669              0           1364
TeX                              1             26              0            233
JSON                             1              0              0            104
YAML                             2             15              3             50
-------------------------------------------------------------------------------
SUM:                           161           2628           2051          15193
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

editorialbot commented 2 years ago

Wordcount for paper.md is 1126

editorialbot commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1109/THMS.2018.2791570 is OK
- 10.1007/978-3-319-67361-5_40 is OK
- 10.1016/j.apergo.2020.103196 is OK
- 10.1177/0018720812443984 is OK
- 10.31234/osf.io/p8dxn is OK
- 10.4324/9781315565712 is OK
- 10.2514/6.2000-4503 is OK
- 10.1016/j.aap.2016.10.016 is OK
- 10.1016/j.trc.2021.103008 is OK
- 10.1155/2021/4396401 is OK

MISSING DOIs

- None

INVALID DOIs

- None

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

taless474 commented 2 years ago

@humanfactors, thank you for your comment on the PRE REVIEW issue. @bnriiitb, thank you for accepting to review this. Please let me know in the future if you can replicate the interface with Carla.

humanfactors commented 2 years ago

Review checklist for @humanfactors

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/tud-hri/joan?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@OlgerSiebinga) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

humanfactors commented 2 years ago

Hi @OlgerSiebinga, I've made an Issue regarding my feedback on the JOAN side (rather than paper itself): https://github.com/tud-hri/joan/issues/10

Overall, I think this is an impressive contribution to human factors research 😊! I admit though, I couldn't get Carla installed, and I am not familiar enough with JOSS to know whether the manual compilation and installation is an issue at all.

The only other question I'll raise is I noticed there were a few "stale" issues (e.g., https://github.com/tud-hri/joan/issues/3), and might suggest it could be good to have a template to request a reproducible example to replicate the bug etc, or identify a pathway to close these kind of things up. This isn't really specific to the review though, but I was curious as it related to contributing guidelines.

Cheers!

OlgerSiebinga commented 2 years ago

Hi All,

We have been working on the feedback in https://github.com/tud-hri/joan/issues/10, and we have addressed most of it. Hopefully, we can close that issue soon. @humanfactors, thanks for summarizing your feedback in that issue, it was really helpful.

@taless474, I saw that there is only a checklist for @humanfactors in this issue. Will @bnriiitb get a checklist once @humanfactors is done? Or was the editorialbot confused when setting up this review issue?

taless474 commented 2 years ago

@OlgerSiebinga, I apologize for the late response. @bnriiitb can use @editorialbot generate my checklist to generate his checklist. How is the review going @bnriiitb ?

humanfactors commented 2 years ago

I have just returned back to work after being largely out of service in remote Australia. I will endeavor to complete my review this week. Thanks.

taless474 commented 2 years ago

@humanfactors how is the review going?

humanfactors commented 2 years ago

Hi apologies for the delays, I've had COVID and been quite unwell.

A few minor tasks on the software paper:

References are appearing incorrect, the authors do need to check up on the reference syntax (e.g., double brackets L41, L97)
SensoDrive ref missing date

As indicated prior, I was unable to install Carla side of the software, so I am unable to actually check off the other tasks. I was waiting for the other review to report how they got along.

Cheers!

OlgerSiebinga commented 2 years ago

@editorialbot generate pdf

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

OlgerSiebinga commented 2 years ago

@editorialbot generate pdf

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

OlgerSiebinga commented 2 years ago

@editorialbot generate pdf

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

OlgerSiebinga commented 2 years ago

I think I've addressed all the reference issues now (after three attempts). Thanks for spotting them.

OlgerSiebinga commented 2 years ago

Dear all,

I was wondering how the review is going. Please let us know if there is anything we (as authors) can do to help finish the review. Thanks!

taless474 commented 2 years ago

Dear all,

I was wondering how the review is going. Please let us know if there is anything we (as authors) can do to help finish the review. Thanks!

@humanfactors @bnriiitb

humanfactors commented 2 years ago

Hi @taless474 and @OlgerSiebinga,

My review is completed insofar as I have checked off every item that I possibly can. I am satisfied with the manuscript and all changes made by the authors. As mentioned previously, I was unable to install the CARLA software and I am not sure what the editorial protocol is in such situations. If another independent review can confirm they can compile the software, then I guess I can also check off those items.

I spent a good day or so attempting the detailed compilation and installation process, and ran into issues with the CARLA compilation process (https://github.com/openjournals/joss-reviews/issues/4162#issuecomment-1067511114). To be totally transparent, I cannot commit further resources to that aspect of the review given the substantial investment that would be required on my part to validate a single test.

Apologies I cannot be of more help, but if @bnriiitb similarly does not have the capacity, perhaps we should be identifying a third reviewer who is able to compile and expedite the review process?

humanfactors commented 2 years ago

Just to follow up again, I can appreciate it has been some time in validating this review. @bnriiitb do you still have capacity to test the installation? If you likewise have trouble, we may need to discuss alternative methods for validating it...

taless474 commented 2 years ago

@bnriiitb would you please test the installation?

OlgerSiebinga commented 2 years ago

Hi All,

We paused the development of JOAN pending this review, but it has been a while now. So, we would like to resume the development on the develop and demo branches (we plan to make some substantial changes to the docs and procedures to allow JOAN to run with newer versions of CARLA). Please use the master branch for this review, we will not touch it until the review is done. That way, the review can continue on the same version of JOAN while we work on newer versions in parallel. Just wanted to let you know, thanks!

taless474 commented 2 years ago

@bnriiitb please let us know your suggestion

humanfactors commented 2 years ago

@taless474 It is plausible that @bnriiitb may not have notifications in this thread on, as I don't believe he's made an active response since 18 Mar (> 4.5 months). Is there a protocol for seeking a third reviewer to replicate the process?

Again, I am confident that the actual software that is being published can install/run, just not how it interacts with Carla.

taless474 commented 2 years ago

@humanfactors that's a good point. I will email them.

OlgerSiebinga commented 2 years ago

Hi All, A quick update from our side. We have updated JOAN to support Carla 0.9.13. This new version of JOAN (v1.2.0) is currently on the develop branch waiting for this review to finish before we merge it in the main branch. This update to Carla 0.9.13 means that the installation procedure does no longer depend on visual studio 2017. Which should make installation a bit easier.

Are there any updates regarding the review?

humanfactors commented 2 years ago

Hi @OlgerSiebinga

In my (somewhat limited) experience, I would say this is an abnormal delay in your paper, but indeed the complexity of the build is part of the issue. I could be wrong (@taless474 can confirm) but I changed the version of my submission for my JOSS paper — so I believe you could adjust your build while this is under review. You need to document and adjust the post details to make clear that there is a new version being submitted that addresses critiques of the review.

I'm happy to attempt to build again from the develop branch, under the assumption that a secondary reviewer is not required. My concern is spending the time to reproduce and approve — but end up still needing a second reviewer, which wouldn't be a productive immediate action to get this through.

Good luck :)

arfon commented 2 years ago

:wave: @taless474 – Is there anything I can do here to help move this submission along? It looks like we will indeed need a second reviewer here.

arfon commented 2 years ago

@editorialbot assign @diehlpk as editor

:wave: folks – @diehlpk has kindly volunteered to step in as the handling editor here as @taless474 is not currently available to edit. Thanks Patrick!

editorialbot commented 2 years ago

Assigned! @diehlpk is now the editor

diehlpk commented 2 years ago

Hi @humanfactors can you please look in the simplified installation process?

diehlpk commented 2 years ago

@OlgerSiebinga can you please recommend a new reviewer?

diehlpk commented 2 years ago

@editorialbot remove @bnriiitb as reviewer

editorialbot commented 2 years ago

@bnriiitb removed from the reviewers list!

OlgerSiebinga commented 2 years ago

👋 @diehlpk, thanks for stepping in. I've had a look at the list of reviewers again and selected the following people as possible reviewers:

eric3722 vmatos adity-om nmullins imcatta

diehlpk commented 2 years ago

Hi @OlgerSiebinga

In my (somewhat limited) experience, I would say this is an abnormal delay in your paper, but indeed the complexity of the build is part of the issue. I could be wrong (@taless474 can confirm) but I changed the version of my submission for my JOSS paper — so I believe you could adjust your build while this is under review. You need to document and adjust the post details to make clear that there is a new version being submitted that addresses critiques of the review.

I'm happy to attempt to build again from the develop branch, under the assumption that a secondary reviewer is not required. My concern is spending the time to reproduce and approve — but end up still needing a second reviewer, which wouldn't be a productive immediate action to get this through.

Good luck :)

You can proceed with your review. We will find a second reviewer soon. It sometimes happens that one reviewer disappears. @humanfactors

diehlpk commented 2 years ago

Hi @eric3722 or @vmatos or @adity-om do you have time to review that paper?

diehlpk commented 1 year ago

Hi @jackbrookes do you have time to review this paper?

diehlpk commented 1 year ago

Hi @ionlights do you have time to review that paper?

diehlpk commented 1 year ago

Hi @nmullins and @imcatta do you have time to review this paper?

diehlpk commented 1 year ago

@humanfactors how is your review going?

imcatta commented 1 year ago

@diehlpk yes! Feel free to assign me this review

adity-om commented 1 year ago

I can review it.

Sincerely, Om

Aditya Om Pronouns: He/Him/His Secretary, Rackham Student Government Rackham Representative, Central Student Government MS Electrical Eng. & Computer Science Candidate Rackham Graduate School, University of Michigan, Ann Arbor @.***

On Mon, Oct 24, 2022, 4:31 PM Patrick Diehl @.***> wrote:

Hi @eric3722 https://github.com/eric3722 or @vmatos https://github.com/vmatos or @adity-om https://github.com/adity-om do you have time to review that paper?

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/4250#issuecomment-1289577571, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHZRF733XRDGKQERH65D53WE3W3DANCNFSM5Q7QLDFA . You are receiving this because you were mentioned.Message ID: @.***>

diehlpk commented 1 year ago

@editorialbot add @imcatta as reviewer

editorialbot commented 1 year ago

@imcatta added to the reviewers list!

diehlpk commented 1 year ago

@editorialbot add @adity-om as reviewer

openjournals / joss-reviews