[REVIEW]: JustSayIt.jl: A Fresh Approach to Open Source Voice Assistant Development

whedon commented 1 year ago

Submitting author: !--author-handle-->@omlins@matbesancon<!--end-editor-- Reviewers: @PetrKryslUCSD, @pfitzseb Archive: 10.5281/zenodo.14057584

Status

Status badge code:

HTML: <a href="https://proceedings.juliacon.org/papers/3325b7091de2a68c2ef6201768f1de79"><img src="https://proceedings.juliacon.org/papers/3325b7091de2a68c2ef6201768f1de79/status.svg"></a>
Markdown: [![status](https://proceedings.juliacon.org/papers/3325b7091de2a68c2ef6201768f1de79/status.svg)](https://proceedings.juliacon.org/papers/3325b7091de2a68c2ef6201768f1de79)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@PetrKryslUCSD & @pfitzseb, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @carstenbauer know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @PetrKryslUCSD

Conflict of interest

[ ] As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[ ] I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

[ ] Repository: Is the source code for this software available at the repository url?
[ ] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[ ] Authorship: Has the submitting author (@omlins) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[ ] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
[ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

[ ] Authors: Does the paper.tex file include a list of authors with their affiliations?
[ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[ ] Page limit: Is the page limit for extended abstracts respected by the submitted document?

Content

[ ] Context: is the scientific context motivating the work correctly presented?
[ ] Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
[ ] Results: are the results presented and compared to approaches with similar goals?

Review checklist for @pfitzseb

Conflict of interest
[x] As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Authorship: Has the submitting author (@omlins) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

[x] Authors: Does the paper.tex file include a list of authors with their affiliations?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Page limit: Is the page limit for extended abstracts respected by the submitted document?

Content

[x] Context: is the scientific context motivating the work correctly presented?
[x] Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
[x] Results: are the results presented and compared to approaches with similar goals?

whedon commented 1 year ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @PetrKryslUCSD, @pfitzseb it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/JuliaCon/proceedings-review) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/JuliaCon/proceedings-review:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 1 year ago

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.04 s (976.8 files/s, 83311.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Julia                           35            405            167           2756
YAML                             3              1              6            116
TOML                             2              4              0             29
Markdown                         1              4              0              9
-------------------------------------------------------------------------------
SUM:                            41            414            173           2910
-------------------------------------------------------------------------------

Statistical information for the repository '9ee0f315c44c9ac759829a5d' was
gathered on 2023/01/09.
No commited files with the specified extensions were found.

whedon commented 1 year ago

PDF failed to compile for issue #121 with the following error:

 Can't find any papers to compile :-(

carstenbauer commented 1 year ago

@whedon generate pdf from branch JuliaConProceeding2022

whedon commented 1 year ago

Attempting PDF compilation from custom branch JuliaConProceeding2022. Reticulating splines etc...

whedon commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 1 year ago

:wave: @pfitzseb, please update us on how your review is going (this is an automated reminder).

whedon commented 1 year ago

:wave: @PetrKryslUCSD, please update us on how your review is going (this is an automated reminder).

pfitzseb commented 1 year ago

Hey, some feedback on this submission:

In Fig 2, a using JustSayIt.API is required (same goes for the documentation at https://omlins.github.io/JustSayIt.jl/stable/usage/
https://github.com/omlins/JustSayIt.jl/issues/87 causes the example in Fig 2 to be non-functional.
https://github.com/omlins/JustSayIt.jl/issues/88 causes the Keyboard.type command to be non-functional.
I'm seeing worse latency than shown in the article, but I'm happy to chalk that up to benchmarking on a signigicantly less powerful machine.

``` julia> versioninfo() Julia Version 1.8.3 Commit 0434deb161e (2022-11-14 20:14 UTC) Platform Info: OS: Linux (x86_64-linux-gnu) CPU: 8 × Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-13.0.1 (ORCJIT, skylake) Threads: 1 on 8 virtual cores julia> start(commands=commands, max_speed_subset=["ma", "select", "okay", "middle", "right", "double", "triple", "copy", "upwards", "downwards", "take"]) [ Info: JustSayIt: I am initializing... [ Info: Listening for commands in English (United States) (say "sleep JustSayIt" to put me to sleep; press CTRL+c to terminate)... [ Info: Starting command: click_left (latency: 34 ms) [ Info: Starting command: click_left (latency: 8 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 25 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 32 ms) [ Info: Starting command: click_left (latency: 30 ms) [ Info: Starting command: click_left (latency: 25 ms) [ Info: Starting command: click_left (latency: 21 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 27 ms) [ Info: Starting command: click_left (latency: 20 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left [ Info: Starting command: click_left (latency: 34 ms) [ Info: Starting command: click_left (latency: 24 ms) [ Info: Starting command: click_left (latency: 29 ms) [ Info: Starting command: click_left (latency: 26 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 20 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 21 ms) [ Info: Starting command: click_left (latency: 21 ms) [ Info: Starting command: click_left (latency: 26 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 25 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 27 ms) [ Info: Starting command: click_left (latency: 30 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 25 ms) [ Info: Starting command: click_left (latency: 24 ms) [ Info: Starting command: click_left (latency: 32 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 27 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 21 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 21 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 25 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 29 ms) [ Info: Starting command: click_left (latency: 24 ms) [ Info: Starting command: click_left (latency: 21 ms) [ Info: Starting command: click_left (latency: 26 ms) [ Info: Starting command: click_left (latency: 26 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 28 ms) [ Info: Starting command: click_left (latency: 25 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 28 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 23 ms) [ Info: Starting command: click_left (latency: 22 ms) [ Info: Starting command: click_left (latency: 28 ms)
There are no community guidelines. I'd recommend linking something like https://github.com/SciML/ColPrac
Some comparisons to other implementations of a text-to-command software would be nice (in terms of API, performance, latency)

carstenbauer commented 1 year ago

Hey @omlins, please take a look at @pfitzseb's suggestions and comments above and try to incorporate or comment on them.

@PetrKryslUCSD Any update on your review? Would be great to get this in soon. Thanks in advance.

carstenbauer commented 1 year ago

@PetrKryslUCSD Can you please provide your review so that we can push this over the line?

@omlins Did you already incorporate the suggestions by @pfitzseb?

omlins commented 1 year ago

@carstenbauer : I had addressed some of the issues only partially until recently as the second review is still pending. However, I have now been able to address all the issues raised by @pfitzseb conclusively, with exception of the last one.

Here, @pfitzseb and @carstenbauer, you find a reply to each issue raised:

* In Fig 2, a `using JustSayIt.API` is required (same goes for the documentation at https://omlins.github.io/JustSayIt.jl/stable/usage/

Some stale commands in the max_speed_subset had caused the error when executing the code in Fig 2 (resolved here: https://github.com/omlins/JustSayIt.jl/pull/91). This error does not seem to be present in the documentation.

* [Usage example fails due to incorrect type checks omlins/JustSayIt.jl#87](https://github.com/omlins/JustSayIt.jl/issues/87) causes the example in Fig 2 to be non-functional.

This code issue has been solved and the corresponding pull request merged (https://github.com/omlins/JustSayIt.jl/pull/89).

* [Keyboard.type fails with "insecure command argument recognition" omlins/JustSayIt.jl#88](https://github.com/omlins/JustSayIt.jl/issues/88) causes the `Keyboard.type` command to be non-functional.

This was not an issue with the code, but an incorrect usage of JustSayIt (https://github.com/omlins/JustSayIt.jl/issues/88#issuecomment-1427649211).

* I'm seeing worse latency than shown in the article, but I'm happy to chalk that up to benchmarking on a signigicantly less powerful machine.

I believe it is right to expect that the latency will be different from system to system.

* There are no community guidelines. I'd recommend linking something like https://github.com/SciML/ColPrac

The contribution guide line has been extended and a link to the mentioned guideline added (https://github.com/omlins/JustSayIt.jl/pull/92).

* Some comparisons to other implementations of a text-to-command software would be nice (in terms of API, performance, latency)

I would prefer to wait for the second review to address this issue, because additions to the text mean also removal of existing parts, given the very short format of this paper (we are already at the limit of the two pages).

In addition to these issues, I've also been able to solve the longstanding issue with the automatic installation not working in some corner cases, which @pfitzseb observed as well (resolved here: https://github.com/omlins/JustSayIt.jl/pull/90). Note that independent of all the issues raised, two unit tests fail since recently on some versions of Ubuntu (the unit testing is complex, because the Vosk speech recognition tool kit used is not fully deterministic). They are not critical and could be removed in another pull request.

matbesancon commented 1 year ago

ping @PetrKryslUCSD

matbesancon commented 1 year ago

@whedon assign @matbesancon as editor

lucaferranti commented 2 months ago

@editorialbot assign @matbesancon as editor

editorialbot commented 2 months ago

Assigned! @matbesancon is now the editor

pfitzseb commented 1 month ago

Alright, this ticks all the boxes now :)

matbesancon commented 1 month ago

@editorialbot accept

editorialbot commented 1 month ago

Paper is not ready for acceptance yet, the archive is missing

matbesancon commented 1 month ago

@omlins can you add the version and Zenodo DOI archive?

omlins commented 2 weeks ago

@omlins can you add the version and Zenodo DOI archive?

I just saw the notification. Yes, I will do so.

omlins commented 2 weeks ago

@matbesancon : here you can find the DOI for the paper: https://doi.org/10.5281/zenodo.14057465 here you can find the DOI for the code: https://doi.org/10.5281/zenodo.14057584

matbesancon commented 1 week ago

@editorialbot set 10.5281/zenodo.14057584 as archive