[REVIEW]: RcmdrPlugin.OptimClassifier: Extending the R Commander Interface to create the best train for classification models

[whedon]

Submitting author: @economistgame (Agustín Pérez-Torregrosa) Repository: https://github.com/economistgame/RcmdrPlugin.OptimClassifier Version: v0.1.2 Editor: @karthik Reviewer: @andrewheiss Archive: Pending

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/fb8deffc7e4c42deace35e691e0f8c8d"><img src="http://joss.theoj.org/papers/fb8deffc7e4c42deace35e691e0f8c8d/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/fb8deffc7e4c42deace35e691e0f8c8d/status.svg)](http://joss.theoj.org/papers/fb8deffc7e4c42deace35e691e0f8c8d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@@andrewheiss, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @karthik know.

Review checklist for @@andrewheiss

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Version: Does the release version given match the GitHub release (v0.1.2)?
• [x] Authorship: Has the submitting author (@economistgame) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [ ] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• [ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [ ] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @@andrewheiss it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

--> Check article proof :page_facing_up: <--

[economistgame]

@andrewheiss posted this in the prereview issue (#697): I can't accept the repository invitation, possibly because the invitation was made with two @@ s?

[karthik]

@economistgame Yes we're aware and working on this.

@afron This was my typo. Can you add manually add Andrew as a collaborator so I can assign him the issue.

[andrewheiss]

I've been added now—all is well! Thanks!

[andrewheiss]

RcmdrPlugin.OptimClassifier is a helpful new package that provides a menu-based interface for newer machine learning classification models. It was easy to install and use, and I only came across a few issues, noted below.

The paper has a good, succinct justification for why this library is needed, but the README doesn't. It might be helpful to include some of that justification there (noting that R Commander provides a graphic interface for building models in R, etc.). Additionally, the paper has a list of models that the plugin allows users to build; the README doesn't and probably should.

The paper says that the menus provide 7 different models, but the actual R Commander menus only list 6—classification and regression trees (CART) is not an option.

I've opened a few issues that I ran into when testing and fitting models. I'm not sure if all the problems I ran into are specific to RcmdrPlugin.OptimClassifier or just OptimClassifier, but they definitely affect the use of RcmdrPlugin.OptimClassifier so they should probably be fixed.

It might be helpful to add some tests to ensure that the OptimClassifier::Optim*() functions continue to work after loading RcmdrPlugin.OptimClassifier. In its current form, there are no tests. Part of this is probably because R Commander plugins are harder to write tests for, since they're based on a GUI (my own R Commander plugin package doesn't have any tests 😬—I should fix that someday). However, in this situation, with the possible conflicting namespace issues noted in https://github.com/economistgame/RcmdrPlugin.OptimClassifier/issues/2, some test coverage could catch these problems.

It also might be helpful to include an explanation or screenshot in the README or the JOSS article indicating where to find the new OptimClassifier menu options (similar to this). After loading data, the "Models" menu is still mostly greyed out—users have to go to "Fit models" > "Optimum XXX". Part of this confusion is due to how R Commander works—it only shows stuff in the "Models" menu after a model has been fit, and models can only be fit with "Statistics" > "Fit models". Longtime users of R Commander will know this, but new users (the target audience of R Commander) might struggle with where to find the new menu items.

Explaining what else the package offers could be helpful. As it stands now, it's most obvious that it adds six menu items under "Statistics" > "Fit models". Each of those model types have a few additional options under the "Models" menu—users can summarize them and create diagnostic plots for them (all other model options are greyed out). It's not immediately clear that users can do this with the newly-fit models though.

It would be helpful to have some community guidelines in the package too, such as a CONTRIBUTING.md file (perhaps modeled after something like this or this) and a CONDUCT.md file (like this or this)

One reference in the BibTeX file needs a DOI: Using the R Commander: 10.1201/9781315380537. Perez:2018 doesn't have a DOI yet (but will?), and Fox:2007 and R:2018 don't have one assigned to them as far as I can tell, so they're okay.

Some of the English in the README and the paper needs to be revised and edited. I have suggested a number of edits in https://github.com/economistgame/RcmdrPlugin.OptimClassifier/pull/6 and https://github.com/economistgame/RcmdrPlugin.OptimClassifier/pull/7. In addition to those suggestions, the title doesn't need to have "Interface," "Classification," and "Models" capitalized.

Phew. That's all I can see. This is great!

[arfon]

:wave: @economistgame - please respond to @andrewheiss' review when you get a chance.

[karthik]

@economistgame Any update on this?

[karthik]

@economistgame Could you please give us an update on where things stand?

[karthik]

Hi @economistgame Another gentle ping to get this wrapped up. If you are unable to continue this review please also let us know.

[karthik]

👋 @economistgame Can you please update us on progress?

[economistgame]

I'm very sorry, @karthik but I can not find the solution to the failures. Also, I do not know how I will perform the test

[labarba]

👋 @economistgame — We haven't heard from you in a while. The review of your submission has been paused for 4 months. Do you think you'll be able to work on this, or should we think of withdrawing the submission? Let us know your thoughts!

[arfon]

It's now been more than six months since we've heard from @economistgame so I'm going to assume they are no-longer interested in pursuing this publication with JOSS.

@andrewheiss @karthik thanks for your help with this paper. Sorry this didn't make it completely through review.