[DECISION] Probably Approximately Correct Learning: a machine learning game 06-07-2024

Conflicts of interest

[X] I declare that I have no known conflicts of interest with the authors.

Reviewed version

cc82db3f926b33545925eacf5418729865c205b5

Reviews summarized

R1 #5
R2 #6
R3 #7
Accessibility #10

Meta-Review

Overall, the reviewers agree that this article is well motivated and makes a valuable contribution as a general introduction to PAC learning. That being said, all of the reviewers mention specific revisions they believe are necessary prior to publication. Given that some concerns were raised beyond basic text changes, I believe this submission currently falls between a minor and major revision, which is also reflected in the decisions of the reviewers. For this reason, I suggest the more conservative decision of major revisions.

Revisions to focus on for the next round of review include:

Interactive Game

Reviewers seemed to be most concerned about the clarity and behavior of the interactive game.

At a high level, R1 and R2 request more details and instructions on how the game works in the introduction (#5, #6).
At a low level, several technical issues were raised. R2 notes that the fade in animation for points can be confusing (#6). R3 perceived a difference between the points displayed thus far and the final set of points displayed after clicking the "TEST!" button (#7) I noticed that when no rectangle is drawn in the game, an error is thrown when I hit the "TEST!" button. R1 and R2 observed a similar issue (#5, #6).
These suggestions are optional, but reviewers also mentioned potential enhancements. For example, R3 suggests visualizing true/false positives and negatives to enhance clarity (#7). R1 also suggests some enhancements to make the game more personalized and game-like (#5).

Article Clarity

Apart from the game, reviewers mentioned certain sections/arguments in the article that could be clarified.

R1 and R2 find the Introduction -"Why this is important" section to be a bit too abstract (#5, #6). Further, R2 finds the current positioning of the Gender Shade example to be confusing/misleading (#6).
R2 would like to see a clearer explanation of the visualization in the game within the article text (#6). Similarly, R1 recommends labeling relevant variables within the visualization to clarify their purpose within the text (#5).
R1 finds the "Assuming the Worst" title to be misleading, and recommends framing this section in terms of adversarial examples (#5).
In terms of clarifying concepts, R3 asked whether toughest fit error may have a more general explanation in terms of sum of error rather than precise fractions of epsilon (#7).

Styling

While styling was not a major concern for reviewers, they all mentioned specific styling issues that could be improved.

R1 points out that the text and the visuals may not always be aligned correctly (#5). Similarly, R2 finds that the game may update too early in response to article transitions (#6).
R3 notes that some article styling breaks in the absence of an internet connection (#7). R3 also makes multiple (minor) suggestions for fixing typos and improving the overall styling of the article (#7).

JoVI Required Materials

In my initial review, I noticed that there is not a long-term archive mentioned in the Abstract and Required Materials (#2).
[OPTIONAL] I noticed there is no Table of Contents section (#2). Please include a Table of Contents or let us know if this will remain omitted.

Accessibility Review + Changes

Many of the accessibility checks are covered but some areas can be improved (#10):
- Some colors may be low contrast/difficult to perceive (especially red and green points in the game). When possible, avoiding reliance on colors is recommended.
- Some icons and buttons are missing alt text.
- The article does not appear to be responsive to different window sizes or form factors (such as reading from a phone).

Decision

Major revisions: this paper requires substantial improvements that I will need to re-review to decide whether or not to endorse it.

journalovi / 2024-Cashman-PAC-learning-game