This article is an interesting introductory piece that explores how machine learning techniques can fail in three scenarios. The topic is well-motivated, and I fully support the authors' assertion that it is crucial for general public to understand the reasons behind the errors made by ML systems. For this purpose, the interactive "quiz" is an effective and engaging method for readers to explore these issues. It provides a clear and accessible way for readers to engage with and reflect on the vulnerabilities of machine learning (ERM in the parcular context of this article).
I am generally supportive of this article; however, I recommend several improvements for clarity and detail.
Clarify
Game description: The game needs to be more detailed in the introduction. For example, it is not immediately clear that players are asked to drag a rectangle to interact with the game. Including detailed instructions about user interactions and goals would be beneficial.
Animation sssues: The animation process (i.e., the fade-in of points) is somewhat confusing. It is unclear how many points are to be drawn, which might perplex readers. Having a fast-forward button or a progress bar might help clarify the progression of points being displayed.
Scroll interaction. The current setup updates the chart on the right whenever the corresponding text section head becomes visible in the browser window. This can be somewhat confusing and disruptive. To enhance clarity and user experience, I recommend triggering the update only when the corresponding text reaches the center of the window.
Content
While the game presents an useful abstraction of a machine learning problem, its representation might be excessively abstract. At first glance of the game, one might ask - what is the graph showing? What does the x/y axis encode? The text briefly mentions that this graph is for deciding "men of medium height and weight"; however, a clearer connection between the text and the graph would enhance understanding.
Moreover, the "Why this is Important" section seems loosely related to the chart/game. Readers might mistakenly believe that the referenced Gender Shade example is linked to the adjacent chart, which it is not. To improve coherence, it would be beneficial to discuss this example within the context of the "training-testing mismatch" section, where it might be more relevant.
Bugs
I sometimes triggered the error - Cannot read properties of null (reading 'x'), when clicking on the "TEST" button. It might has something to do with the animations - users clicked this button before all points were drawn.
Openness/Transparency
N/A
Submission categories
[ ] Registered Report
[X] Replication Study
[ ] Empirical Research - Quantitative
[ ] Empirical Research - Qualitative
[ ] Systems or design research
[ ] Commentary
[ ] Systematic Literature Review
Suggested outcome
Minor revisions: this paper requires some smaller changes, after which I am confident I would be able to endorse it.
Requested changes
To fix the programming bugs and minor UX issues in the interactive demo.
To provide detailed explainations and instructions on the "Four German Game".
Conflicts of interest
Reviewed version
fef096a
Review
This article is an interesting introductory piece that explores how machine learning techniques can fail in three scenarios. The topic is well-motivated, and I fully support the authors' assertion that it is crucial for general public to understand the reasons behind the errors made by ML systems. For this purpose, the interactive "quiz" is an effective and engaging method for readers to explore these issues. It provides a clear and accessible way for readers to engage with and reflect on the vulnerabilities of machine learning (ERM in the parcular context of this article).
I am generally supportive of this article; however, I recommend several improvements for clarity and detail.
Clarify
Game description: The game needs to be more detailed in the introduction. For example, it is not immediately clear that players are asked to drag a rectangle to interact with the game. Including detailed instructions about user interactions and goals would be beneficial.
Animation sssues: The animation process (i.e., the fade-in of points) is somewhat confusing. It is unclear how many points are to be drawn, which might perplex readers. Having a fast-forward button or a progress bar might help clarify the progression of points being displayed.
Scroll interaction. The current setup updates the chart on the right whenever the corresponding text section head becomes visible in the browser window. This can be somewhat confusing and disruptive. To enhance clarity and user experience, I recommend triggering the update only when the corresponding text reaches the center of the window.
Content
While the game presents an useful abstraction of a machine learning problem, its representation might be excessively abstract. At first glance of the game, one might ask - what is the graph showing? What does the x/y axis encode? The text briefly mentions that this graph is for deciding "men of medium height and weight"; however, a clearer connection between the text and the graph would enhance understanding.
Moreover, the "Why this is Important" section seems loosely related to the chart/game. Readers might mistakenly believe that the referenced Gender Shade example is linked to the adjacent chart, which it is not. To improve coherence, it would be beneficial to discuss this example within the context of the "training-testing mismatch" section, where it might be more relevant.
Bugs
I sometimes triggered the error - Cannot read properties of null (reading 'x'), when clicking on the "TEST" button. It might has something to do with the animations - users clicked this button before all points were drawn.
Openness/Transparency
N/A
Submission categories
Suggested outcome
Minor revisions: this paper requires some smaller changes, after which I am confident I would be able to endorse it.
Requested changes
To fix the programming bugs and minor UX issues in the interactive demo.
To provide detailed explainations and instructions on the "Four German Game".
ORCID
No response