search

journalovi / 2024-Cashman-PAC-learning-game

UNDER REVIEW
http://www.journalovi.org/2024-Cashman-PAC-learning-game/
Creative Commons Attribution Share Alike 4.0 International
2 stars 1 forks source link

[REVIEW] PAC Learning Or: Why We Should (and Shouldn't) Trust Machine Learning #5

Open Chenglong-MS opened 4 months ago

Chenglong-MS commented 4 months ago

Conflicts of interest

Reviewed version

fef096a

Review

Review

This is a fantastic paper demonstrating how interactive plots can help introducing PAC learning to the general audience.

First, I'm in full support of this paper. I believe the interactive paper like this will be very beneficial in teaching and education towards general audience. Let me directly jump into some suggestions I think might help with paper presentation.

  1. The first paragraph of intro does not give a sense of "game" for the audience: there is no clear explanation of the user task, the interactive plot, and the user's goal. I would suggest (1) clarifying the task and goal e.g., "you will try to find the best rectangle that defines the concept `men of median height and weight' with certain number of examples, and your correctness is measured by the metrics FP/FN/TP/TN", (2) explain the interaction that the user needs to draw rectangles, (3) explain the difference between training and testing samples (in particular, I highly suggest labeling training and testing samples differently, so that the reader can clearly see what points they use to derive their bounds, and what are new points used in tests).
  2. The "Why is it important" section is a little to generic. While I understand it's necessary to relate existing AI advancement like LLM so that general public would better appreciate the article, it's better to start with some smaller yet more related real world problem (in relation to PAC learning). E.g., regression/classification tasks that are used in business, banking, real-estate.
  3. In the PAC explanation section, it would be a good idea to label variables in the plot directly to help reader leverage the plot to understand the derivation process.
  4. The title "Assuming the worst" is somehow misleading, since it's more or less related to adversarial examples. In fact, it's very common in practice to have these issues. Furthermore, it's also a good idea to provide more examples here about why these non-ideal situations would happen. For example, you can use medical example to explain why samples are non IID, since patients won't get certain tests unless they are in severe situations.
  5. There could be an additional section before conclusion to point out what are further reading beyond PAC learning, and how it is related to existing ML practices.

Additional suggestions

Here are some suggestions that are not required (since they may require additional programming), but they could be nice to have.

  1. I kinda hope the sense of the "game" can be more highlighted in the interactive chart. For example, in the model class selection, it could be fun to let readers choose what model class they like.
  2. Before conclusion section, it could be nice to have several interactive examples to let the user play with some real data, to understand the gap between the theory and practice, but also highlight why theory would be helpful. If I were you, I might build a few examples that are combinations of different conditions (eg.., non-iid, small data // iid, small data // iid, non-rec model class) without telling the author, and ask them to play for prediction.

Technical issues:

  1. When hitting "TEST!" with a rectangle, an error message Cannot read properties of null (reading 'x') would show up and prevents further interaction. Would suggest showing an error message you need to draw a rectangle that shows your concept guess before hitting TEST!.
  2. The layouts of the paper is some what wierd, since the left texts and righthand side plots are not always aligned. The uses of blank space between small sections are nice for the screen, but they make navigation difficult (related bullet points are too far apart). I think it would be a good idea to include some visual cue which bullet point / section the right hand side plot links to.

Conclusion

I'm fully in support of the paper; some revisions would be helpful to improve the paper quality.

Openness/Transparency

not applied since the paper is towards interactive example.

Submission categories

Suggested outcome

Minor revisions: this paper requires some smaller changes, after which I am confident I would be able to endorse it.

Requested changes

Please refer to the reviews.

Requested changes are under review tab. Optional reviews are under additional suggestions tab.

ORCID

No response

dylancashman commented 4 months ago

Hi @Chenglong-MS , thank you for your review! I appreciate all of your comments and will work on them soon.