Participants work through methods section

[agitter]

One of our goals is for participants to be able to read machine learning literature in their domain. How can we reintroduce this as a in-workshop or post-workshop activity? Should we let participants bring their own paper or have some to choose from that span multiple fields?

[dltreu]

I really like this suggestion. What about having this reading as part of the homework in between the sessions, if we are thinking of breaking into a more introductory part and a more applied part, with a week or two in between for "content absorption." This reading could be homework to help them apply the introductory material and glossary of terms, etc.

It might help to identify and offer several articles, at least as examples. Articles that are at a certain level that would be accessible for introductory ML understanding. They could use one of those - or suggest their own.

[agitter]

Yes, if we split the workshop into two sessions then this would be a great homework activity. Offering example articles will likely be necessary. Many papers do not write down their machine learning methodology in sufficient detail.

[agitter]

We could come up with a set of structured questions that would help participants know what they should look for when they read a machine learning paper in their domain. For instance:

• What type of classifier was used to generate the results? What hyperparameters were used?
• How were the classifier and its hyperparameters selected? Were many classifiers considered and evaluated empirically? Were prior results from other papers used to narrow the classifiers considered? Was the selection process conducted in a way that is not biased or over-optimistic?
• How was the classifier's performance evaluated? What metrics were used and are they appropriate for the domain and dataset? Did the evaluation use cross-validation, a held out test set, a completely independent dataset, or something else?
• What baseline classifiers were considered? What does the performance of baseline methods tell us about how difficult the classification task is?
• How was the dataset processed? Were any instances or features excluded from model training or evaluation?

There are other questions I ask when reading a paper, but those are less relevant for our audience. Those questions pertain to the software used, the robustness of the results and trained model, the optimization algorithm used for training, etc.

We could go through one example manuscript and answer questions like this during the workshop. Then, we can ask participants to work through another one of our example manuscripts on their own as homework.

[agitter]

For the February 25, 2021 workshop we added an assessment activity to work example text from a paper. It worked well and should remain part of the workshop.

One minor comment is that when the example text uses different terms that we use in the workshop, we should replace or define them. Participants will not know that recall is the same as sensitivity even though it appears in the table in the lesson.