[16/05/2020] Saturday 9:00 PM GMT+2 Simple Introduction to Active Learning

[Omarito2412]

Join us on Hangout: https://hangouts.google.com/group/kUxBAunjGittAkBUA

Active Learning Tutorial (blog post) Simple inro: https://towardsdatascience.com/introduction-to-active-learning-117e0740d7cc Detailed intro: https://towardsdatascience.com/active-learning-tutorial-57c3398e34d

Practical Obstacles to Deploying Active Learning https://www.aclweb.org/anthology/D19-1003/

[ibrahimsharaf]

Active learning seems like a good semi-supervised approach in theory, but in practice (NLP), as the paper mentioned it provides limited improvements.

• In case of text classification (balanced training data), it doesn't offer any consistent improvements.
• In sequence tagging (NER) case, it added a small, but consistent improvement, most probably because the data imbalance.
• Another practical problem is the need to couple the data/annotation/classification with the same model, because when they used different combinations of acquisition/successor models, the performance degraded, so this is quite limiting, as data outlive algorithms, the opposite isn't true.

Interesting follow-ups:

• Practical Obstacles to Deploying Active Learning paper video at EMNLP (https://vimeo.com/404746677)
• Does active learning work? (Prodigy) https://support.prodi.gy/t/active-learning-does-it-work/542
• Prodigy introduction, how does it use active learning. (https://explosion.ai/blog/prodigy-annotation-tool-active-learning)

[hadyelsahar]

my 2 cents:

• a personal opinion: dataset selection methods has to be w.r.t the model in hand. This can be confirmed with what has been shown in the paper about the -ve results of AL acquired dataset transfer.
• SOTA confidence calibration methods: https://www.reddit.com/r/MachineLearning/comments/907p7a/d_what_is_the_current_state_of_the_art_in/
• Considering Muhammad's Question on data unbalance, you might be interested to look this zack lipton's talk: https://www.youtube.com/watch?v=zo4oRqfMrgo

[Omarito2412]

Active learning is not continuous learning

The benefits of AL are questionable. In some cases, it might even be useless. So apparently the return of using AL depends on the model itself. Also, another downside is that if we use AL and decide to change the model we're training, will we lose our data? since AL produces samples based on a certain model, then it's coupling the data and the model together. That's not a good thing.

Overall, AL can be beneficial in some settings where it's expensive to label data. I think further research is needed to determine how to choose samples that improve the labeled data set rather than improve a certain model.

[mukhal]

My very brief take on AL

Why and How? Minimize annotation effort. The model is actively participating in the learning process by selecting for YOU which samples YOU should label.

Does it work? Not really

Why?

1. If your model is unfit for the task, the model will be inaccurate in selecting suitable data points. This means you'll spend much time labeling data for that model, which are the wrong data points, and which will not work at the end - driving you to search for another model which you'll have to relabel data for it.
2. Active learning this way seems like a self-sabotaging concept. While its whole premise is to reduce labeling, in practice, it will lead to even MORE labeling.

Future Directions? We must find a way to decouple data selection from the model performance. It's only by finding a more general, model-agnostic approach to perform AL that we can utilize AL in practical settings.