04/30: Munro

[shevajia]

Comment below with questions or thoughts about the reading for this week's workshop.

Please make your comments by Wednesday 11:59 PM, and upvote at least five of your peers' comments on Thursday prior to the workshop. You need to use 'thumbs-up' for your reactions to count towards 'top comments,' but you can use other emojis on top of the thumbs up.

[wanitchayap]

I think the idea of Active Learning is very fascinating! Thank you so much for sharing such interesting work with us. I have 2 questions:

1) For Uncertainty Sampling and Adaptive Sampling, your algorithm adds a new output layer to the model to classify whether each unlabeled observation will correctly or incorrectly be classified by the original model. I believe this will only work with a classification that has non-overlapping labels? How could we apply this idea for classification that can classify an observation as several labels? How can we define correct or incorrect? I am thinking that instead of adding a binary classifier as the new output layer, maybe we can add a regression layer to predict how correct the prediction will be (ex. correct 1 out of 3, all correct, all incorrect, etc.)? Would this be too 'messy'?

2) You mention that Active Learning can be implemented with Text Generation too. I wonder if doing so could lead to a more consistent text generating model but less 'human-like' one? That is languages generated by humans are more than just being correct or incorrect, but also creativity, changes, flexibility, pragmatic, and such. I might not understand how Active Learning will be applied to Text Generation well enough. Could you elaborate on this?

[SoyBison]

I'm really interested in this concept of making a model predict it's own error. In a way this is an implementation of a "Boosting" method to increase accuracy, but with the added benefit that you have a human in the loop. Related to this I have two questions.

1. To what extent can we consider this technique as a way to construct a "Confidence Level" for a neural network?

2. More generally, how much of a performance increase have you seen in using these active transfer learning techniques?

[bakerwho]

Thanks for sharing your work! I love the mathematically principled approach to increasing the interpretability of neural networks.

My questions are largely about ways to interpret neural networks based on your 2x2 matrix of (un)known (un)knowns.

• For known unknowns - technically, don't we interpret the final layer of a classification DNN as an m-D array of the DNN's levels of 'certainty' that the input belongs to each of m classes? I'm a little confused then how to 'double up' the DNN's output into both, a prediction and a certainty. As in your example, is it sufficient to train an auxiliary layer to predict certainty on a validated set for which we have computed the accuracy of the DNN?
• For unknown knowns - how do we effectively guide our 'search' for new tasks that a pre-trained model can transfer onto? Are there any heuristics you have found to be useful?
• For unknown unknowns - epistemologically, it feels like we can only push these frontiers and never really exhaust the space. If this is true and our exploration will never truly be complete, is there still a way to measure how well we are doing?

Thanks again!

[bhargavvader]

Thanks a bunch Rob - this is super useful, and most ML is definitely a process which requires active human "intervention", it is great to have a text focusing on this. I especially liked the chapters on sampling and diversity sampling.

I have two not very technical questions for you:

1) What do you think needs to change now in ML pedagogy? You say that this text is the book you wish you had when you started out: what "bad habits" do you think we pick up when we approach ML from a more old fashioned way?

2) Prof. Evans has often talked about an "alien intelligence", or ML and AI algorithms which perform tasks in a non human way: they have the potential to discover things which humans might not, and largely because they navigate problem spaces in ways which humans won't. How do you think human guided ML builds into this? Do you think that with more human intervention in different stages of ML, we will just get better "human-like" results, but not explore the space of results we don't know we could possibly get? Or do you think that paradigm may not be as useful for the more industry oriented techniques in your book?

Excited to hear your talk!

[nwrim]

First, I just want to say that I love your writing! Phrases like

“Exploitation” and “Exploration”, which are clever names that alliterate and rhyme, but are not otherwise very transparent.

are awesome remarks. Also, I agree with @wanitchayap's remark that Active Learning is a fascinating concept. I agree that giving human annotations on what the model is confused about (uncertainty sampling) and on what the model did not see much before (diversity sampling) will be a very powerful tool in enhancing the performance of the model.

That being said, I could also imagine some scenario where this kind of active learning can detour - perhaps by some misfortune, the initial set of labels fed into the model was really biased so the trained model make a wrong guess very confidently? Supplementing a lot of randomly sampled annotations will eventually solve this issue, but I feel like diversity sampling + uncertainty sampling might not be able to correct for this hypothetical (admittingly a bit unrealistic) case. Is there an aspect of active learning that I did not catch that solve this kind of problem? If not, what could be a possible solution to account for this issue?

[zeyuxu1997]

Thanks for your presentation. The topic of human interaction with machine learning is exciting and meaningful. My question is whether it's possible to have a model that creates new labels automatically? You have discussed on how humans can select and create labeled dataset to train the model and show how the model can learn from interaction with humans, but that only helps to recognize known labels better. If the model can recognize new data points that are too different from existing data points to be classified as known labels, that will help people to recognize the change happening in the reality. Perhaps that's how machine learning can help to discover something which humans may not, as Prof. Evans said.

[liu431]

In Chp 8, you mention that there is a privacy issue of tracking the demographics of human experts. As people are increasingly aware of their data being tracked, what kinds of laws or regulations do you think is missing now for ensuring privacy in machine learning?

[policyglot]

Hi Rob, You got the highest votes in our poll for industry experts at the workshop. So it's great to have you join us at UChicago, and share how machine learning research applies to the real world. Just before reading your chapters, I was going through Settles (2009) work on the same topic. He mentions the 'Query by Disagreement' technique as one of the earliest algorithms within active learning. Where do you think this would along the continuum of human-assisted machine learning, vs machine-assisted human learning?

Settles, B. (2009). Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences.

[rkcatipon]

Thank you for sharing your work! This is yet another non-technical question but I would love to hear more about how you're applying Active Learning to disaster relief and response operations. How do you imagine Active Learning can improve the application of computational methods in this time of COVID-19, for example?

[anqi-hu]

Thank you for sharing your work! It is inspiring to learn about how humans can step in on and aid the machine learning processes. My question has to do with the ethical aspect of active learning/ active transfer learning. I'm unsure about the fairness/ unbiasedness of these models as they incorporate both regular machine learning models and human labeling before all items can be thrown into the training phase. Are there any ethical concerns due to the human interaction in these processes? That is, will the biases unintentionally introduced to the learning/ annotation phases be exacerbated in active learning? Are there mechanisms to address such issues?

[di-Tong]

This is super cool! I wonder if the active learning processes and performance vary between different types of data or different domains.

[linghui-wu]

I really like the idea of Active Learning and its application in NLP. Even though from my limited experience with contextual modeling, the situation cannot be more common that only a small fraction of the sample is annotated and can be utilized as a training set. And I also really appreciate the idea to create a user-friendly interface for annotation out of the respect of participants, which seems to be ignored in practice.

My question is about diversity sampling in text generation. In p208, a concrete implementation is to create fake-but-realistic messages about disease outbreaks in North America using the pre-trained model. I wonder will the correlation between the model that capture diseases and locations and the text generation models be adversely affected by the bias in our existing models? If so, how to deal with that?

[ShuyanHuang]

Thanks for presenting! In your book, human evaluation is involved in active learning, and you give examples of active learning in classification and clustering tasks. But what if the task is regression? How can we decide which items are "uncertain"? Is it possible to train the model with the same set of data, but with different random initializations, and look at the variance of the resulting predictions?

[vinsonyz]

Thanks for your presentation! I am curious that how active learning can be applied in social science research. What are the appropriate ways to choose the hidden layer?

[dongchengecon]

Thank a lot for your presentation! The interaction between humans and machine learning algorithms is quite an essential topic. It seems to be very dangerous in some case if there are collective mistakes from the training data which are not detected by the algorithm. And maybe one of the solutions is to adjust the human task in some way that could avoid introducing irrecoverable bias from the human side. For example, we may change some subjective tasks to more objective ones, and draw some linkages between the outcome to make them equally useful. Are there any specific examples about the switching process?

[YuxinNg]

Thanks for the presentation. Since you are the Founder & CEO of Machine Learning Consulting, I am wondering if you or your firm ever apply this active learning processes to the business. I am particularly interested in how it could benefit consulting projects. It would be nice to have a few real cases introduced. Thanks!

[goldengua]

Thanks for sharing your work. I was interested in the concept of entropy you introduced, and I want to relate this concept to the interpretability of machine learning. For example, if we use the (conditional) frequency of the words to calculate the entropy of a sequence of words, what does the entropy mean for this word sequence. Does the sentence with lower entropy have less 'information', and is easier to process?

[anuraag94]

Thanks for sharing your work with us!

In 2015-16, I involved in a startup whose product involved active learning to uncover actionable insights for VCs from human-tagged news feeds.

I think this application echoes some of the excellent comments brought up by @bhargavvader et. al. on alien ML, where it would be most effective to uncover those insights which humans (in this case, VCs) are not readily able to discover on their own.

Taking this is a concrete case study, what role could active machine learning play in implementing the sort of adversarial/alien ML that would make this a much more effective product for venture capital investing?

[SiyuanPengMike]

Thanks a lot for your fascinating book. Through your introduction, I have a deeper understanding of active learning and the science behind input data. Just a few days ago, the Turing Award winner Yoshua Bengio shared his newest finding of the attention model in ICLR. Through his research result, we could find that this kind of model works just like human attention and its ability of generalization is incredible. Considering that one of the main purposes of active learning is also generalization, do you have any comments about the attention model? Do you also believe that AI could form its own consciousness in the near future?

[chiayunc]

Thank you very much for sharing your work. The concept of HITL really fascinates me. I feel like in this kind of approach, teaching a machine is not that much different than teaching a person. We label the data (create the learning material), tune the model (help the learner perform better) and evaluate performance (have learners correct their mistakes). This is very understandable if the goal is to have the learner, at a high level, to copy or mimic the teacher, to perform as the teacher would. However, can this type of approach, then, be applied to problems where the teachers don't have an answer? For example, finding inferences or latent relationships. Could this approach be potentially dangerous?

Also, I just need a quick clarification on the understanding of active learning and HITL. My apologies if I missed it in the book. Do you consider them the same, or there lies some difference that calls for the differentiation in the field? for example, one term is broader or there is fundamental difference in paradigm.

[Yilun0221]

Thanks for the presentaion! My question is similar to @YuxinNg. AI consultants will certainly have many advantages like strong self-learning abilities. One key technology to boost AI in consultants, from my perspective, is on NLP. My question is, in the consulting industry, besides data collection and analysis, what kind of work can be replaced by AI? Which ones cannot be replaced by AI?

[RuoyunTan]

As many of us work in the social sciences and we constantly deal with human feedback, I wonder whether there are certain features of human review or human response that may influence how we sample. Are we making any assumptions about human feedback, such as consistency, when we design algorithms of active learning?

[PAHADRIANUS]

Thank you for your participation in our workshop! As you made very clear in your book, selecting proper people and managing them to annotate the data is a crucial step in the Active Learning process. When it comes to managing other people, though, potential problems could occur. You have listed the mainstream methods of recruiting annotators or outsourcing the job. My question is that: how do you determine the appropriate amount of workforce to strike a balance between itsrepresentativeness and unbiasedness and its potential huge cost?

[dhruvalb]

Thank you for sharing your work - I am looking forward to the talk tomorrow! This is general question - in the chapter you mention that "there can be two distinct goals in Human-in-the-Loop Machine Learning: making a Machine Learning application more accurate with human input, and improving a human task with the aid of Machine Learning." Focusing on the latter, what mindset, skills and knowledge would programmers/businesses need to improve human tasks without causing unintended harmful consequences? I would think hiring computational social scientists would be a great start :) However, are there any practical tools or training or industry practices that enhance programmers perspective on humans and human systems?

[tianyueniu]

Thank you for your presentation! My question is about uncertainty and diversity sampling. In particular, I find all types of sampling methods stated valid. How should we generally choose which method to use? Should we try all of them, and select the one that eventually results in the most improvement in our model?

[YanjieZhou]

Thanks very much for your presentation! I find this idea of Active Learning very interesting, and what I am wondering is about the output layer in the model of sampling. It can screen the output but how in general does it define some output as correct and others as incorrect?

[ShanglunLi]

Thank you for providing such an interesting paper to read! My question is that depending on the data and research domain, will the performance of active learning processes change?

[chun-hu]

Thanks for sharing! Your book mainly gives examples of active learning in classification and clustering tasks. Like other students, I'm wondering how they can apply to regression tasks? And how can we determine the threshold/boundary?

[ZhouXing19]

Thanks for the presentation. I can not agree more that the human annotation imposes limit to supervised learning, and the importance of sampling and diversity of training set is essential for the learning process. I would like to ask more about the relationship of active learning and reinforcement learning, as RL also stress the exploration of data instead of exploitation. Thanks!

[lulululugagaga]

Welcome and thank you for your sharing! My question is similar to @zeyuxu as in my research experience, a lot of the work is in labelling. Do you think the process of labelling, either picture or text labelling, can be automatically simplified by machine? That would really save time.

[Leahjl]

Thank you in advance for your presentation! The idea of Active Learning and its application is really fascinating. My question is that the active learning process involves human interactions, is there any bias during sampling?

[romanticmonkey]

Thank you for your presentation! The book you’ve written is very interesting. I just wonder if there’s a good way to educate the public, who’s with little knowledge about ML, about active learning.

[bjcliang-uchi]

Thank for the presentation! I am wondering when we should just use unsupervised ML and in which case human annotations are most necessary.

[WMhYang]

Thank you very much for this fascinating topic. Human participation becomes really important when we want to evaluate our ml models and improve its accuracy. However, it is also quite expansive. I wonder what is the common proportion of human coded samples in real applications to make our models reliable?

[minminfly68]

Thanks for presenting this interesting topic. I would really appreciate if you can share with us some real cases in your own business when applying the ML technique to consulting? Is there any area that (un)supervised ML and human annotation are really helpful?

[MegicLF]

Thank you for bringing out this interesting discussion about active learning! As social scientists, we may not know much about the concrete machine learning concepts. Then in a research project, people may just directly implementing the algorithm without thinking over the "unknown unknowns" and "known unknowns". My question is, is there any standardized procedure you suggest to follow in a project to keep important data while hitting corner cases? Is there any way that we do not need to really learn a lot of machine learning concepts but still use them correctly?

[hihowme]

Thanks a lot for your presentation in advance! I am wondering how would those unsupervised machine learning methods would be applied to a more general perspectives in other business areas?

[ydeng117]

Thanks for your presentation! The concept of active learning and human involvement in it is a very fascinating idea. My question is that how can it further advance to make classification on things that needs value or moral judgments? In these cases, different people might give distinctive and equivocal labels. For example, deciding a suspect is guilty or not can be very difficult or even confusing for an individual person. Is there any way to deal with such an issue?

[heathercchen]

Thank you very much in advance for your presentation! I am wondering what is the best timing for human intervention in machine learning?

[ruixili]

Thank you for sharing! This topic is so interesting! In the reading, there are many ways of sampling. I wonder when should we use which sampling strategy, are there specific guidelines for choosing strategies?

[ChivLiu]

Thank you for the presentation! I am very interested in active learning and cloud computing in education and tourism. I wonder that NLP could be applied to acknowledge students performance from cloud stored homework assignments if there is a solution provided. Therefore, it would be easier for it to grade filling-blank questions and multiple choices. However, I wonder for some open questions with no particular correct answer, could an active learning model train itself to find fair scores for all students?

[SixueLiu96]

Thanks a lot for your presentation! This book will be of importance to the machine learning and artificial intelligence. You mentioned a lot of human-computer interaction. Therefore, I wonder is computer work serving as a tool for human's goal, and human cannot be drived by computer, or human and computer should work together?

[fulinguo]

Thanks so much for your research. My question is whether we have strategies of choosing variables in machine learning algorithms. The number of variables that might be potentially included in a model may be infinite. Does the variable selection mainly depend on the dataset? Another question is that if we hope to use machine learning to analyze questions other than prediction, like causal relationships in social sciences, what the role of human in these analyses is.

[yutianlai]

Thanks so much for coming! I'm wondering how you guarantee sampling effectiveness since much human interaction is involved.

[harryx113]

Thanks for the good read! I was wondering if there is any good way to optimize the workflow of active learning and annotation? Now it seems that data selection and human labeling are the most time-consuming parts of improving ML algos, so coming up with time and cost efficient ways of dealing with these two stages is crucial.

[hanjiaxu]

Thanks for the presentation! I am wondering if you could give us some examples of how to use these algorithms to approach actual social science problems? In light of the COVID-19 situation, I am wondering whether it is possible to use this approach to not only distinguish disaster-related and disaster-unrelated messages but also take "misinformation" into consideration? Thank you!

[sunying2018]

Thanks for your presentation! I'm interested in the concept of transfer learning. I have a question about the interpretability of the new output layer. As an example in the medium blog, trained the new labeled data (W, X, Y, Z) instead of previous labels (A, B, C, D). How should these two sets of labels connected and how to interpret the result using the new labels?

[jtschoi]

Thank you for your presentation in advance! Having thought erroneously that your book was going to be a heavy read, I was pleasantly surprised to see that much of the material was easy to interpret. I just have a question about machine learning's involvement in making causal claims, whether you yourself are involved in that field, and how might HuML play a role in that area. Thank you once again!

[lyl010]

Thank you for your presentation! I am interested in the definition of uncertainty felt by the machine: what does the lack of information means when we are training a model to learn something from data? Does it mean we do not have so much data or the data provided is not sufficient for making a clear prediction? But what if the data is the truth that a prediction is really hard to make and will our machine learning plan to consider dealing with this kind of uncertainty? Thank you!

[nt546]

Thank you for your talk! I have a question on how to effectively navigate the three areas in the knowledge quadrant with Transfer Learning, Diversity Sampling, and Uncertainty sampling. How should one prioritize between them and measure progress discovering the unknowns?