5. Machine Learning to Classify and Relate Meanings - orienting

[lkcao]

Post questions here for this week's oritenting readings:

Te’eni, D., Yahav, I., Zagalsky, A., Schwartz, D., Silverman, G., Cohen, D., Mann, Y., & Lewinsky, D. 2023. Reciprocal Human-Machine Learning: A Theory and an Instantiation for the Case of Message Classification. Management Science.

[yuzhouw313]

In this research paper, the corpora retrieved from the Darknet are prime subjects for weak classification, enhanced by Reciprocal Human-Machine Learning (RHML). Since such corpuses' probable hidden messages are often elusive to human analysis how can we prime the initial corpus when deciphering '"hidden answers" even given there are another 8 cycles of human-machine feedback exchange. The inclusion of tokens, special symbols, or characters, which might embody concealed messages but are conventionally disregarded as mere noise in human-led preprocessing, raises a pertinent question: Would it be more effective to initiate RHML with unsupervised machine learning, sidestepping the reliance on human-labeled ground truth? This approach seems particularly relevant when considering the application of N-grams or skip-grams in RHML, where the choice of window size becomes a significant challenge.

[XiaotongCui]

I find the idea of RHML very intriguing. However, I'm considering whether its applicability might be limited. After all, these models are typically trained by human experts and subsequently operate autonomously. Therefore, each time the model is trained, it requires the involvement of an expert in the training process. Could this potentially result in very high labor costs, making it challenging to widely adopt this approach?

[chanteriam]

The authors develop a theory of reciprocal learning between humans and machine learning algorithms. I wonder; does there exist theory about what data should, and should not, be used in these models? In general, is there any data that is off limits in this way, and/or should there be?

[sborislo]

I think there are cases in which reciprocal learning between humans and machine learning algorithms would help with prediction tasks (for instance, the human picking up on something the algorithm is doing that it shouldn't; or something the human is assuming that should not be assumed). However, I attended a talk a couple of weeks ago about large machine learning algorithms in marketing, and it was stated that the algorithms have advanced such that they are picking up on patterns that are meaningless to humans (but that clearly aid in prediction). Is there an extent to which reciprocal learning will progressively become less feasible as learning algorithms grow more advanced, due to their complexity? Should we balance interpretability with predictive power, or just focus on making more accurate predictions with algorithms?

[yueqil2]

Although this paper delineates the instantiation, Fusion, I still have several questions need to be clarified. When would be a point of saturation to stop the cycles of reciprocal learning? Is there any standard? When machine updates its classification models and enters a new cycle to classify new messages, does it nee to reclassify messages that were previously classified? How to ensure consistency and continuity of all message classifications?

[volt-1]

When the opinions of human experts represent a minority rather than the majority, but our ultimate concern is to ensure that the machine model maintains its generalization ability for the entire population (THE majority) after learning these expert opinions through reinforcement, does this mean that RHML are essentially setting a higher quality, more diverse data labeling benchmark to enable the machine to perform tasks better?

[Caojie2001]

The basic idea of RHML and the instantiation of Fusion is very interesting, as it enables the interaction of information between human researchers and machine learning, and it indeed improves the performance of ML models in the two given examples. I wonder if it would be possible to develop a public research platform supported by the scientific community that allows for the implementation of RHML for a wider range of ML subjects.

[bucketteOfIvy]

Te'eni et al. (2023) propose the following as their RHML loop:

It constitutes a continual reciprocal learning scenario for the context of text classification: (1) expert generates a conceptualization; (2) machine accepts feedback from the expert (elements of the conceptualization); (3) machine runs classification models; (4) machine returns feedback to the expert; and (5) expert sees the feedback and manually revises the conceptualization.

At a broad level, this feels really similar to exploratory data analysis (and possibly abductive approaches more broadly) in which it's advised (1) to create a hypothesis (conceptualization), then (2) translate that hypothesis into code so that it can (3) run some relevant models and (4) return the feedback to the user who (5) sees the feedback and manually revises their hypothesis. To what extent is it accurate to view the RHML loop as analogous to applying the EDA methodology to supervised classification tasks in ML? Does this trend apply to other human in the loop designs?

[ethanjkoz]

This paper seems to exemplify the point made early on in Text as Data that computational approaches do not replace humans, it augments them. However there seems to be a focus on augmentation of knowledge among domain experts. What happens when there is a significant mismatch between what a machine can learn from a human and what said human can learn from the machine? For example, what can a machine learn from an academic early on in their career or a collective of laypeople (wisdom of the crowd?)?

[chenyt16]

The "continual reciprocal learning scenario" (p.9) introduced in this paper sounds like a typical AI model optimization process that aims to improve the accuracy and efficiency of the model output. This process is quite normal in industrial scenarios, such as movie labeling and recommendation. The question is whether the conceptualization updated after receiving machine feedback has other application scenarios and can continuously benefit humans. If the human learning part is only model or dataset-specific and cannot be applied to other (real-life) scenarios, then the ultimate beneficiary is still the machine, and human learning is just one step of it.

[michplunkett]

Lots of industries utilize people to help in their recommendation, image tagging, etc. services. Those relationships though, often work as vehicles where they pass the psychological or manual labor burden down the SES ladder. How can we, going forward, make sure that this work is done in an ethical way that doesn't further compound the current economic disparities that are being experienced by those on the lower end of the tech power ladder?

[h-karyn]

On an high level, their methods remind me of reinforcement learning with human feedback (RLHF). How are they different? And which one should we use in different situations?

[Dededon]

This RHML framework is the basic framework of nowadays large language models. Speaking of LLMs, I wonder whether will we cover some contents of prompt engineering in this course? How can we use prompt engineering to let LLMs to clean and classify our data?

[cty20010831]

I think the idea of adding human voice in machine learning is indeed insightful and particularly nowadays with the growing development and application of AI. However, one question is about the feedback mechanism when applying this Reciprocal Human-Machine to fields where outcomes are not immediately observable or quantifiable (e.g., Organizational Behavior)?

[Marugannwg]

(This article reminds me endless hours of editing ChatGPT prompts and tinkering GPT agents to seek for desirable/consistent output through trial and error...)

The major theme I'm curious is the distinction between human "knowledge" and model "accuracy/performance" --- It sounds extremely challenging to have a complicated ML model and, at the same time, making decision in a sense-making way to common folks. As the paper suggests, it is a challenging but necessary task for human component in the research cycle to provide explainability to those black box models. There seems to be a lot of gray area here. e.g., if I just know that a particular part of input data would improve/sabotage the result, does it mean I can explain it? It almost feels like we are assigning reasons to the patterns discovered by a powerful machine.

[joylin0209]

Beyond classification accuracy, what other metrics or criteria were used to evaluate the success of the reciprocal human-machine learning system in improving both machine performance and human expertise? Were there any measures related to user satisfaction or the efficiency of human-machine collaboration?

[naivetoad]

What are the challenges and limitations in implementing this model in real-world scenarios, considering the necessity of expert human input and the complexity of machine learning algorithms?

[donatellafelice]

i was interested in how this was pitched kind of like a conversation - a kind of synergy between the two object (machine and human) wherein both parties are learning. Given that I am studying conversation and specifically looking at what might cause people to actually spontaneously exhibit linguistic markers of certain types of conversation, the Havruta concept seemed to have application in my context as well. I could not help but think how many applications this type of framework would have - imagine having a model that you could talk to over and over while you were learning a concept that then grew with you. I wondered, then, given the different sort of cognition mentioned in 3.2.1, how long would you need to be learning like this with the system in order for it to actually teach either party something?

(bonus this paper mentions the ethical implications of totally autonomous systems. i am wondering, are there any situations where you want a totally autonomous system that is learning, without human oversight? I am wondering what the simplified forms are that might be well suited for that?)

[Vindmn1234]

I'm curious about how scalable and generalizable is the RHML configuration presented in Fusion? Can this framework be easily adapted to other domains beyond cybersecurity, such as healthcare or finance, where text classification is also crucial?

[anzhichen1999]

Regarding what the author mentioned in the paper about "experts' judgment is subjective which threatens the validity of their decisions" and the issue of "duplication of human bias in ML algorithms and low interrater reliability when establishing ground truth," I'm curious how Fusion deals with the different opinions and biases that experts might have. When different experts see the same data differently, how does Fusion make sure this doesn't mess up the learning for the machine learning models?​

[runlinw0525]

I am wondering how does RHML specifically enhance the machine learning process compared to traditional machine learning models? And in what ways does it benefit human learners, especially in terms of their decision-making and problem-solving skills?

[QIXIN-ACT]

The concept of Reciprocal Human-Machine Learning is intriguing, yet it raises a question: Are there specific prerequisites for humans to learn from machines? It appears that roles such as Mechanical Turk Workers or individuals who categorize images may not actually learn from the machine, but rather, they "serve" the machine.

[Brian-W00]

How does the RHML configuration, as instantiated in Fusion and tested in cybersecurity contexts, adapt to and scale for different domains or datasets with vastly different characteristics, such as informal social media language, multilingual content, or non-textual data?

[erikaz1]

I have observations similar to BuckettetofIvy's above. The RHML loop as demonstrated through Fusion replicates a process that is similar to EDA, feature selection, model tuning, etc., that data scientists and researchers likely do on a regular basis. (One standout point here, though, is that of new perspective: to try and learn from computer logic as much as the computer is forced to learn from us.) However, Fusion has default parameters and makes some decisions regarding the model and collaborative learning process without human input. The interface for collaborative learning seems to limit what researchers are able to do. Do Interfaces such as these actually distance us from the nuances of model building, despite making some aspect easier to understand and use?

[ddlxdd]

I think the topic is interesting; it is surprising that we can involve humans in the cycle of the machine learning process. I am wondering if there is a specific advantage to involving humans in text message learning, or if this approach can be used in learning in other fields, like finance decision-making or other fields that humans may struggle with.

[YucanLei]

The paper focuses on two case studies in cybersecurity forums, but it would be beneficial to investigate the scalability of the proposed RHML configuration to larger datasets and different domains. How does the framework perform when applied to more extensive and diverse datasets?

[HamsterradYC]

The RHML framework proposed in this paper provides a theoretical basis for exploring the possibility of human-machine co-learning in different application fields. But article focuses more on the interaction and learning cycle between a single user. So Whether the RHML configuration support multi-user collaboration, such as having multiple domain experts on a team interacting with machine learning models at the same time? How can potential conflict and consistency issues be resolved?

[Twilight233333]

I may have missed a part of the article, but I was wondering how the reciprocity part of machine learning affects experts, and I only seem to know that experts affected AI in the previous cycle. It's more like an expert instructs and lets the AI do it on its own

[beilrz]

I think this is a very interesting research. However, one concern I have is the efficiency of training such model and the resource required. One advantage of deep learning and machine learning, comparing to previous models, is they require less human insights and assistance in the training process. By reintroducing human into the the system, it could overshadow the benefits of the machine learning.

[floriatea]

Considering RHML's application in classifying text messages within cybersecurity forums, what strategies could enhance RHML's capability to integrate and learn from highly unstructured data sources, including images, videos, and audio, in a cohesive manner?

[Carolineyx]

how can we ensure that the machine learning component of RHML adequately captures and learns from these nuances, particularly when dealing with slang, idioms, or culturally specific references that may not be easily quantifiable? Additionally, how does the reciprocal learning process adapt to evolving language use over time, ensuring that both human and machine learners continuously improve their understanding and classification accuracy?

[JessicaCaishanghai]

How does the reciprocal human-machine learning (RHML) configuration proposed for text message classification, particularly in cybersecurity contexts, ensure that the feedback loop enhances both the AI's accuracy and the human operator's decision-making capabilities over successive learning cycles?