6. Large Language Models (LLMs) to Predict and Simulate Language - challenge

[lkcao]

Post your response to our challenge questions.

Think about how Large Language Models (LLMs) may assist with your planned final project. For example, they could help in labeling data, searching for relevant papers, or identifying bugs in your code. These are general examples. Reflect in detail on how LLMs may specifically benefit your project. Additionally, compare them to other candidate methods (like other machine learning classifier, NLP package, etc.), listing the unique strengths of LLMs (you can also mention disadvantages). If you intend to use LLMs as research subjects rather than as tools in your final project, feel free to share your thoughts on this as well! Feel free to reach out to James or the TAs for assistance if you plan to leverage LLMs for your research but haven't yet developed a complete plan. If you believe no parts of your final project involve LLMs, take some time to envision a scenario where LLMs could be used for research.

Also remember to vote 5+ your classmates' posts!

[XiaotongCui]

Large Language Models (LLMs) present a valuable asset for my final project in various aspects. One immediate and crucial advantage lies in their ability to assist with debugging code, offering a nuanced understanding of errors and potential improvements. This feature is exceptionally beneficial, saving time and resources in the development phase.

Furthermore, LLMs could enhance our project by delving into the realm of dating app bios. They have the potential to extract relevant labels or keywords, providing valuable insights into user preferences and trends. Although this application is promising, it's worth noting that our plans involving LLMs are still in the early stages and lack a fully formed structure.

[bucketteOfIvy]

Fine-tuned LLMs could help save me some time on my research project for data classification. In particular, I have a lot of data that is a bit of a congitohazard to read -- as one would expect 4chan to be -- and am unlikely to find volunteers to help me hand-label the data prior to conducting analyses. It's possible that I could use an LLM to help me with my data labeling efforts. Assuming the labels, when validated, seem sufficiently reliable, it's possible that LLM usage could let me scale up my analysis much more quickly than if I had to hand label my data for multiple identity groups.

Nonetheless, one of the main limitations is going to be domain specific jargon. For instance, if a /lgbt/ post contained the abbreviation "IWNBAW," that would be a pretty clear indicator to someone reading the post that the user is likely trans and posting about dysphoria. However, it's not clear that LLMs are going to be aware of the meaning of that acronym, which could lead them to misclassify such posts as "not posts about trans people."

Alternatively, I can probably use ChatGPT specifically in the very practical pursuit of getting a quick explanation of a (sufficiently old) method or package that I can want to utilize when writing my code.

[sborislo]

I will certainly be using ChatGPT to resolve bugs in my code as well as provide a baseline for running analyses I am not familiar with (such as multinomial logistic classification). LLMs like ChatGPT are often imperfect, but much more efficient than scraping the internet for coding solutions. I also believe the high-dimensional text analysis provided by LLMs could very well perform better than machine learning classifiers in learning and identifying the genres associated with lists of reviews. At minimum, I know LLMs can perform sentiment analysis a lot better, assuming they are pre-trained.

I've also typically found ChatGPT to be good for providing suggestions about why analyses might have turned out surprisingly. Although the explanations provided are often somewhat useless, there are occasionally some explanations that trigger new hypotheses (ones that I wouldn't have thought of!).

I do think LLMs might be a good amount slower at tasks like classification, however, even if they are more accurate. So it'll be a tradeoff.

[yuzhouw313]

Given the primary focus of my final research on YouTube comment classification and sentiment analysis, I believe LLMs will play an integral role, especially considering that my initial experiments with traditional machine learning models from last week's homework haven't yielded satisfactory results. Also, BERT's emotion detection classifier demonstrates superior performance in sequence classification, offering a more nuanced understanding of sentiment categories compared to the more basic analysis capabilities of NLTK's sentiment analysis package. I will probably continue to use LLMs (ChatGPT) to help me understand the lecture code provided by Jame, as well as debugging for my final project.

[YucanLei]

Definitely GPT can be used for assistance in coding. However, that is most likely the only advantage I will find for it. Perhaps GPT could help in labeling but I am unsure about it. I will also consider more about using GPT to help me with idea generations.

[volt-1]

In my project, I'm dealing with user text that often contains internet/regional slang and typos. LLMs like GPT are ideal for this challenge. They excel in understanding and processing non-standard language forms, making them superior to traditional machine learning classifiers or standard NLP packages, which might struggle with such irregularities. LLMs can efficiently handle tokenization and stemming without compromising the text's structure. Additionally, they offer invaluable assistance in understanding and even modifying my code, which is crucial for debugging and optimization. However, the higher computational resources it requires, and its less transparent decision-making process compared to more traditional methods.

[Marugannwg]

I'm currently tinkering with a GPT agent to help me clean my text data. My corpora consist of informal speech transcriptions and slangs, which can hardly be cleaned, normalized, or lemmatized well with traditional LLM methods --- most of the default functions in (scipy and our lucem_illud package) would either remove too many words or fail to identify some transformed vocabulary and pause/casual terms (e.g. y’know, uhm, um...) AgentGPT function can handle this much better under a semi-supervised sense: I simply give it a few examples of what to retain and clean, and I'm currently trying to implement in my pipeline instead of just use it in playground.

[yunfeiavawang]

For my project, LLMs could serve as invaluable assets, not only in data annotation but also in expediting literature review and debugging code. Specifically, their ability to understand and generate human-like text can streamline the process of labeling data, which is often labor-intensive and subjective. This capability is distinct from traditional methods like manual annotation or simpler NLP tools, which may not offer the same level of insight or efficiency.

[donatellafelice]

LLMs have an immense possibility for my final project. Aside from my obvious reliance on them for coding, explaining math i do not understand, troubleshooting and generally thinking out when I get stuck, they can also provide insight into conversation specifically. In order to design my pilot study, I already used an LLM to create example text for people to code, as I was curious what differences ChatGPT might produce when feeding it my different conditions (debate, dialouge, neutral conversation). As Marugannwg mentioned, I can use it to clean my data without removing too many words or reordering based on different splits of the data.

[ethanjkoz]

I definitely see the use in LLMs like Chat GPT 3.5 being used for debugging code. It has helped me on previous homeworks and I definitely see how it can be a useful tool for understanding errors. Particularly when I am trying to figure out controlling data types for input and outputs from functions. Chat GPT also helps me understand and better explain my own code or results that I find. I can ask it questions about the differences in evaluation metrics like recall and precision and gain clarity with ease. However, I do see a few disadvantages for using LLMs in some research topics, where we would like to study things like suicide, racism, etc. Trying to conduct research exploring racism or suicide through an LLM like ChatGPT which has been trained and sanitized would not be fruitful. For my own project, I can see how LLMs would be good for classifying my documents for sentiment.

[cty20010831]

I think one interesting way to incorporate LLM in my final project is to ask it to classify field of study from the abstract. One of the reasons why it is superior than traditional ML classification algorithms is that it can elaborate on its "reasoning" of why it belongs to the specific field of study (rather than just some numbers from ML classification algorithms). These "reasoning" text may shed light on the "contextual" meaning characterizing a specific field of study when scaling up to a larger level.

[naivetoad]

Large Language Models could be useful for my final project analyzing the impact of National Science Foundation funding on research output. LLMs can go beyond keyword searches to understand context. For comparing abstracts before and after funding, LLMs can also perform deep semantic analysis to highlight shifts in research focus or methodology. In contrast, traditional classifiers require extensive feature engineering and labeled data for training. NLP Packages are powerful for specific tasks like tokenization or part-of-speech tagging.

[runlinw0525]

For my final project, I will use GPT-4 to label data. Since I am analyzing attitudes towards AI and generative AI in over 2000 course syllabi, I couldn't think of a better way to label each syllabus than using LLMs like GPT-4. I will parse each syllabus into sentences, detect sentences that contain AI-related words, combine them, and then send them to GPT-4 with a suitable prompt for labeling. However, using GPT models through API calls still has limitations, such as token limits for input and output, and inconsistent results. Therefore, to ensure accurate labeling for further text analysis, I would manually label each syllabus, even though the workload could be significant.

[joylin0209]

In my research, Large Language Models (LLMs) will provide valuable assistance during the web scraping and data collection phase. Firstly, they can help me more efficiently extract and filter posts that match the key term (e.g., "Tai-Nu台女") using natural language processing techniques, which will significantly expedite the process of gathering data. LLMs can also further assist me in categorizing these posts based on themes or sentiments, helping me gain a deeper understanding of their content while determining which posts are most relevant to my research question. This classification and filtering process will save me a significant amount of time and human resources while ensuring that my dataset maintains high quality and relevance.

[beilrz]

My final project involving analyze a news headline dataset I collected, without around 250,000 unique headlines. I am specially interested in analyze the political bias in the headlines, so I was thinking maybe I could utilize LLM to label political figures and organization involved in the headline, based on the existing knowledge of LLM. Comparing to other NLP methods, LLM are more capable at processing unstructured tasks that involve contextual information. I am also consider the method from this week's reading, where the authors utilized a fine-tuned BERT for extracting information from the dataset.

[Caojie2001]

I think that LLMs would not be my final research subject. However, LLMs could be beneficial for the overall working process from various different perspectives. Firstly, I think that LLMs are helpful when coding. In most cases, they can detect the errors in my code, and they are also helpful in creating simple helper functions. Secondly, they help me clarify and organize my research ideas and strategies in the early stage. Finally, LLMs are also useful in polishing research papers or reports.

[anzhichen1999]

I think LLM would not benefit my code a lot. But it can be helpful in helping label texts to create training sets.

[h-karyn]

In our dating app dataset, we have religion and essay answers to various prompts. We are hoping to establish the predictive relationship from essays to religion.

In addition to traditional supervised learning, we will be exploring few shot inference and see if LLM is better at predicting individual's religion from the essays.

Dataset: The dataset: https://www.kaggle.com/datasets/andrewmvd/okcupid-profiles/data.

[QIXIN-ACT]

Utilizing LLMs in my final project could significantly enhance coding tasks and introduce innovative approaches to content labeling and summarization. LLMs offer a unique blend of adaptability and natural language processing capabilities, setting them apart from traditional machine learning classifiers and NLP packages. They can quickly generate code, identify bugs, and creatively analyze data, offering fresh perspectives. However, the potential for biases and inaccuracies in LLM outputs necessitates careful human verification - it would be a challenge for me.

[Twilight233333]

LLM is very useful for my research, it can help me identify the president's attitude towards Mexico in my database more quickly. I can combine cluster analysis and dynamic topic modeling to analyze the relationship between the president's attitude and assistance, as well as the relationship as a whole, so I will use it in the final study.

[HamsterradYC]

LLM, although not the direct subject of my project, has provided significant assistance in several aspects. Firstly, LLMs can greatly save time and cost in the binary labeling of emotional datasets. With their extensive pre-trained knowledge and learning capabilities, LLMs can understand complex texts and accurately label them based on this understanding. While the accuracy compared to human labeling still needs to be verified, using LLMs as an initial step in labeling can reduce the burden on human annotators and increase the efficiency of the labeling process. In terms of modifying and explaining code, compared to directly consulting Google and official documentation, LLMs can provide faster solutions and explanations.. Additionally, relying on LLMs' answers might lead to an overreliance on machine-generated content, neglecting the necessity of in-depth analysis and verification.

[chenyt16]

Large language models can help me fix bugs in my code and generate data. As generative models, they perform well in problems requiring extensive background knowledge. However, many large language models, like ChatGPT, cannot be retrained. Therefore, in cases requiring training, such as classification problems, traditional models may perform better.

[Dededon]

I believe LLMs can help me with tasks like generation of representation embeddings (using BERT-based models, longformer, or openAI apis for GPT2 or other models) for unsupervised learning, and summarizing, information retrieval and classification of the case background and legal questions in the legal documents I'm processing. The bottleneck for using the large language models in my case is the input token length: I have to make tradedoffs of segmenting the legal documents, as they are in average ~5000 tokens long. Also another concern in representation vector learning is loss of the context-richness in the legal documents.

[erikaz1]

I employed LLMs, specifically LLaMa and GPT-3.5, to aid in code labeling and validation. Echoing the findings of Vicinanza et al. (2022), I discovered that GPT's predictions exhibited increased noise levels, which I attributed to the prevalence of short sentences in my model. Nonetheless, I find LLMs potentially valuable for my project overall. Their ability to contextualize word embeddings facilitates the generation of predictions and responses, allowing me to assess their similarity to the writing style of my case study.

[floriatea]

• LLMs like GPT-4 or BERT can excel in grasping the context surrounding telehealth discussions, which is crucial for differentiating between discussions of telehealth as a technology versus its application in specific health contexts. This deep contextual comprehension surpasses most traditional NLP packages that rely heavily on keyword spotting and surface-level analysis. It can provide nuanced sentiment analysis, identifying not just positive or negative sentiments, but also capturing the intensity and subtleties of emotions expressed in telehealth discourse. This nuanced analysis is particularly beneficial for understanding public perception and attitudes towards telehealth over time or in response to specific events. It can be fine-tuned on a specific corpus of telehealth-related texts to become even more effective at understanding and generating insights specific to the telehealth domain. Beyond analysis, LLMs can generate summaries, pose questions, or even suggest research directions based on the corpus of telehealth texts.

Comparison can be made with traditional classifiers (e.g., SVM, Random Forest) that require manual feature engineering and are limited in their ability to understand context or nuanced sentiment, providing richer insights into telehealth discourse.

Strength:

• Smaller models like traditional LSTM networks may struggle with the vast context and complex semantics present in telehealth discourse. LLMs, with their significantly larger parameter space, are better equipped to handle these complexities. The ability to fine-tune LLMs on specific datasets means they can become increasingly effective at tasks directly relevant to telehealth, offering insights that are both deep and domain-specific.

Disadvantages:

• Deploying LLMs can be computationally expensive and may require significant resources, particularly for training or fine-tuning on large datasets. The "black box" nature of LLMs can make it challenging to understand how they arrive at specific conclusions.

[Brian-W00]

Our research question is about the sentiment differences between different types of communities on Reddit. We could use LLMs to do sentiment analysis on those texts. I think pre-trained models are way much better than basic machine learning models. The complex structure of LLMs allows them to learn more complicated text features. However, the mechanism of deep learning is more like a black box. The internal reason is hard to describe.

[Carolineyx]

Benefits

1. LLMs excel in understanding and generating nuanced human-like text. This makes them particularly suited for analyzing the psychological richness and interestingness of 'how they met' stories. Unlike simpler NLP tools that may rely on keyword spotting or sentiment analysis, LLMs can grasp the narrative's context, emotion, and subtleties, providing a deeper analysis of the story's elements that contribute to relationship longevity.

2. For our project, human raters initially labeled the stories as "interesting" or "not interesting." LLMs could assist in semi-automating this process by pre-labeling stories based on learned criteria, significantly reducing the human effort required and increasing the scalability of data processing. This is especially beneficial compared to traditional machine learning classifiers that require extensive feature engineering and pre-defined labels.

Comparison to Other Methods:

Versus Traditional Machine Learning Classifiers: Machine learning classifiers (e.g., SVM, Random Forest) require substantial feature engineering and are limited by the predefined nature of their input features. LLMs, conversely, automatically extract and learn from the data's inherent features, making them more flexible and powerful in handling complex narrative data.

Versus Standard NLP Packages: While standard NLP packages offer valuable tools for text processing and analysis (tokenization, part-of-speech tagging, etc.), they lack the advanced comprehension and generation capabilities of LLMs. LLMs can analyze text at a higher level of abstraction, identifying themes, sentiments, and narrative structures beyond the reach of conventional NLP methods.

Disadvantages of LLMs:

One of the key challenges with LLMs is their "black-box" nature, making it difficult to understand how they arrive at certain conclusions or predictions. This can pose challenges for researchers who need to explain their analytical process and findings.

Expansive with training the model