8. LLMs to Model Agents & Interactions - Challenge

[lkcao]

Post your response to our challenge questions.

First, describe a conversation explicit within, implicit from or underlying your data. This could be the interaction between posters on a social media platform, or comments and reactions on a discussion site, or back-and-forth in a parliamentary debate, or shared stance on an issue (e.g., a stock price, political perspective), or a shared style of speech or focus, or characters within a fanfiction universe, or concepts within a discourse, or constitutions sharing ideas and phrases. Second, state two hunches you have about patterns in this conversation, with an asterisk (*) after the one about which you are most certain, and a plus (+) after the one that, if true, would be the biggest or most important surprise to others (especially the research community to whom you might communicate it, if robustly supported). Third, describe the dataset from which you will construct or extract this conversation for exploration and analysis and note whether this data could be made available to class this week for evaluation (not required...but if you offer it, you might get some free work done!) If available, place (a) a link, (b) a script (to download and/or clean), (c) a reference to a class dataset, (d) or an invitation for a TA to contact you to get it. Fourth, list in numbered steps what you would do to construct/extract the conversation from this data (You can even think of using LLMs!). Please do NOT spend time/space explaining the analytical strategy through which you would explore your conversation and consider your hunches (Then upvote the 5 most interesting, relevant and challenging challenge responses from others).

[sborislo]

Conversation: Sometimes explicitly, and sometimes implicitly, reviewers of videogames on Steam "respond" to another person's review in their review. This is not a direct response like you'd see on social media in the form of comments, but you'll sometimes see people say, "I don't know what the other reviews are talking about when they say ___". It is also possible that reviewers are responding to prior ones implicitly; for instance, covering new information.

Predictions: Prediction 1*: Reviewers' responses to others' prior reviews are more often opposing in sentiment than complementary (i.e., a conversation is more likely to be between a positive and negative review than between two reviews of the same valence).

Prediction 2+: Later reviews are less diverse than earlier reviews (in this case, because later reviewers might feel like they have to cover new ground in response to prior reviews).

Dataset: For all intuitions, this dataset scraped from Steam. More games can be scraped in this manner, and the amount of reviews scraped should probably be reduced substantially when doing so. This dataset contains all reviews for any game on Steam that is specified, and can include time information too (to see the temporal flow of conversation).

[yuzhouw313]

1. Conversation In analyzing the dataset comprising comments and replies from COVID-19 related YouTube news videos in 2020, the majority of interactions in these sections are instances of individuals expressing their opinions in a standalone manner. Nevertheless, explicit interactions are observed when users actively engage with previous comments through replies, especially when they resonate with the expressed viewpoints. On the other hand, implicit interactions are identified through users 'liking' a comment, signifying agreement or support without articulating a separate response.
2. Predictions (1)* Echo chamber might manifest as clusters of comments that strongly agree with each other, reinforcing specific viewpoints while potentially dismissing or not engaging with dissenting opinions. Such polarization could be further amplified by the explicit interactions where individuals choose to reply to comments that align with their views and attack comments that contradict with their views. (2) + Potentially we may see threads of conversation where individuals with differing viewpoints engage in substantive discussions, share personal anecdotes to illustrate broader points, and even alter their perspectives based on the information exchanged.
3. Dataset I will use this dataset scraped from YouTube using its official API. However, I might need to re-scrape the comments to specify which comments are standalone and which are replies corresponding to existing comments.
4. Steps to construct the conversation (1) Re-scrape comments and indicate the relationship between standalone comments and their replies (2) Data preprocess and normalization (3) Identify Top-Level Comments and Replies (4) Conversation Reconstruction: For each top-level comment, gather all associated replies to reconstruct the conversation threads and sort replies by timestamp or using reply-to identifiers

[joylin0209]

Conversation Description: On Reddit, discussions about a specific topic are usually made up of replies and responses between different users. For example, if we want to study discussions about artificial intelligence, we can collect posts and replies under that topic. While these replies do not directly respond to a specific user's comment, they often reflect points made in previous posts or counter or supplement points made in previous discussions.

Predictions: 1* In discussions on Reddit, we may find more opposing views than mutually consistent views. 2+ Over time, we may observe the content of discussions becoming more in-depth and specialized, reflecting the continued evolution of the topic and increased user understanding of it.

Dataset: I don't have a database of data that has been scraped, but the Reddit API provides a public interface that allows developers to access posts and replies on different topics.

Steps to Construct/Extract the Conversation: Use the Reddit API or a web crawler to collect data on posts and replies on specific topics. Preprocess data, such as removing HTML tags, special characters and stop words. Use natural language processing techniques, such as sentiment analysis or keyword extraction, to identify responses or additions to previous discussions in posts and replies. Structure conversations by grouping replies based on their content and relevance. This can be done by tracking quotes or shared ideas in replies. Analyze opinions and emotions in conversations to identify opposing or congruent trends. Conversations can be further serialized over time to observe changes and trends over time. Use visualization tools to present analysis results to better understand conversation dynamics and trends.

[Marugannwg]

Conversation: Well, my dataset is the fictional conversation around different characters in games and fan fiction. Everything about the fictional world is embedded in the dialogue format, and I believe there should be some pattern when different archetypical characters are put on the same stage and talk.

Predictions:

1. *The conflict resolution pattern among different character archetypes is different and observable. (some conflicts can be resolved in talk, while other (e.g. between a hero-like character and a shadow, hard antagonist) would only resolve when an external factor, or a third person is introduced)
2. +The language and vocabulary used in dialogue might become more diverse and esoteric as time goes by. Or probably some character's use of language (e.g., vocabulary, slang, politeness strategies, neologisms) might be more contagious?
3. About LLM agents: Upon creating agents with the character's respective dialogues (or personality descriptions), the fictional agents would either reach an agreement quickly or fail to stick to a given topic in a simulation.

Dataset: Full dialogues from game/fanfiction dramas. -- already some kind of fictional conversations.

[YucanLei]

Conversation: Sometimes explicitly, and sometimes implicitly, reviewers of videogames on Steam "respond" to another person's review in their review. This is not a direct response like you'd see on social media in the form of comments, but you'll sometimes see people say, "I don't know what the other reviews are talking about when they say ___". It is also possible that reviewers are responding to prior ones implicitly; for instance, covering new information.

Predictions:

Prediction 1: Later reviews are less diverse than earlier reviews (in this case, because later reviewers might feel like they have to cover new ground in response to prior reviews).

Prediction 2: Based on prediction 1, later reviews will likely contain repetitions from the early reviews, and most likely, these repetitive reviews are the less technical or serious reviews, and likely memable.

Dataset: For all intuitions, this dataset scraped from Steam. More games can be scraped in this manner, and the amount of reviews scraped should probably be reduced substantially when doing so. This dataset contains all reviews for any game on Steam that is specified, and can include time information too (to see the temporal flow of conversation).

[bucketteOfIvy]

Conversation: Threads from the /lgbt/ board of 4chan are inherently conversational, with users often replying to one another explicitly or implicitly.

Hunches:

1. Conversations between users with known tripcodes will feature references to those tripcodes and characteristics of the individual, as they slowly build up a persona on the website. (+).
2. /lgbt/ conversations will feature heavy intermixing between users with large ideological differences, with both individuals often replying to a single user (*).

Dataset: A few weeks worth of /lgbt/ posts scraped every 30 minutes. Findable on my repo for this course, here.

Conversation Extraction: Most conversational messages on 4chan are "low hanging fruit" that indicate directly which message that are in response to with a tag in their text content. I've already built a codebase that uses this tag to build a network of message responses. However, some messages do not indicate what they are in direct response to -- these messages could be cleaned as follows:

1. Split up my corpus by individual threads.
2. Randomly select a few threads to (by hand) build out conversations on. This will be a validation set.
3. Split posts in a thread into 3-grams (i.e. series of 3 adjacent posts).
4. Append the initial message of the thread to the 3-gram, creating a ""4""-gram.*
5. Hand ChatGPT (or you're favorite LLM) each ""4""-gram, and ask it if any of the messages in the ""4""-gram are in response to one another. If yes, tag the messages as conversational. If no, tag them as not conversational.
6. Validate against the validation set from 2.

*I'm calling this a ""4""-gram as, while it consists of 4 messages, the 4 messages are not entirely adjacent).

[chanteriam]

Conversation My dataset involves Supreme Court arguments that are explicitly siting previous legislation either in accordance or dissent to it. For example, the 2022 Dobbs decision explicit converses with the 1973 Roe decision in order to reject it's conclusions.

Hunches:

1. *SCOTUS opinions will implicitly be conversing with events happening in the United States, particularly events and moral arguments communicated by popular and social media platforms.
2. +Conversations with previous legislation will reveal not only the opinions of the speaker in regard to that legislation, but also largely in regard to the decider of that piece of legislation. This is particularly interesting given the increased polarization in the United States, and the use of more inflammatory language toward individuals on the opposite political side than oneself. This will reveal itself in more short or terse sentences in response to conclusions made in the conversing opinion.

Dataset(s) Ideally, the first hunch will be explored by analyzing all abortion-related pieces of media directly preceding a SCOTUS decision, though that would require an extensive amount of data and computational power. The second hunch can be examined within a SCOTUS opinion using sentiment analysis and other sentiment-detecting NLP methods when references to other legislation are made.

[ethanjkoz]

Due to the nature of the organization of reddit posts and comments, there is an explicit conversation within each post and comment pair. The original post is the start of a conversation and is sometimes even a continuation or reaction towards a conversation from another subreddit (in my limited experience going through the subreddits r/Adopted users sometimes react to posts from r/Adoption. There are also implicit conversations over time with trends in some discussions or based off of previous post history of users.

My two hunches: *adoptees who share negative sentiment on their posts will share this sentiment in the comment section. In pro adoptee subreddits, comments that share this sentiment will generally receive favorable scores. In more general adoption spaces, more conversation will be directed towards diverse opinions

• disagreement in comment sections will be more vitriolic (more negative scores and more extremely positive or negative) within posts showing negative sentiment towards adoption.

my data: the same as before, I would have to format it such that posts and associated comments are linked together because as they stand now they are not linked. For the data I do have this would not be entirely difficult but would be time consuming.

[QIXIN-ACT]

The conversation within your fan-fiction database revolves around the creative narratives crafted by authors and the interactions among characters within stories. This multi-layered dialogue reflects interpretations of original works, shared themes, and collective stylistic preferences within the fanfiction community.

Two Hunches About Patterns:

• A collective stylistic preference or thematic focus across fanfictions related to certain characters or events. This indicates a shared community narrative that diverges from the original works.
• The formation of sub-communities prioritizing specific genres or themes (e.g., romance vs. adventure; original backgroung vs. AU), which could reveal unexpected engagement patterns within fan culture.

Data Sources: Same as before, the AO3 fanfiction dataset, encompassing metadata for fanfictions and statistics on reader feedback.

Steps to Construct/Extract the Conversation

• Text Analysis: Apply NLP (and LLM!) for sentiment analysis and thematic exploration.
• Comparison: Contrast fanfiction content with original material to identify divergences (with LLM?)
• Visualization: Create visual representations of findings.

[donatellafelice]

My data set is made up of conversations between pairs. Most revolve around a disagreement on an issue both parties have rated as important to them. I believe that we may see open ended questions at the beginning of a conversations specifically regarding the topic more prevalent in the dialogue condition. * I also think that we will see mirroring as we did in the echoes of power article if one person exhibits a very aggressive or a very positive way of speaking. + I have already cleaned the transcripts into a series of different turn level, speaker level and conversation level documents. My dataset is available for the class, if we would work on it the TA should contact me and I can request from Booth.

[erikaz1]

Personal journals could be thought of as one-way conversations. Perhaps more in line with this task would be interview data, where there is a back-and-forth quality between multiple speakers who are discussing a particular person’s life, legacy, character, etc.

Some hunches about the patterns in these conversations include:

• Interview questions always center around a theme (or have some kind of direction), a theme that is triggered by current events, public achievement, etc. (*)
• Interviewees offer more value and garner more surprise when they slightly diverge from the interview questions. (Though perhaps this is simply a question about the nature or structure of interview as a whole.) (+)

The data will be difficult to collect at scale. They would have to come from old newspapers and other scattered instances of conversations.

[chenyt16]

Conversation Comments and replies under abortion-related news

Hunches

• Conversations may be more polarized if the news article has strong tendentiousness (*）
• If the conversation goes on for many rounds, the content of the conversation will deviate from discussing viewpoints and turn towards more extreme personal attacks.

Dataset: I didn't include these comments in my dataset. But they can also be scrapped from these new sites.

[naivetoad]

Conversation: The underlying conversation in my data can be conceptualized as the evolution of collaboration patterns within the academic community before and after receiving funding.

Hunches: Researchers tend to collaborate with a broader network of co-authors after receiving funding.* The citation impact of papers published after receiving funding does not increase significantly compared to those published before.+

Data: The dataset consists of metadata from academic papers authored by researchers who have received funding.

Procedure

1. Clean and preprocess the text data from abstracts to facilitate analysis.
2. Construct co-authorship networks for the periods before and after funding.
3. Compare the citation counts of papers published before and after funding.

[runlinw0525]

Conversation: Certain departments may tend to support the use of (generative) AI while others not.

Hunches: * Departments with a more technical or scientific focus, such as Computer Science or Mechanic Engineering, are more likely to have course syllabi that integrate and support the usage of AI, given the disciplines' more abstract nature. + Less technical departments, such as Linguistics or History, may take an opposing stance or not mention AI at all, reflecting an emphasis on self-understanding and less direct engagement with AI technology.

Dataset: Course syllabi (published in or after 2023) scraped from the largest syllabi archive of the University of Michigan.

Steps: Use LLMs like GPT-4 to perform labeling tasks -> Conduct a comparative analysis -> Use chi-squared tests and logistic regression for prediction

[h-karyn]

Conversation: The conversation in this context occurs at poster sessions during academic conferences. The explicit conversations include direct interactions between poster presenters and attendees, involving questions, clarifications, and discussions about the research. Implicitly, the conversation extends to the comparison of research methodologies, the identification of emerging trends and gaps in the field, and networking opportunities among participants. An underlying conversation might revolve around the competition for recognition, funding opportunities, and collaborations, as well as the validation of novel ideas and approaches within the academic community.

Hunches *Participants tend to cluster around posters that align with current 'hot topics' in their field, indicating a possible herd mentality or echo chamber effect in academic interests and discussions. +The language and jargon used in these conversations might subtly signal the inclusion or exclusion of certain groups within the academic community, potentially impacting diversity and interdisciplinary collaboration.

Dataset Description The dataset for this exploration could be constructed from a collection of audio recordings, transcriptions, and observational notes taken during these poster sessions. Additional data could include the posters themselves, any distributed materials, and possibly pre- and post-session surveys or interviews with participants about their experiences and perceptions. Given the need for consent and potential privacy concerns, the availability of this data for class evaluation would depend on the ethical guidelines adhered to during collection, including anonymization and participant consent.

[Twilight233333]

My research deals not directly with conversations but with presidential press conferences that produce conversations with reporters.

It's a pattern where the reporter starts the conversation with a question about a particular policy or the president's behavior. (*) On the other hand, the president's answer usually does not contain an apparent attitude but instead provides background and reasons.(+)

The dataset was derived from the UCSB Presidential Database

[anzhichen1999]

Conversation: In the People's Daily, a prominent newspaper in China, editors often write comments or editorials about various speeches, events, or discussions happening on the internet. This represents an intriguing conversation between the official media and the digital public sphere. While not a direct interaction, these editorials often reflect, counter, or shape public opinion and discourse found on social media and internet forums. For example, an editorial might directly address themes or sentiments appearing in popular online discussions, indicating a form of indirect dialogue between the state media and the netizens.

Predictions: Prediction 1*: Editorials in the People's Daily tend to subtly shift the focus of online discussions, steering public opinion in a more state-aligned direction. This indicates a form of indirect influence where the state media doesn't directly contradict public opinion but gently guides it.

Prediction 2+: The tone and focus of the People's Daily editorials change significantly during periods of national or international crisis, perhaps reflecting a more direct and assertive approach to managing online discourse.

Dataset: The dataset consists of a collection of editorials from the People's Daily, spanning a range of dates and covering various topics. These editorials are paired with corresponding popular discussions from major Chinese social media platforms and internet forums that occurred around the same time. This dataset allows for an analysis of the interplay between state media narratives and public discourse. The dataset includes metadata such as publication dates, topics, and the number of comments or reactions on social media platforms.

[HamsterradYC]

Conversation Description: The subreddit dataset encompasses posts from both burnout and non-burnout communities, capturing diverse reactions to expressions of burnout. An underlying conversation is the shifting nature of responses to these expressions, particularly before and after significant events. This shift suggests a changing community dynamic, from one possibly characterized by negative feedback to a more supportive and understanding engagement. People are more tolerant of negative emotions because of empathy.

Predictions:

1. **There is a discernible shift in the community's response to burnout expressions post-major events, with a notable increase in supportive interactions.

2. The change in response tone from negative to supportive could significantly impact the social media discourse on mental health, marking a pivotal shift in public perception and support for burnout.

Dataset Description: This dataset includes posts and replies from both burnout-focused and general communities on Reddit and Weibo, segmented around major events like the COVID-19 pandemic.

[Caojie2001]

My research does not involve conversation. However, the interaction between editors of local newspapers and the central newspaper might be interesting.

Hunches: The editors of local newspapers may pay more attention to a certain topic when the central newspaper has focused on it for a certain period. When the publisher department of a local newspaper is involved in the topic, the previous response might be vague or even absent.

The dataset can be scraped from newspaper websites such as XinMin.

[XiaotongCui]

Conversation Description: The conversation implicit within the OkCupid dataset could revolve around the nuances of dating interactions influenced by wealth. This might include how individuals mention or respond to income-related topics, the use of language indicating status or financial stability, and the general tone of exchanges where wealth is a factor.

Hunches:

Individuals with higher reported incomes will use more positive and confident language in their messages. +Messages from individuals with lower reported incomes will show a greater use of humor and creativity in trying to attract potential partners. Dataset Description: The dataset would consist of anonymized user profiles and message histories from OkCupid, focusing on income levels as reported by users. Unfortunately, I don't have direct access to this data.

Steps to Extract the Conversation:

Download the dataset containing user profiles and message histories from OkCupid. Filter the dataset to include only profiles where income information is available. Extract messages exchanged between users, focusing on the content and language used. Create categories for different income levels (e.g., low, medium, high). Analyze the language used in messages for each income category, looking for patterns in sentiment, tone, and linguistic features. Compare the frequency of positive and confident language across different income groups. Look for instances of humor, creativity, or other unique language features in messages from lower income individuals. Consider the context of conversations where income is mentioned and how it influences the overall tone and style of communication. Summarize findings and draw conclusions about the relationship between wealth and language use in dating app messages.

[cty20010831]

Conversation: Conversations between participants advocating for various climate change policies on Reddit.

Hunches:

1. The more emotionally charged a post (e.g., expressing urgency about climate change impacts), the more engagement it receives (likes, comments). This hunch, marked with an asterisk, is based on the assumption that emotional content drives higher interaction rates.
2. A significant portion of the discourse might revolve around a few influential users who set the tone or topics of conversation, highlighting the potential power dynamics and influence certain individuals hold within seemingly democratic online communities.

Steps to Extract the Conversation: I will first scrape the data using Python Reddit API Wrapper. Afterwards, I will clean and preprocess the data, which might involve removing duplicates, filtering irrelevant posts, and anonymizing user information. Later, I will use LLMs to analyze sentiment, categorize posts by theme, or identify patterns in engagement based on content characteristics.

[ana-yurt]

Conversation: Zhihu responses to answers; comments to responses. Hunches:

1. Comments to an answer tend to be distributed in the space adjacent to the answer
2. The emotional polarity of the question affects both the answers and comments under the answers

Steps to Extract the Conversation: I will use BeautifulSoup to write a scraper to scrape all the comments relevant to my dataset. Then I will use LLM (or other modules) to extract emotional polarity.

[floriatea]

Conversation: My final project does not have explicit conversation. The conversation implicit within the telehealth data revolves around the evolving public and professional discourse on telehealth technologies and services. This includes how different stakeholders, ranging from healthcare providers and patients to policymakers and tech companies, discuss, perceive, and react to the development, implementation, and outcomes of telehealth solutions.

Hunches:

• The sentiment towards telehealth has become significantly more positive post-major global health events (e.g., the COVID-19 pandemic).This hunch is grounded in the assumption that the necessity of telehealth services during health crises has led to a greater appreciation of their value and potential.
• Discussions on telehealth from healthcare providers differ markedly in linguistic complexity and topic focus compared to those from the general public. If true, this would highlight a significant divide in the discourse on telehealth, suggesting that professionals and the public might prioritize different aspects of telehealth, possibly leading to misaligned expectations and communications about telehealth services.

Dataset: The dataset for exploring and analyzing this conversation is the NOW dataset. The dataset includes fields that are crucial for this analysis, such as text, date, country, website, and linguistic features extracted from the content.

Steps to Construct the Conversation:

1. Classify the filtered texts by stakeholder group (e.g., healthcare providers, patients, policymakers, tech companies) based on the website field and linguistic cues in the text.
2. Group the texts by the date field to analyze how the discourse changes over time, particularly before and after significant health events or policy announcements to analyze temporal trends.
3. Apply sentiment analysis to understand the general sentiment towards telehealth within each stakeholder group and how it evolves over time.
4. Use NLP tools to assess the linguistic complexity of discussions (e.g., readability scores, lexical diversity) and compare between stakeholder groups.
5. Employ LLMs or other topic modeling techniques to extract prevalent themes and topics from the discussions, identifying key areas of interest and concern for different stakeholders.
6. Compare the findings between different stakeholder groups and over time to identify any diverging or converging trends in the conversation on telehealth.

[beilrz]

Conversation: My dataset is about news headlines, so there may not be direct "conversations". However, news headlines represents news media 's position in the boarder discussion of news, and reflect the media's pre-existing political bias. Thus, how a news media criticize the other side of news, or comment on them could be seen as a form of conversation.

Predictions: Prediction 1*: News media hold negative opinion toward the other spectrum of political news bias .

Prediction 2+: News media do not directly engage on the critique of the other side, but instead focusing on praising their own stance.

Dataset: My dataset is scarped from the popular news media frontpage of the US, according to allsides.com. The frontpages are collected every 2 hours, since the new year.

[Brian-W00]

First, I am considering conversations in online forums where people share their experiences and advice about home gardening. This could include interactions about plant care, pest control, and gardening tools.

Second, my hunches are:

1. People are more likely to share their successful gardening stories than their failures to present themselves in a better light.
2. Unexpectedly, discussions about organic gardening methods might reveal a strong community bias against chemical pesticides, even if scientific evidence supports their controlled use.

Third, the dataset could come from posts and comments from a popular gardening forum or social media group. I can't provide a direct link now, but I can prepare a script to scrape the latest discussions from a public forum and clean the data for class analysis.

Fourth, steps to construct/extract the conversation:

1. Identify a popular online gardening forum with a public API or allow web scraping.
2. Write a script to scrape posts and comments about plant care and pest control.
3. Clean the data to remove irrelevant posts, spam, and personal information.
4. Organize the data by topic (e.g., plant types, gardening methods) for more straightforward analysis.

Then, I will upvote the most exciting challenge responses from others in the forum.

[Dededon]

Conversation: My project is not directly about conversations between LLMs. However, I can refer to JLM's idea of political-ethical space to test the AI's understanding of the social and legal context in my police misconduct case

Predictions: I can extract the case briefing, and motion part of my corpus and try to let AI predict the results as jury.

Dataset: I will use the CourtListener datset.

[Carolineyx]

Within the movie corpus of "how they met" movies, a rich tapestry of conversations unfolds as characters recount their stories of love and serendipity.

Hunches about Patterns: The dialogue surrounding the "how they met" moments may vary in tone and intensity, reflecting the unique personalities and circumstances of the characters involved. Characters who share unconventional or unexpected meeting stories may elicit stronger emotional responses from the audience, leading to increased engagement and resonance with the narrative.

Dataset Description: The dataset comprises a collection of movie scripts from romantic comedies and dramas, focusing specifically on scenes where characters recount how they met. Unfortunately, due to copyright restrictions, the movie corpus cannot be made publicly available. However, interested researchers or teaching assistants are welcome to contact me for further discussion on accessing and analyzing the dataset.