[homework: doing, reading, watching] Linear, logistic regressions and embedding visualizations of Zulip data

Doing

[x] Clean Data Thinking Zulip chat data, located at https://github.com/onefact/datathinking.org-codespace/blob/main/data/datathinking.zulipchat.com/raw/messages-000001.json - put it in a polars dataframe and compute summary statistics of the dataset
[x] Analyze this Zulip chat data using logistic regression, linear regression, and embeddings with the tools we have learned in the lectures (don't forget to ask ChatGPT, Claude, Lex, GPT-4 for help as much as you need, and ask for help on the Data Thinking Zulip chat :)
[x] Create a visualization of logistic regression of the Data Thinking Zulip chat data
[x] Create a visualization of linear regression applied to the Data Thinking Zulip chat data
[x] Create a visualization of embeddings using the Data Thinking Zulip Chat data
[x] Make a copy of the Overleaf template: https://www.overleaf.com/read/ghpyzqwqwxpv (need to create an account and/or sign in if this is your first time using Overleaf). To make a copy, open the project after signing in using this link, and click on Menu, then Copy Project:
[x] In Overleaf, edit the template and figure out how to include a PDF figure in the report, alongside a brief description (a few sentences or paragraphs is fine!) of each of the analyses you performed, why you chose them, and the math equation for the linear regression, logistic regression, and embedding you used.
[x] Add the PDF of the report to this issue as a comment.
[x] Send a message on Zulip with a link to this comment, alongside the image representing your favorite visualization

Reviewing

[x] Review how Jaan got unstuck in the lecture recordings at https://panopto.ut.ee/Panopto/Pages/Sessions/List.aspx?folderID=43bb180c-79a6-4324-b055-afa400ecd1a0
[x] Review collaborative whiteboards from past classes: listed at https://www.datathinking.org/university-of-tartu
[x] Review Jupyter notebooks from past classes:

Reading

[x] [inspiration for how to categorize discussion data in json format with chatgpt] https://genmon.github.io/braggoscope/about & https://news.ycombinator.com/item?id=35073603
[x] [help with helping you learn to be a prompt engineer] https://github.com/dair-ai/Prompt-Engineering-Guide
[x] [help with storytelling] https://themarkup.org/hello-world/2023/02/04/journalistic-lessons-for-the-algorithmic-age
[x] [help with debugging] https://wizardzines.com/images/debugging/toc-letter.pdf
[x] [design, visual communication] https://anthonyhobday.com/sideprojects/saferules/ & https://news.ycombinator.com/item?id=34684761
[x] [background on ChatGPT and Claude from an ex-OpenAI, Anthropic founder] read The Boy Whose Light Went Out by Jack Clark http://techpolicylab.uw.edu/wp-content/uploads/2022/04/Telling_Stories_Pages_4-4-22.pdf
[x] [history of OpenAI internal difficulties] https://www.technologyreview.com/2020/02/17/844721/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/
[x] [context for supply chain ban on GPU exports/imports] https://www.palladiummag.com/2021/10/11/the-triumph-and-terror-of-wang-huning/
[x] [starting to think about bias in what you read from chatgpt or elsewhere] https://catalogofbias.org/biases/
[x] [psychological tools for support in dealing with ChatGPT, especially if you decide to try jailbreaks] https://www.bellingcat.com/resources/2022/11/23/how-to-maintain-mental-hygiene-as-an-open-source-researcher/
[x] [how data is being used by open source intelligence researchers in war] https://www.cbsnews.com/news/bellingcat-russia-putin-ukraine-60-minutes-2022-08-21/
[x] [background on social media and recommendation algorithms] https://knightcolumbia.org/content/understanding-social-media-recommendation-algorithms

Watching

(message Jaan if you need a VPN or these links don't work)

[x] [context for popular media portrayals of chatgpt] South Park S26E04 - "Deep Learning" - https://southpark.cc.com/episodes/8byci4/south-park-deep-learning-season-26-ep-4
[x] [context for how the CEO of OpenAI talks about it] https://abcnews.go.com/Technology/video/openai-ceo-cto-risks-ai-reshape-society-97949497 article
[x] [an emotional history of the world; context for our data thinking practice of feel, ask, do, think -- and how the thoughts in people's heads can shape society] 1 episode of https://thoughtmaybe.com/cant-get-you-out-of-my-head/

Red Teaming

My own homework critique (10 mins):

The abstract doesn’t say anything about expectations for the research results - it’s too short, and you need to read more to understand the goal.

When discussing data cleaning, I talk about using df.describe() function, but don’t explain what df is, where it comes from etc. More explanation about the methodology and libraries used would be nice.

In some parts, I talk about what exact functions I used and partly show the code, but in others, I just say I did x or y. The approach to explaining what’s going on should be standardized.

I forgot to add the mathematical equations for linear and logistic regression.

Figures are not mentioned in the text, so when reading you have to go and find the correct figure manually.

Typo in TfIdfVectorizer class name - should re-read parts like class names (which are not detected by grammar-checking tools manually) before submission.

I used logistic regression for multiple classes but did not explain how multiclass logistic regression works.

My homework critique from chatGPT (I Don’t have access to GPT-4 yet):

Overall, your report is well-organized and provides a clear overview of the approach and methods used to analyze the Zulip chat data. However, there are a few areas that could be improved:

Lack of clarity in the research question: While you mention that the purpose of the analysis is to learn the process of data thinking, it is not entirely clear what research question or objective you are trying to answer with your analysis. This could be made more explicit in the introduction.

Lack of context: It would be helpful to provide some context about the Data Thinking course and the Zulip chat data, such as the purpose of the course and who the participants are. This would help the reader better understand the significance of the analysis and the relevance of the results.

Lack of detail in the methods: While you briefly describe the methods used for linear regression and embeddings, there is not much detail provided about how these models were implemented or how the results were interpreted. Providing more detail about the specific methods and approaches used would help the reader better understand the analysis.

Lack of discussion of limitations and potential biases: It would be useful to include a discussion of any limitations or potential biases in the analysis, such as limitations in the data or assumptions made in the models. This would help the reader better understand the reliability and generalizability of the results.

Inconsistent use of language: There are a few instances where the language used in the report is informal or unclear. For example, in the introduction, you use the phrase "with the help(?) of large language models," which is not entirely clear. Using more formal and consistent language throughout the report would improve its clarity and readability.

Overall, your report provides a good overview of the process and results of your analysis, but there is room for improvement in terms of clarity, detail, and consistency.

1- The report is well-organized and written. 2- I feel there was a nice effort spent on the data preparation and data thinking 3- I like the way of thinking to split the data into different target classes to use in the regression. 4- The correlation between Zulip message length and time of day is an interesting way to analyze the data. 5- Figure 2, which shows message embeddings projected onto a 2D plot using PCA, is too small for me to understand the idea clearly. 6- Figure 4, which visualizes the performance of the logistic regression model, appears to have some issues. This could be due to data bias, as the decision surfaces for multinomial logistic regression are not visible. Finally, Good job and great work :)

I inputted the datathinking wikipedia article preceded by the prompt "Memorize this definition of Datathinking. I will ask you questions about it later.". Then I inputted a text only version (aquired using pdftotext) preceded by the prompt "Critize the following report using the definition of Datathinking". The results are below. View the conversation here

The given report demonstrates an attempt to analyze Zulip chat data from the Data Thinking course at the University of Tartu. While the report provides some insights and results, there are several areas where it could be criticized based on the definition of Data Thinking:

Lack of clear research question: The report states that the goal is not necessarily to find precise conclusions but rather to learn the process of data thinking. However, it is essential in any data analysis to have a clear research question or objective to guide the analysis and interpretation of results. Without a specific question, the analysis may lack focus and fail to provide meaningful insights.

Inadequate data cleaning explanation: The report mentions the removal of certain columns based on their perceived lack of value. However, it does not provide a comprehensive explanation or justification for these decisions. It is important to clearly outline the criteria used for data cleaning and provide reasoning behind the removal of specific columns. This helps ensure transparency and reproducibility of the analysis.

Insufficient exploration of alternative models: The report solely focuses on linear regression and logistic regression models without exploring other potential models that could be more appropriate for the given data. Data thinking involves considering various modeling techniques and selecting the most suitable one based on the data characteristics and research question. The report does not justify why linear and logistic regression were chosen over other alternatives.

Limited evaluation of results: The report briefly presents the results of the linear regression and logistic regression models but does not provide a comprehensive evaluation or interpretation of these results. It is crucial to critically analyze and interpret the model outputs, assess their performance, and discuss any limitations or uncertainties. This would enhance the validity and reliability of the analysis.

Inadequate discussion of visualization choices: The report mentions attempts to visualize the results using different techniques but fails to provide a thorough discussion of the visualization choices made. Data thinking involves selecting appropriate visualizations to effectively communicate insights and patterns in the data. It is important to explain the rationale behind the chosen visualizations and discuss their effectiveness in conveying the intended message.

Limited reflection on the data thinking process: While the report briefly touches upon the value and principle of data thinking, it lacks a comprehensive reflection on the entire data thinking process. Data thinking involves iterative exploration, questioning, and critical analysis of the data. The report could benefit from a more in-depth reflection on the challenges faced, lessons learned, and areas for improvement in the data thinking process.

Overall, the report demonstrates some efforts in analyzing the Zulip chat data but falls short in certain aspects of data thinking, such as the formulation of research questions, comprehensive data cleaning explanation, exploration of alternative models, evaluation of results, discussion of visualization choices, and reflection on the data thinking process. Improvements in these areas would strengthen the analysis and make it more aligned with the principles of data thinking.

onefact / datathinking.org