JayThibs
commented
1 year ago I used GPT-4 to come up with a little estimate of the cost:
To estimate the cost of using GPT-3.5-turbo to verify whether a tweet contains misinformation, we need to consider both input and output token costs and the average token count per tweet analysis.
Average Token Count per Tweet Analysis:
A tweet can be up to 280 characters long. Considering that an average English word is around 4.5 characters plus a space, this translates to about 280 / 5.5 ≈ 51 words per tweet. Tokens are not equivalent to words; a token can be a word, a part of a word, or punctuation. On average, we can estimate around 1.5 tokens per word, considering that longer words might be split into multiple tokens and punctuation also counts as tokens. Therefore, each tweet can be estimated to involve about 51 * 1.5 ≈ 77 tokens.
Input Cost Calculation: If you analyze 10,000 tweets per day, and each tweet is approximately 77 tokens, the total daily input tokens are 10,000 77 = 770,000 tokens. The cost for input usage is $0.0030 per 1,000 tokens. So, the daily input cost is 770,000 / 1,000 $0.0030 = $2.31.
Output Token Estimation: The output for each tweet verification might vary, but let's assume an average of around 50 tokens per response (this is a rough estimate, as the model might provide a brief or detailed analysis depending on the complexity of the tweet). For 10,000 tweets, the total output tokens would be 10,000 * 50 = 500,000 tokens.
Output Cost Calculation: The cost for output usage is $0.0060 per 1,000 tokens. So, the daily output cost is 500,000 / 1,000 * $0.0060 = $3.00.
Total Daily Cost: Adding both input and output costs, the total daily cost would be $2.31 (input) + $3.00 (output) = $5.31.
Monthly Cost Estimation: If this operation runs every day for a month (30 days), the monthly cost would be $5.31 * 30 = $159.30 (per month). This is a rough estimate based on average values for token length and response size. The actual cost may vary depending on the exact length of each tweet and the verbosity of the model's responses.
This seems like a reasonable cost to aim for. For big social media companies, this is nothing (even if we increase it to $1000/month). I picked 10k tweets per day because we could use other cheaper methods to filter down to about that many tweets per day. For example, only throw in tweets that have over 100 likes (or more), rank how important the tweet should be sent to 3.5-turbo or 4 based on a score that takes into account a list of sensitive words and embedding scores. I'm sure there's additional stuff we could add here. Clip anything with a low score + max out at x number of tweets per day.
You could add gpt-4-vision for tweets that contain images or gpt-4 for the y number of most-liked tweets.
Also, it would be good to make it even easier for people to write great community notes. I'm sure GPT-4 + search and other models can help with this.
(Of course, the companies could probably save even more money if it just had a GPU with a fine-tuned model and maybe fine-tuned embeddings.)
luckybear97
TheApproach
xcsf6
tactipus
In order to improve the speed at which important community notes get added and to help community noters write better notes, I'm curious if people have put some effort into using a mic of AI (language models) and more simple methods. I'd like to help with this if we can make it work economically.
For example, you could have a scaffolding approach that looks for specific words, which then feeds into an embedding for semantic similarity to contentious issues, and then finally into an LLM that ranks how important the tweet is to have a community and some additional context (through a web search and internal knowledge within the LLM) to help the community noter. I think there's a way to make this economically viable for companies.
Yes, companies, I want Community Notes to expand beyond X. Let's figure out how to connect it to YouTube. Why haven't other social media websites picked it up yet? If they care about truth, this would be a considerable step forward beyond. Notes like “this video is funded by x nation” or “this video talks about health info; go here to learn more” messages are simply not good enough. We need to improve the state of truth-seeking on the internet.
Not just that, as an AI Safety researcher, this is particularly important to me. Don't forget that we train language models on the internet! The more truthful your dataset is, the more truthful the models will be! Let's revamp the internet for truthfulness, and we'll subsequently improve truthfulness in our AI systems!!