vvidday
commented
2 years ago Approach
Frontend (Avoid duplicates + Reduce API calls)
Instead of a fresh API call being made every round (thus incurring 10 tweets called per round), one API call is made per account at the start of the game. The API call fetches 50 tweets, up from 10.
This was implemented in bb6bdb6f8e1d549c53cf7a62b06dc01f96953c39 and b059d8fe2c4060b60fee64a543b742516bdd83ce.
Instead of one tweet being selected and the other 9 being discarded, the 50 tweets per account are stored in memory throughout the duration of the game. Each time a tweet is selected for a round, it is removed from memory, thus avoiding duplicates.
This was implemented in bcb3b3487126d63b3f102e357c9352bda79a6fbc, a26ecb578cb8e657629e087388bf479b75b66290 and cf0552c8ba86726e6208b1385b0079856aa51921.
Backend (Reduce API Calls)
A very obvious problem is that we are making many repeated calls - For instance, we might make a call to get the most recent 50 tweets from Barack Obama many times even though the data doesn't change.
One obvious solution was to store the tweet data somewhere, and direct all subsequent calls to the stored data - only occasionally calling the actual Twitter API to keep the data updated in accordance with the twitter developer policy.
We implemented this via a separate webserver. The webserver exposes endpoints for tweets, where a request is processed as follows:
- If tweets do not exist in the database, tweets are retrieved from the twitter API and stored in the database
- If tweets already exist in the database, the time they were retrieved is checked.
- If the tweets were retrieved more than 12* hours ago, a fresh call to the twitter API is made, and the database entries are updated
- Else, the existing tweets from the database are returned, avoiding the need for a call to the twitter API.
- This thus enables a large reduction in calls made to the twitter API, while still maintaining the validity of tweets.
*Note: 12 hours is the current implementation, but this value can be freely modified.
The webserver can be found in a separate repository. It is written in Go and uses a PostgresQL database for storage. It is currently deployed on AWS Elastic Beanstalk. Due to AWS not providing https, we've moved the deployment to heroku. It is functionally the same, but just swaps out Echo for Gin (due to configuration issues with Heroku) and some heroku-specific files.
Commits 644e78a86e2830ddaa52815bdf5f40e5e1488029, dfe0fa4f823064f0199bf89758e6e7372a8fae1c and 8de6fee0779f76c15e1bd2163c1221bdb4a23859 deal with interfacing with this new web server.
As stated in our MS2 Documentation, we plan to tackle the problem of duplicate tweets without unnecessarily increasing the number of calls to the twitter API.