search

osome-iu / hoaxy-backend

Backend component for Hoaxy, a tool to visualize the spread of claims and fact checking
http://hoaxy.iuni.iu.edu/
GNU General Public License v3.0
139 stars 44 forks source link

Comply with new Twitter API #59

Closed filmenczer closed 1 year ago

filmenczer commented 4 years ago

We need to update our code to comply with the new format of the streaming API: https://blog.twitter.com/developer/en_us/topics/tools/2020/introducing_new_twitter_api.html

filmenczer commented 3 years ago

@shaochengcheng would you like to look into the new Twitter API v2 that was just released: https://developer.twitter.com/en/docs/twitter-api/early-access We need to make changes in preparation for the deprecation of v1.

shaochengcheng commented 3 years ago

Hi @filmenczer, I just roughly went through the documentation of version 2. Honestly speaking, it is a big improvement that gives us finer control over Twitter data. But this big change also means a lot of work to migrate, I would like to present my two main concerns.

  1. About the limits for the number of tweets. According to the Filtered Stream API Reference, the basic access level is 500,000 Tweets per month per project, which is far from our needs.

  2. The fundamental data model changed a lot. The two main object tweet and user data models have been dramatically changed in version two, please check v1.1 and v2. In simple words, in the data model version 2, only a small set of fields are set as default returned, and several new fields are introduced as well. In fact, the API migration documented that (the returned tweet objects format is) 'determined by fields and expansions request parameters, not backward-compatible with v1.1 formats'. All of these indicate a huge work on migration.

There are also good news from the new APIs:

  1. Direct search of the URL string, instead of searching the whole tweet;
  2. No need to unshorten URLs anymore, as the introduction of entities.urls.expanded_url;
  3. A better way to handle reply and quote type, as the introduction of referenced_tweets.

We can have a meeting about the code migration to talk about the technical details.

Thanks Chengcheng