search

iit-cs579 / main

CS579: Online Social Network Analysis at the Illinois Institute of Technology
147 stars 204 forks source link

how to effectively fetch Tweets #362

Closed changediyasunny closed 7 years ago

changediyasunny commented 7 years ago

@ClaytonTurner , I am facing issue in fetching tweets and not sure whether they are unique or duplicated. I am using "search/tweets" to get tweets containing terms "abcd".

When I send requests, for 1st request I get (100) tweets. but for next subsequent requests I get number of tweets as 1, 0, 0, 0, 0, 0, 0, 24, 24, 6, 9, 99.

after 1st request, I am updating since_id using max_id as since_id = "max_id from previous request" and count = 100. Is there a way around to get 100 tweets each time ?

davidghiurco commented 7 years ago

I'm pretty sure you're using the streaming API which gives you live (real-time) data and so it cannot guarantee that at any given point when you request tweets according to some criteria, you will get the exact number you are asking for, because it's live data.

Solution: use REST API

changediyasunny commented 7 years ago

I think it is a REST api only. Also request() method of twitterAPI handles whether endpoint is GET or POST.

It is given in source (https://github.com/geduldig/TwitterAPI/blob/master/TwitterAPI/constants.py) that "Search/tweets" (GET, api) is a REST and makes GET call.

Let me know if I'm missing ?

davidghiurco commented 7 years ago

Ah ok. you're right. I have no idea then :\