search

cjbarrie / academictwitteR

Repo for academictwitteR package to query the Twitter Academic Research Product Track v2 API endpoint.
Other
272 stars 59 forks source link

get_user_followers() does not seem to work fine #187

Closed yuanzhouIR closed 2 years ago

yuanzhouIR commented 3 years ago

I want to retrieve the followers of thousands of users' followers. However, the get_user_followers() seems to return only approximately 5% users' followers, and the remaining returns NA. To reproduce this problem, simply to run get_user_followers(500users, bearer_token = get_bearer()). Because Twitter API's rate limit, this may take a long time. For the NA returned users, if I run get_user_followers(one_NA_returned_user, bearer_token = get_bearer()), I can get the data successfully.

chainsawriot commented 3 years ago

Related to #192

We can't stress test the code. And fetching 500 users in one go is untestable (Remember the rate limit is 15 requests per 15 minutes. And fetching 500 users, assuming only one page per user, is >500 minutes. That's >8 hrs.)

Of course, we can add a warning or whatever telling users not to stress get_user_followers with 500 users. But at the moment without any fix, please practice divide et impera and keep the queries small. We will investigate a way to test this.

yuanzhouIR commented 3 years ago

@chainsawriot Thanks for your reply and suggestion. I will try data_container <- vector("list", length(users)) for (i in seq_along(users) { data_container[[i]] <- get_user_followers(users[i], bearer_token = get_bearer()) }