get_all_tweets reaches rate limit after four queries max

Please confirm the following

[X] I have searched the existing issues
[X] The behaviour of the program is deviated from what is described in the documentation.
[X] I can reproduce this problem for more than one time.
[X] This is NOT a 3-digit error -- it does not display an error message like something went wrong. Status code: 400.
[ ] This is a 3-digit error and I have consulted the Understanding API errors vignette and the suggestions do not help.

Describe the bug

I use get_all_tweets to retrieve tweets that contain a certain URL (example "querie=url:www.cats.com"). But I always hit the rate limit after a maximum of four queries. Usually, it sleeps for 15 minutes and then continues, but sometimes it just freezes during the 15 minutes wait.

Why does it reach the rate limit so fast?

Expected Behavior

According to Twitter the v2 rate limit for full archive tweet search is 180 requests/ per user/ per 15 minutes (https://developer.twitter.com/en/docs/twitter-api/rate-limits#v2-limits)

I thought that one querie equals one request.

Steps To Reproduce


data <- c("cats.com","pandas.com","fish.com","dogs.com","elephants.com")
df <- data.frame()

for (i in data){

  query <- build_query(
    query = paste0('url:www.',i),
    is_retweet = FALSE,
    is_reply = FALSE,
    is_quote = FALSE)

new <- get_all_tweets(
  query,
  "2009-03-26T00:00:00Z",
  "2022-01-01T00:00:00Z",
  bearer_token=bearer_token,
  n= Inf,
  bind_tweets = TRUE) 
if(ncol(new)>10){
  new <- new %>% select(id,author_id,text,entities,lang,public_metrics,source,possibly_sensitive,created_at,conversation_id)
} # sometimes the output contains extra columns (such as geo-tag) which throw error in rbind, so need to remove those
df <- rbind(df,new)
}

Environment

R version 4.0.4 (2021-02-15) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS 12.2

Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] forcats_0.5.1 purrr_0.3.4 tidyr_1.1.4 tibble_3.1.6
[5] tidyverse_1.3.1 ggVennDiagram_1.2.0 ggplot2_3.3.5 academictwitteR_0.3.0 [9] RecordLinkage_0.4-12.3 ff_4.0.5 bit_4.0.4 RSQLite_2.2.9
[13] DBI_1.1.2 BBmisc_1.11 httr_1.4.2 stringr_1.4.0
[17] stringi_1.7.6 dplyr_1.0.7 readr_1.4.0 rtweet_0.7.0
[21] rmarkdown_2.11 reprex_2.0.1

loaded via a namespace (and not attached): [1] fs_1.5.2 usethis_2.1.5 lubridate_1.8.0 bit64_4.0.5 evd_2.3-3
[6] tools_4.0.4 backports_1.4.1 bslib_0.3.1 utf8_1.2.2 R6_2.5.1
[11] rpart_4.1.16 colorspace_2.0-2 nnet_7.3-17 withr_2.4.3 processx_3.5.2
[16] tidyselect_1.1.1 curl_4.3.2 compiler_4.0.4 rvest_1.0.2 cli_3.1.1
[21] xml2_1.3.3 sass_0.4.0 scales_1.1.1 checkmate_2.0.0 callr_3.7.0
[26] proxy_0.4-26 digest_0.6.29 pkgconfig_2.0.3 htmltools_0.5.2 parallelly_1.30.0 [31] dbplyr_2.1.1 fastmap_1.1.0 highr_0.9 readxl_1.3.1 rlang_1.0.0
[36] rstudioapi_0.13 jquerylib_0.1.4 generics_0.1.1 jsonlite_1.7.3 magrittr_2.0.2
[41] ada_2.0-5 Matrix_1.4-0 Rcpp_1.0.8 munsell_0.5.0 fansi_1.0.2
[46] clipr_0.7.1 lifecycle_1.0.1 yaml_2.2.2 MASS_7.3-55 grid_4.0.4
[51] blob_1.2.2 parallel_4.0.4 listenv_0.8.0 crayon_1.4.2 lattice_0.20-45
[56] haven_2.4.3 splines_4.0.4 hms_1.1.1 ps_1.6.0 knitr_1.37
[61] pillar_1.6.5 future.apply_1.8.1 codetools_0.2-18 glue_1.6.1 evaluate_0.14
[66] data.table_1.14.2 modelr_0.1.8 vctrs_0.3.8 cellranger_1.1.0 gtable_0.3.0
[71] future_1.23.0 assertthat_0.2.1 cachem_1.0.6 xfun_0.29 prodlim_2019.11.13 [76] xtable_1.8-4 broom_0.7.12 e1071_1.7-9 class_7.3-20 survival_3.2-13
[81] RVenn_1.1.0 memoise_2.0.1 lava_1.6.10 globals_0.14.0 ellipsis_0.3.2
[86] ipred_0.9-12

Anything else?

No response

cjbarrie / academictwitteR