API rate limit exceeded with scrape_profile() function

[secreit]

first off all thanks for the package.

i'm getting this error:

ValueError: {'message': "API rate limit exceeded for (my ip adress) (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)", 'documentation_url': 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'}

any way to get around this?

[shaikhsajid1111]

That link redirects to GitHub docs. I don't understand how it showed up?, What were you doing?. Is it coming from webdriver manager I guess? Because you spawned multiple instances of browser

[secreit]

i am just scraping a twitter profiles last tweet. when i do it once every 30 seconds i get this message after thirty minutes. there appears to be a rate limit so its only possible to send 60 requests an hour as mentioned in the link "For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the person making requests.".

i don't really understand why or how this works. i'm just using the scrape_profile function.

[secreit]

also i just noticed that when checking for the most recent tweet when a user has a pinned tweet that one will be chosen instead,

[shaikhsajid1111]

That rate limit is basically on webdriver manager, one of the dependency that installs webdriver for automating web browser

[phanxuanphucnd]

@shaikhsajid1111 I use this function to crawl all tweets of 1 user but only get 7 tweets? Can you tell me how to get all of that person's tweets?