imperatrona / twitter-scraper

Scrape the Twitter frontend API without authentication with Golang.
MIT License
117 stars 18 forks source link

fix: change bearer token for authentication that is less restrictive #28

Open thewh1teagle opened 1 day ago

thewh1teagle commented 1 day ago

Fix https://github.com/imperatrona/twitter-scraper/issues/27

dirdigeng commented 1 day ago

Many thanks to @thewh1teagle. This patch worked for me and the original code was still failing as I switched between the two.

However with this bearer token there seems to be a lower-than-previously rate limit on the number of scraper.IsLoggedIn() calls in a 15-minute period (seems to be ~75 calls) that did not previously apply. My use case is a simple replacement for the cli twurl call I used to use to get a single tweet, so that now is rate limited by the limit on IsLoggedIn rather than the limit on GetTweet. However with multiple GetTweet calls after a single IsLoggedIn I can hit the 150/15 minutes on GetTweet before I hit the ~75/15 minutes on IsLoggedIn. Short-term fix is just to lower the rate of the cli calls, long term fix of batching the GetTweet calls means more upstream re-engineering (sigh). But credit to @thewh1teagle. @imperatrona, @n0madic, and others for being able to do this at all!

cmj commented 1 day ago

The other Bearer token that worked for me was: "AAAAAAAAAAAAAAAAAAAAAFQODgEAAAAAVHTp76lzh3rFzcHbmHVvQxYYpTw%3DckAlMINMjmCwxUcaXbAN4XqJVdgMJaHqNOFgPMK0zN1qLqLQCF"

This seems to be more widely-used token so the limits might be higher?

Just checked headers with both. The first one given in the patch is x-rate-limit-limit: 75

This one is x-rate-limit-limit: 2000

thewh1teagle commented 1 day ago

Thanks. I changed it to the bearer token with the highest rate limit. I can squash the commits though it may be useful to have the previous bearer token in the history.