Open jonathanrstern opened 1 month ago
Changed my User-Agent to Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36
and it worked!
One question @imperatrona - let's say I'm using the library's search feature... do you recommend I log in every time I want to do a new search?
At what point will Twitter flag my account for too many logins?
@jonathanrstern you shouldn’t log in often. after you log in you should save cookies the twitter returns in set-cookie header and reuse them as long as they valid instead of log in every time.
Got it. That’s what I sort of thought, but glad to have confirmation. Thank you @imperatrona :)
Two follow up Qs:
I’ve noticed User-Agent making a pretty big difference regarding whether I’m able to log in successfully. For example, TwitterAndroid/99 is not working but regular Mozilla Chrome is. Have you observed Twitter/X updating what is acceptable for user agents?
For a single user, how much scraping is too much? i.e, at what point does one risk suspension? Is 1 search per minute on the low end or high end?
@jonathanrstern sorry for not getting back to you sooner!
Thanks @imperatrona
"you should use browsers user-agents" -- that's in line with my experience as well. Thanks for confirming.
I'm using the SearchTimeline feature. A few questions:
Appreciate all the help :)
@jonathanrstern
Some endpoints are little buggy:) For example SearchTimeline for diffrent users can return different results with same request. Or GetTweetRetweeters can sometimes returns empty results with cursor only.
Tweets count doesn’t rate limited, only requests count. So if your one request returns 300+ tweets its still counts as one request.
I hadn’t encounter any restrictions for a new accounts. Though couple of my new accounts was suspended, but it was 3 pieces from 300 accounts batch. But anyway be safe, if your account got rate limit error wait enough time before sending requests.
Happy to help!
@imperatrona
I am back. some API is too expensive. So we want to use your project to make a single API request.
response status 403 Forbidden: {"errors":[{"code":64,"message":"Your account is suspended and is not permitted to access this feature."}]}
How many source public IP addresses do you use? I guess there is IP restrictions.
@wade-liwei i have not encountered any ip restrictions with hundreds of accounts on one ip. i’m using single ip. you should try make less requests than rate limit allows, for example ~100 instead of 150 requests per 15 minute
Could you please share the request rate for one account?
Right now, I am testing 1 request per 10 minutes per account.
~100 instead of 150 requests per 15 minutes, for 20 accounts?
@wade-liwei rate liming is per endpoint, not per account. in readme.md i have rate limits for each endpoints
50 requests / 15 minutes = 1 request / 18 seconds
How many accounts should I use?
twitter always locked my account. So I think I should find out the locked account reason.
@wade-liwei it's depends, how many data you need to get?
a little data, we just need to get the new follower.
not very big data, but I need the real time data(delay 5 minutes).
@wade-liwei
GetProfileByID
to check if followers count changed. it doesn't require auth, and after reaching limit you can generate new guest session with GetGuestToken
. then if followers count changed you can use authorized client to fetch new followers. this way you'll avoid unnecessary FetchFollowers
requests. for public method i still recomend to use some adequate delay, or your ip may get banned.FetchFollowers
is limited for 50 requests, but you should stop somewhere between 30-40. constantly reaching limits will lead to account suspension.Thank you. I will implement it tomorrow. Right now, sleeping time is coming in China.
@imperatrona good solution.
RIght now, If my account can not login (DenyLoginSubtask), I replace the username and password login with cookies(from the browser and can not logout on browser (expirationDate: a year later) :).
@imperatrona
You said: one request can return up to 20 followers. I see the parameter(maxUsersNbr) is 20 too.
But I can get 70 or 69 followers response every request. Available followers are 20?
@wade-liwei when making requests we set count to 20 as twitter web client do, but often server returns more, sometimes it can return 0 tho, its just the way the twitter coded now 🤡
Running into this error on
login()
... thoughts?Auth error (366): flow name LoginFlow is currently not accessible
It happens right after the Account duplication check:
(I'm using python)