imperatrona / twitter-scraper

Scrape the Twitter frontend API without authentication with Golang.
MIT License
118 stars 20 forks source link

Follows #4

Closed imperatrona closed 9 months ago

imperatrona commented 9 months ago

add method for getting user follows and followers

wade-liwei commented 4 months ago

How do you get the request URL?

https://twitter.com/i/api/graphql/jwbfbSzn0FRL_AMZGsYDag/Followers

I tried the URL https://github.com/fa0311/TwitterInternalAPIDocument/blob/master/docs/markdown/GraphQL.md#followers but failed.

Do you have a document? I want to get the other URL.

imperatrona commented 4 months ago

@wade-liwei It's probably failed caused of lack of param variables in your request. TwitterInternalAPIDocument is a great project but it doesn't have required variables.

Use Network tab in Developer Tools (Alt+Ctrl+I). Go to page that have required data, and then look for request that you need. In Payload tab you will find all data twitter required for this endpoint.

https://github.com/imperatrona/twitter-scraper/assets/37719998/b62e804b-4c1a-4bbe-97a1-0b25994393d3

wade-liwei commented 4 months ago

thank you make the video for me.

So can I find and use the post method like post twitter or retwitter by the browser Developer Tools?

imperatrona commented 4 months ago

@wade-liwei Yes you can create new methods by copying requests data from Developer Tools. If you implement new methods please maake pull request, i will be grateful!

ps. i just now added CreateTweet, DeleteTweet, CreateRetweet and DeleteRetweet methods wich works with previuosly implemented UploadMedia. also CreateScheduledTweet, DeleteScheduledTweet and FetchScheduledTweets is available

wade-liwei commented 4 months ago

Got it, thank you.

If I need to implement new methods, I will do and make a pull request.

thanks for your project help me.

wade-liwei commented 4 months ago

Could you please help me to resolve my problem.

right now, I can get the user's followers with my backend service, then I need to bind the follower with the real twitter account in my app.

I want to use twitter Oauth 2.0, but we just get the name field from Oauth 2.0. I found out the API https://api.twitter.com/2/users/me , but I am not sure how to use it after login with twitter Oauth 2.0 on the app(https://dashboard-1-gray.vercel.app/login), and not sure can resolve my problem.

imperatrona commented 4 months ago

@wade-liwei with oauth2 you recieve authorization code wich you can use to create access token (Step 3 at this article - https://developer.x.com/en/docs/authentication/oauth-2-0/user-access-token)

and then you should make requests to api with access token in header Authorization

token := "auth_token"
req, err := http.NewRequest("GET", "https://api.twitter.com/2/users/me", nil)
req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", token))

client := &http.Client{}
resp, err := client.Do(req)

i’m also recomend you to use user ids intead of usernames for binding users, since usernames changeble it can cause a lot of problems.

wade-liwei commented 4 months ago
  1. Got it, thank you.

  2. I will use use user ids instead of usernames for binding users.

  3. https://api.twitter.com/2/users/me

    X API v2 free 25 requests / 24 hours PER USER. 0.0002/s basic 250 requests / 24 hours PER USER. 0.002/s pro 75 requests / 15 mins. PER USER. 0.08/s

    I guess the rate limiter is for every user.

wade-liwei commented 4 months ago

Hey @imperatrona

Thank you so much for helping me. I finished the job.

wade-liwei commented 4 months ago

Hey @imperatrona

Do you have a document about how to implement like(FavoriteTweet) interface?

imperatrona commented 4 months ago

@wade-liwei

hi! sorry it’s not implemented yet. if you want you can implement this feature by yourself. you can do it by using Developer Tools - like/unlike tweet and copy the request data to new methods. please make a pull request if you implement this

imperatrona commented 4 months ago

hey @wade-liwei As I was implementing today new GetSpace method, i also added LikeTweet and UnlikeTweet, you can read more here - https://github.com/imperatrona/twitter-scraper?tab=readme-ov-file#like-tweet

wade-liwei commented 4 months ago

works.

I found out the link, https://x.com/i/api/graphql/lI07N6Otwv1PhnEgXILM7A/FavoriteTweet

and you use the link: https://twitter.com/i/api/graphql/lI07N6Otwv1PhnEgXILM7A/FavoriteTweet

I am not sure the reason. and I have to use auth_token and oath_token and cookie.

wade-liwei commented 4 months ago

Thank you so much.

Do you know there is a users list that have liked a Tweet and a users list that have Retweeted a Tweet?