imperatrona / twitter-scraper

Scrape the Twitter frontend API without authentication with Golang.
MIT License
103 stars 15 forks source link

Auth error (353): This request requires a matching csrf cookie and header. #16

Closed M1chlCZ closed 1 month ago

M1chlCZ commented 1 month ago

I am suddenly getting this error, which I don't know how to fix, this is relevant part of the code. So if you could help me with that, I am going to be glad.

    scraper := twitterscraper.New()
    loggedIn := false
    f, err := os.Open("cookies.json")
    if err == nil {
        defer f.Close()
        var cookies []*http.Cookie
        err := json.NewDecoder(f).Decode(&cookies)
        if err == nil {
            scraper.SetCookies(cookies)
            loggedIn = scraper.IsLoggedIn()
        }
    }
    if !loggedIn {
        _ = os.Remove("cookies.json")
        err := scraper.Login("usr", "pass")
        if err != nil {
            utils.WrapErrorLog(err.Error() + " 1")
            return
        }
        cookies := scraper.GetCookies()
        js, _ := json.Marshal(cookies)
        f, _ := os.Create("cookies.json")
        _, err = f.Write(js)
        if err != nil {
            utils.WrapErrorLog(err.Error() + " 2")
            return
        }
    }
imperatrona commented 1 month ago

sorry but Login method doesn't work right now. recently it's require X-Client-Transaction-Id header wich i've tried to implement but wasn't succeed:( if someone want to dive into it i can share my recent code

teslashibe commented 1 month ago

We would love to help and contribute. Please share and we can collaborate

teslashibe commented 1 month ago

@imperatrona I wrote a fix for this today and it works. We should collaborate on maintaining this package together. We are going to fork and maintain the fork carefully. We are also going to refactor the code so it's not a flat package which will make it easier to maintain. We are an open source company and have resources to maintain this consistently. Why don't we team up and build a community around a stable go package together? We will make sure it's maintained in a daily basis and we can collaborate together with you getting full credit for all your amazing work.

I love the work you have done to iterate on n0madics work.

I sent you a twitter dm let's chat there. We are hiring great people to work with as well 🤩

imperatrona commented 1 month ago

@teslashibe sure i'm open to chat:)

rectid commented 1 month ago

@teslashibe Hello! You wrote that you were able to fix the authentication. Can you upload this solution to your fork/make a pull request in this repo? I have been racking my brains over this problem for several days, I would be very grateful to you.

imperatrona commented 1 month ago

@rectid i haven't check but here is their fork: https://github.com/masa-finance/masa-twitter-scraper/tree/feat-fix-login

M1chlCZ commented 1 month ago

@rectid i haven't check but here is their fork: masa-finance/masa-twitter-scraper@feat-fix-login

Just want to confirm, that these changes are working, so if someone can incorporate them here, it would be nice.

imperatrona commented 1 month ago

@M1chlCZ thank you i'll check them and add them here in couple days:)

imperatrona commented 1 month ago

hi @M1chlCZ i've add changes, can you please check if latest version fixes this error for you?

M1chlCZ commented 1 month ago

It did, thank you!

ealmuina commented 1 month ago

@imperatrona This is still happening to me after updating to 0.0.13 😔

imperatrona commented 1 month ago

@ealmuina do you get same 353 error?

ealmuina commented 1 month ago

@ealmuina do you get same 353 error?

Yes, same 353: This request requires a matching csrf cookie and header

imperatrona commented 1 month ago

@ealmuina thats probably my fault, can you check v0.0.14?

go get -u github.com/imperatrona/twitter-scraper@v0.0.14
ealmuina commented 1 month ago

Awesome! That one does solve it for me. Thanks a lot!!

imperatrona commented 1 month ago

@ealmuina Thank you so much for helping me!