[Twitter] Protected Accounts No Longer Downloading Anything

[MarketApplier]

The last time downloading protected accounts worked for me on Twitter was May 8th, looking at my files. After that day, it seems that downloading from said accounts no longer worked. Even after redownloading cookies for my account, nothing. I also tried to clear any cache for Twitter for gallery-dl, using "gallery-dl --clear-cache twitter", but also did not work. Logging out and logging back into the account, and re-exporting the cookies, did not fix my issue, either.

The command I use is the following:

gallery-dl --download-archive twitter.sqlite3 --cookies cookies-twitter-com.txt https://twitter.com/USERNAME

I'm not sure if I can post the full log here, without leaking credentials, but I can at least provide this info, from the verbose command:

[gallery-dl][debug] Version 1.22.0-dev - Executable [gallery-dl][debug] Python 3.9.12 - Windows-10-10.0.19044-SP0 [gallery-dl][debug] requests 2.27.1 - urllib3 1.26.9

If more information is needed, I'll provide what I can, when requested.

[AlttiRi]

It works for me now.

I use only "user-agent": "XXX" and "cookies": { "auth_token": "XXX" } taken from my browser.

[MarketApplier]

I found my auth_token from the dumped cookies, but I don't see where the user agent could be. Any idea where I could get that, for my browser?

[AlttiRi]

https://browserleaks.com/ip https://whatmyuseragent.com/

[MarketApplier]

Presuming all of this info goes under the category right under "extractor", it didn't seem to work, at all. No difference is made, and I still can't download protected tweets. I'm not really sure what could have changed, aside from me getting the latest dev build of gallery-dl. The odd thing is, exporting cookies for another account to download protected tweets DOES work. But cookies for the other account, don't seem to be working. I even tried logging out of all of the accounts, and logged back into the one that wasn't working, to export it's cookies, but that didn't seem to help, either.

EDIT: I decided to try and revert to release version v1.21.2, and now the cookies are working again, as they should. I was originally using a dev build from after this one: https://github.com/mikf/gallery-dl/commit/c4b9f7bab813139b277eed42289092bd667edb61

So I suspect the cookie code was changed in a way that ended up breaking my download system.

[mikf]

Presuming all of this info goes under the category right under "extractor",

It does, but you can also use -o user-agent="..." as command-line flag together with the others.

{
  "extractor": {
    "twitter": {
        "cookies": { "auth_token": "XXX" },
        "user-agent": "XXX"
    }
  }
}

You could try using -o write-pages=all to have gallery-dl write the contents and headers of all HTTP requests to the current directory. Check if the Cookies header send to Twitter includes auth_token

Cookie: _twitter_sess=***; ads_prefs=***; auth_token=***; ct0=***; dnt=***; guest_id=***; kdt=***; twid=***

The odd thing is, exporting cookies for another account to download protected tweets DOES work. But cookies for the other account, don't seem to be working

So it does work in general, and only one account makes problems? Weird.

EDIT: I decided to try and revert to release version v1.21.2, and now the cookies are working again, as they should. I was originally using a dev build from after this one: https://github.com/mikf/gallery-dl/commit/c4b9f7bab813139b277eed42289092bd667edb61

So I suspect the cookie code was changed in a way that ended up breaking my download system.

Seems very plausible that this was caused by some of the changes since v1.21.2, but comparing the cookiejar contents (*) between that version and the current code yields the exact same results when using --cookies, so that can't be it.

(*) I added a print("\n".join(repr(c) for c in self.session.cookies)) after https://github.com/mikf/gallery-dl/blob/eeef9ccdc1813787c5e1327ae2e376e45852c562/gallery_dl/extractor/common.py#L309 and diff-ed the results

[github-account1111]

I use only "user-agent": "XXX" and "cookies": { "auth_token": "XXX" } taken from my browser.

Does it expire? I thought Twitter only supported user-password authentication.

[AlttiRi]

It does not expire. (I use it already several months) As far I know username + password are only needed once to get "auth_token" cookie on log in step.

[github-account1111]

So I don't have to store my credentials in a text file? Does this work for any website that supports authentication?

[AlttiRi]

Cookies/tokens that were received with an authentication (by gallery-dl) are stored in "...\AppData\Roaming\gallery-dl\cache.sqlite3".

So, if you will delete "username", "password" from the config it will continue to work until you "--clear-cache (service-name)" (or the token is expired).

Did not test it, but it should work such way.

[github-account1111]

But I assume both username and password would still be stored plainly, without any encryption. I was more asking if I can go the auth_token route for every website that supports authentication, as that would eliminate the need to enter username and password altogether.