Document Nitter's use of Twitter's internal APIs

[girst]

@zedeus has done a great deal of reverse engineering Twitter's internal API, and I believe it would be very useful if it were documented somewhere. This would allow others (e.g. me) to build upon it directly, maybe help acquire new contributors, and those asking for a Nitter API (e.g. #299, #192) could be pointed to this document instead.

Things that could be documented:

• API URLs and what they do, including
  • HTTP method
  • required/tested headers
  • query strings/post body
  • expected status codes
  • expected output
• authentication (fetching tokens)
• known quirks and workarounds
• rate limits
• media retrieval
• ???

I know this is a huge amount of work, and I neither can nor want to demand its realization, but maybe knowing it'd be useful would motivate zedeus to write some stuff down. :)

[prologic]

Where is this currently documented? I'd like to build a Go library that implements the Twitter Unofficial API for use in the feeds.twtxt.net service I run (Source code: https://git.mills.io/yarnsocial/feeds) Thanks! 🙏

[zedeus]

It's not documented, that's kinda the nature of it. There are a few articles and PDFs lying around, not too hard to find.

[prologic]

It's not documented, that's kinda the nature of it. There are a few articles and PDFs lying around, not too hard to find.

Do you mind linking those? Or sharing somehow?

[u0nel]

I found a python library, that uses twitters internal api: https://github.com/mahrtayyab/tweety

[kleozzy]

I am also interested in what are nitter's Rate limits if any and if there is any connection with Twitters rate limits ? For example if i host my own instance do i poke twitter api from my vps or is there data already cached from other instances that i grab from there ? How does it all work .

Very interested to find out.

[back-2-95]

Reading that python repo ^^ and got one of the requests working (using HTTPie):

• x-csrf-token is just random string of 32 chars
• token was copied from the repo (dunno where that came)

http POST https://api.twitter.com/1.1/guest/activate.json\
  'x-csrf-token: q2lUzQ0kWwUZkVq8FlX8Lss0jgoJCj7Z' \
  'content-type: application/x-www-form-urlencoded' \
  'accept: */*' \
  'authority: twitter.com' 'referer: https://twitter.com/AmitabhJha3' \
  'authorization: Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA'

Response:

HTTP/1.1 200 OK
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-disposition: attachment; filename=json.json
content-encoding: gzip
content-length: 63
content-type: application/json; charset=utf-8
date: Sun, 18 Dec 2022 18:19:52 GMT
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Sun, 18 Dec 2022 18:19:52 GMT
perf: 7626143928
pragma: no-cache
server: tsa_o
set-cookie: guest_id=v1%3A167138759290782939; Max-Age=34214400; Expires=Thu, 18 Jan 2024 18:19:52 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
strict-transport-security: max-age=631138519
x-connection-hash: 32088f28ee48e45f3224af0c13224fcdc36db3ab8544467b89245c5b48d39b08
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-response-time: 117
x-transaction-id: ce82d112097ad393
x-twitter-response-tags: BouncerCompliant
x-xss-protection: 0

{
    "guest_token": "1604541976950497282"
}

[prologic]

@back-2-95 Mate if you could turn this into a bunch of curl examples like this documenting the entirety of the API and its usage, someone like me could go write say a proper nice Go library for it 👌

[u0nel]

I started working on a go library a while ago: https://github.com/u0nel/twitter I was able to get profile information, but struggled parsing the timeline response

[prologic]

@u0nel What problems did you run in to? 🤔 -- I basically want a Go client so I can better support Twitter™ public feeds in feeds.twtxt.net (which I had to recently remove support for, because the library I was using broke and Twitter™ started responding with "Too many requests" or something...) -- Basically I just want to pull and convert public Twitter™ feeds into Twtxt that any Yarn.social client can utilise.

[back-2-95]

@prologic that would be nice but might not have time for that. For my short term needs I created this dockerfile https://github.com/back-2-95/mastodon-poster/tree/main/twitter with that tweety so I can fetch tweets by user in json.

docker image build -t twitter .
docker run --rm twitter some_twitter_username

[prologic]

@back-2-95 Thanks! 🙏 But I'll hang out for a native Go client (after we figure out how it all works) 👌

[back-2-95]

Yeah, I'm also watching this issue if more insight comes out.

[back-2-95]

https://github.com/mahrtayyab/tweety still works even if Twitter has closed apis