FixTweet is currently broken due to Twitter API changes

[Ai-Himmel]

Run curl https://d.fxtwitter.com/Twitter/status/1601692766257709056 would get Sorry, that Tweet doesn't exist :( Running in the browser will redirect to the raw link.

[inverlock]

Can confirm I am experiencing this issue with any fxtwitter embed

[hexa0]

the same issue has just started happening to me too, probably twitter doing some api nonsense

[Apocolypto]

Likewise, I am experiencing the issue on my end as well.

[dangeredwolf]

It seems Twitter broke the method we use to acocess the API. Specifically, we were using Twitter API v2 REST conversation API, but the Twitter's apps moved to using GraphQL API some time ago. Now it seems they removed that API, which is really annoying. But by moving to the GraphQL API this will be resolved.

[dangeredwolf]

Twitter’s web app also no longer lets you view content when signed out, which is alarming. However, it seems it might not be the full story. I tried seeing if authenticating these requests as a user makes them function, and it seems it still fails to go through. Furthermore, while Twitter made it so you can’t view literally anything, FixTweet’s user API is still working with guest token API. So I’m pretty sure Twitter pushed several changes at once to try to lock down scraping of their API… This will likely not be impossible to work around, but it presents us with quite the challenge…

Edit: At the time of writing, what I said was true, and the user API still worked, but later on it seems Twitter even clamped down on guest token access of API using GraphQL. The syndication API should let us have at least some functionality as we look for longer term solutions. Thank you for your patience!

[dangeredwolf]

Yep it seems moving to GraphQL fixes these issues, thank you so much @Wazbat for helping fix our API stack aaaaaaaaa

I’ve also updated the title as the issue is more all-encompassing.

[Wazbat]

Aaaa no problem! I'll try to have a fix ready soon!

[Mazmol]

Apparently Elon said this was a temp fix, https://twitter.com/TitterDaily/status/1674867516747702277

so don't know if you need to change code or not.

[Haxorzz]

Well let's see how temporary this "temporary" fix is.

[JeremieGiffard]

API access is now based on tier levels with paid features. Fun times, does it relate to this issue ? https://twittercommunity.com/t/announcing-new-access-tiers-for-the-twitter-api/188728

[NicoTheCinderace]

API access is now based on tier levels with paid features. Fun times https://twittercommunity.com/t/announcing-new-access-tiers-for-the-twitter-api/188728

God, I hate Elon Musk.

[allanf181]

https://cdn.syndication.twimg.com/tweet-result?id=1674865731136020505&lang=en This endpoint is working for non-NSFW tweets

[Wazbat]

I just finished refactoring everything to use the graphQL API in #325, and then an hour later it completely stops working with guest tokens that previously worked fine...

[EthanC]

https://cdn.syndication.twimg.com/tweet-result?id=1674865731136020505&lang=en This endpoint is working for non-NSFW tweets

@allanf181 Do we have something like this for accounts and/or search queries?

[t-wy]

That endpoint is for embeds, so seems that for accounts / search there are no embed API alternatives. Also note that retweet count is not included in that endpoint.

[mesmere]

even if no other options open up, that syndication API would be enough for our use case. please don't give up on fixtweet 🙂

[Wazbat]

It looks like they changed logged out twitter. It no longer just tells you to login and tries to show the tweet, but the endpoint https://twitter.com/i/api/graphql/3XDB26fBve-MmjHaWTUZxA/TweetDetail responds with a 404 and an empty body, even in the browser?

Maybe they'll fix this soon?

[Apocolypto]

vxtwitter seems to work fine at the time of writing this

[Apocolypto]

It looks like they changed logged out twitter. It no longer just tells you to login and tries to show the tweet, but the endpoint https://twitter.com/i/api/graphql/3XDB26fBve-MmjHaWTUZxA/TweetDetail responds with a 404 and an empty body, even in the browser?

Maybe they'll fix this soon?

https://twitter.com/nearcyan/status/1674919375600492546 https://twitter.com/elonmusk/status/1674942336583757825

Musk stated that it'll be unlocked shortly and that "drastic & immediate action was necessary due to EXTREME levels of data scraping."

[Wazbat]

https://twitter.com/nearcyan/status/1674919375600492546 https://twitter.com/elonmusk/status/1674942336583757825

I'm unable to view those tweets on my desktop due to the new rate limits that appeared as of an hour ago lol. Had to use mobile

I'll try to refactor the PR again to use this other endpoint so that we're back with at least basic functionality. This sucks

[allanf181]

cdn.syndication.twimg.com/tweet-result?id=1674865731136020505&lang=en This endpoint is working for non-NSFW tweets

@allanf181 Do we have something like this for accounts and/or search queries?

https://syndication.twitter.com/srv/timeline-profile/screen-name/TwitterDev?showReplies=true there is this one that shows the timeline, but does not give profile details

[allanf181]

https://platform.twitter.com/embed/Tweet.html?id=1675187969420828672 it would be interesting to redirect users to the official embed url so they don't need to log into twitter to read it since the embeds are not working

[slord399]

I don't know it is temp or not but seems fxtwitter start working again?

[Wazbat]

I don't know it is temp or not but seems fxtwitter start working again?

PR branch was deployed directly. It works for sfw tweets, albeit missing some information that that endpoint does not expose. Going to clean up the messy janky implementation while waiting to see how everything unfolds, to see if the much better graphQL API becomes available

[t-wy]

Some summary: https://cdn.syndication.twimg.com/tweet-result?id={tweet_id} endpoint:

• Accept a Tweet ID
• JSON format if Response Code is not 404
• Tweet Object is in another format
  • created_at format is different
  • photos and video keys are used instead of extended_entities
  • No full_text, only text
  • No retweet count is provided

https://syndication.twitter.com/srv/timeline-profile/screen-name/{handle} endpoint:

• Accept a Twitter handle
• HTML format, with JSON response inside a script tag
  • Tweet Objects are located under .props.pageProps.timeline.entries
    • In the same format as the original Guest API
• Seems do not support arguments for loading beyond the oldest tweet provided

[Mazmol]

Works now just a heads up :)

[Wazbat]

Seeing how things are playing out I'm kind losing hope that the old endpoints start working again for logged out users

[cendyne]

It looks like created_timestamp went missing

[Wazbat]

https://syndication.twitter.com/srv/timeline-profile/screen-name/{handle} endpoint:

• Accept a Twitter handle

• HTML format, with JSON response inside a script tag

  • Tweet Objects are located under .props.pageProps.timeline.entries

  • In the same format as the original Guest API

• Seems do not support arguments for loading beyond the oldest tweet provided

Welp, looks like this endpoint has stopped working now too. Right when I was in the process of refactoring things again... At least the other one still does.... For now...

[EthanC]

Lots of good research is being done at https://github.com/zedeus/nitter/issues/919#issuecomment-1619067142

[EthanC]

Using research from https://github.com/zedeus/nitter/issues/919#issuecomment-1619067142, it seems we have a new path to getting the data that FixTweet provided.

https://github.com/12joan/twitter-client/blob/main/src/twitter.ts

What are the chances the User and Tweet APIs could be fully re-populated with this approach?

[dangeredwolf]

@EthanC It does look very promising that we'll be able to restore functionality 100%, it's just a matter of changing how we request guest tokens (to be more like their method) and migrate to GraphQL endpoints

[buxy-xyz]

@dangeredwolf login-only update has been undo'd by twitter, public embeds by twitter.com also work again. am curious if the public guest endpoint survived. (not syndication)

[dangeredwolf]

@buxy-xyz From my understanding, Twitter still removed the conversation api we used before so we still need to move to GraphQL, but tweets appearing while logged out again is at least good news

[Wazbat]

@buxy-xyz The entire update hasn't been undone, as you still can't view profiles while logged out, and tweet replies don't show. It's also using a completely diferent endpoint. Also random requests still 404 when viewing tweets while logged out

Situation still sucks

I've updated the fix PR to use this new endpoint, but user fetching may need another fix

[dangeredwolf]

Interestingly the TweetResultByRestId graphql query works logged out using guest token, but dies when logged in, and TweetDetail works logged in but apparently not while logged out...

[LightBeamOvHe]

I've seen that quoted medias aren't embedded anymore. Should I open another issue ? I thought that the API changes might be at fault so I figured it might be better to directly answer here.

Example here : https://twitter.com/Harada_TEKKEN/status/1676953575342358528 Vx is embedding the video, but not fx. Even the default twitter implementation embeds the quoted video.

[KonoromiHimaries]

Why not try use nitter to scrape from working instances https://github.com/zedeus/nitter/wiki/Instances

[buxy-xyz]

Why not try use nitter to scrape from working instances https://github.com/zedeus/nitter/wiki/Instances

no, scraping an instance that completely ignores nsfw content is not more suitable than the syndication endpoint fx temporarily uses at the moment.

worth taking a look into the recent commits tho. https://github.com/zedeus/nitter/commit/0bc3c153d9b38a3c02f321fb64a375fef6b97e8e

[NicoTheCinderace]

Surprise! fxtwitter is broken ONCE AGAIN thanks to Elon!

[Haxorzz]

What'd he do

[NicoTheCinderace]

What'd he do

whenever you try to replace links that go to twitter from x.com to fxtwitter, it reads as follows:

"Sorry, due to Twitter API changes, accessing some tweets is temporarily broken. We'll be back soon!" More API bullshit.

[Haxorzz]

Damn. Elon be a hard working API engineer

[NicoTheCinderace]

@dangeredwolf @Wazbat

Tagging you two to let you know about the situation.

[dangeredwolf]

@NicoTheCinderace Wait, did Elon change anything else? NSFW Tweets are still broken (sorry), and other Tweets I tried seemed to work okay. Did Twitter/X undo the change? If not, please give me an example URL and I'll see what I can do

[NicoTheCinderace]

@NicoTheCinderace Wait, did Elon change anything else? NSFW Tweets are still broken (sorry), and other Tweets I tried seemed to work okay. Did Twitter/X undo the change? If not, please give me an example URL and I'll see what I can do

Seems like it fixed itself for some of the tweets I tried earlier. I just wish Elon would stop breaking shit so easily.

[dangeredwolf]

@NicoTheCinderace lol even before Elon bought Twitter i've noticed Twitter engineers break specific routes (causing them to always return 404 not found) and since Elon this has become way more common

[dangeredwolf]

Looking at my Sentry it seems this is the corresponding issue... Cannot read properties of undefined (reading 'media') My sourcemaps have unfortunately been broken in Sentry for a while, but maybe Twitter accidentally removed media from the API and fixed it?

But yeah, you can see the time window it was seemingly broken lol

[TomoSuperFan]

Okay, today fxtwitter broke again. I have a feeling this is related to Twitter switching to X.com. @dangeredwolf, you have to update before things get worse.

[NicoTheCinderace]

@NicoTheCinderace Wait, did Elon change anything else? NSFW Tweets are still broken (sorry), and other Tweets I tried seemed to work okay. Did Twitter/X undo the change? If not, please give me an example URL and I'll see what I can do

https://fxtwitter.com/Xezeno1/status/1691100785546027008?t=mTtrwksFxgpuhXDWc7CD0A&s=19

Fxtwitter is broken again. You guys need to fix this before it gets worse. @dangeredwolf @Wazbat