Some tweets labeled as non existant

[Kyrela]

The title is pretty explicit: on some specific tweets, FixTweet doesn't find the tweet. The first link problem is reproductible at the time I'm writing this, but reposting the second one cause the embed to properly displays, so I guess this isn't a permanent problem...? Don't know, maybe you'll have a clue

https://twitter.com/SheepEsports_FR/status/1719059184191168997 https://x.com/udpj2/status/1718652251919802880?s=20

[dangeredwolf]

It seems whatever happened with those particular URLs got cleared up in the time between then and now as they both work.

Twitter's upstream API is preeeeetty unstable these days and it's been pretty much a game of whack-a-mole trying to make my own implementation of it more reliable because of how often their servers have issues. We very rarely had upstream issues until we were forced onto Twitter's GraphQL API by them randomly killing the REST API 2.0 endpoints we were previously relying on.

The good news is that I'm working on some changes in the thread-api branch that should (hopefully) help improve overall reliability as well.

It's worth mentioning that fxtwitter.com in the past month kinda blew up and we are handling nearly 4x the requests per day now than we were a month ago (from ~500k per day 30 days ago to ~2M per day today). From a performance standpoint we can handle a lot more requests than even that, but the bottleneck is always how fat of a pipe I can build to extract data from X/Twitter's API. There's certainly a lot more work to be done to make FixTweet more reliable.

[Kyrela]

Good to know, thank you!

[Tempystral]

I'm using the fxtwitter api for a bot of my own and I'm noticing we get 200 roughly half the time and 404 the other half. You can see it for yourself if you load up the api URL for a tweet and do a hard refresh a few times.

I absolutely understand you're under a lot of load right now though, by all means take your time. It's not your fault Twitter is busted.

[dangeredwolf]

I managed to reproduce it returning 404 with logging. I think a recent change I made to FixTweet/elongator might be to blame as it was assuming it shouldn't retry if Twitter returns a downstream error (which happens a lot with their GraphQL server...). Hopefully this'll make a difference to ensuring the API is more reliable? I'm still making other changes to try to improve API reliability so hang tight for a bit.

[dangeredwolf]

I merged the thread-api changes into main now, Hopefully this will increase API stability!

[Kyrela]

Thank you! I'll comment this issue if the problem re-appears.

[dangeredwolf]

I'll close the issue for now, but it can be reopened if issues arise again.

[dangeredwolf]

Reopening this because it does (quite rarely) still happen and I haven't gotten to the bottom of why

[slord399]

Seems happen at increased rate since yesterday. (maybe something messed up by storm on US last week...)

[Kyrela]

Example of tweets labeled as non existant:

• https://x.com/Aiekillu/status/1793969340750131494
• https://x.com/CerfiaFR/status/1812254615779316019

[Kyrela]

Problem still exists, even without using translation. Updated my last comment to add another example.

[dangeredwolf]

FixTweet in general had some hiccups earlier today due to the request volume being 4x normal because of world events: https://www.cnn.com/2024/07/13/politics/trump-injured-pennsylvania-rally/index.html

You can even see exactly when it happened, oops.

Tweaked some stuff, should be better now. Still need to figure out why translations are having issues.

[Kyrela]

Oh, ok, mb. I hadn't made the connection between the two events. Glad it's temporary. Thanks for your reactivity, as always.