search

dylanpdx / BetterTwitFix

Fix Twitter video embeds in Discord (and Telegram!)
Do What The F*ck You Want To Public License
1.17k stars 48 forks source link

Failed to scan your tweet #74

Closed Ashuraman666 closed 1 year ago

Ashuraman666 commented 1 year ago

As of today, no matter what tweet I try to post on discord with vx, I get the same error for all of them Sem Título This happens on every tweet

Nama commented 1 year ago

This should happen only on some NSFW tweets.

Twitter changed their API restrictions and doesn't allow access to some NSFW tweets.

Ashuraman666 commented 1 year ago

This should happen only on some NSFW tweets.

Twitter changed their API restrictions and doesn't allow access to some NSFW tweets.

Is there any way to know which NSFW tweets it blocks and which it doesn't?

dylanpdx commented 1 year ago

Can I have an example of a non-working tweet so I can add it to my tests?

Edit: Was able to find one

dylanpdx commented 1 year ago

It seems like Twitter is returning 404 (Not found) for some tweets, with a message of "Sorry, that page does not exist." I'm not sure if this is intentional or as a result of some API limit. I'd expect another error code if this was an API limitation

Ashuraman666 commented 1 year ago

It seems like Twitter is returning 404 (Not found) for some tweets, with a message of "Sorry, that page does not exist." I'm not sure if this is intentional or as a result of some API limit. I'd expect another error code if this was an API limitation

Here's the thing, I tried the same tweets with a different tweet fix (The one where you replace the i in Twitter with an x) and that one did work

dylanpdx commented 1 year ago

I have a possible workaround in the works, although I'm not sure how long before Twitter blocks that too

dylanpdx commented 1 year ago

I've deployed a fix, please double check that the issue is fixed.

It's not a perfect fix since it can fail at times, but this is the most I can do as a temporary patch

Ashuraman666 commented 1 year ago

I've deployed a fix, please double check that the issue is fixed.

It's not a perfect fix since it can fail at times, but this is the most I can do as a temporary patch

Seems to be working for images for the most part but for videos it still gives the same "Tweet not found" error

dylanpdx commented 1 year ago

Might be due to Discord's caching of some links, it should start updating them in around 30 minutes or so Workaround at the moment is to somehow change the URL so that Discord fetches a fresh copy, adding ?a at the end usually works

Scrxtchy commented 1 year ago

It seems like Twitter is returning 404 (Not found) for some tweets, with a message of "Sorry, that page does not exist."

This is the standard error for adult-restricted tweets via the 1.1 api, you'll get these if you're account isn't old enough for adult content image image

Scrxtchy commented 1 year ago

Is there any way to know which NSFW tweets it blocks and which it doesn't?

Twitter store a possibly_sensitive as a part of a Tweet object in the json, if this is true, you'll get the error, opening a tweet in a private tab/logged out is a good way to tell since with this change twitter should be restricting Adult Restricted tweets in your region. It has been restricted in Australia for some time already

dylanpdx commented 1 year ago

It seems like Twitter is returning 404 (Not found) for some tweets, with a message of "Sorry, that page does not exist."

This is the standard error for adult-restricted tweets via the 1.1 api, you'll get these if you're account isn't old enough for adult content image image

In the past, I seem to remember it returning a message similar to what Twitter shows while logged out image but I might be mis-remembering. Either way, this makes it annoying to check if a tweet simply doesn't exist or is just age restricted. Some other 4XX error (403?) would probably be more informative.

I also noticed that this change affected a lot of Twitter's own tools as well, i.e https://publish.twitter.com fails for NSFW tweets.

I'm still unsure if this was a deliberate change or some mistake...

Scrxtchy commented 1 year ago

In the past, I seem to remember it returning a message similar to what Twitter shows while logged out

No, that's the one, This is the error you get when logged in image

Not important to the issue, but I made a thread about my experience with it. I sent you an email earlier regarding this issue but I'm fine with discussing it here henceforth, now that we have a better understanding. But my understanding is that this seems like an intentional change

I also noticed that this change affected a lot of Twitter's own tools as well, i.e https://publish.twitter.com fails for NSFW tweets.

at least the error message here is a lot more meaningful, the api also is used on tweetdeck without leaving the page, it's just weird seeing an api controller on the platform.twitter.com domain 23-04-09_06-58-38-Developer_Tools_—_()_TweetDeck_—_httpstweetdeck tw

dylanpdx commented 1 year ago

I sent you an email earlier regarding this issue but I'm fine with discussing it here henceforth

Oh, your email got sent to spam, I'm so sorry I wasn't able to see it! I'm fine discussing this here from now on

vxTwitter is hosted in the US, so they probably rolled that change out today to that region. I guess switching regions could work, but since Discord's link unfurler servers are also in the US it would probably add latency, and not to mention that Twitter has a high chance of switching it off there as well

How I see it at the moment, there's no way to get info about NSFW tweets using any public API, authentication is required. Both v1.1 and v2 API versions return a 404, as do the internal API's like the GraphQL and cdn.syndication.twimg.com. There might be another obscure public API out there but I haven't been able to find anything regarding that.

Overall a pretty upsetting change but I can see the reasoning behind it. This may be related to the new API pricing as well since it went into effect just recently

Delphox commented 1 year ago

Yea I can confirm it's not working for videos. Here's an example for a video that fails (its not NSFW but gets detected as such): https://vxtwitter.com/TheFunny_mp4/status/1529475039833337858

dylanpdx commented 1 year ago

Looking into it - Thanks for the example tweet

dylanpdx commented 1 year ago

This should be fixed now- Keep in mind Discord might take a while to refresh their link cache (usually up to 30 minutes)

Scrxtchy commented 1 year ago

I deleted my earlier post since I misread the comment, apologies

Delphox commented 1 year ago

That was quick, thanks! Just one small issue, some of those videos are embedding super tiny. Here's another example tweet: https://vxtwitter.com/NightShiftNyx/status/1598396995877699602 (also not NSFW but detected as such, but keep in mind this one is from a NSFW account)

dylanpdx commented 1 year ago

This should now be fixed as well! Sorry about that, my mistake

Scrxtchy commented 1 year ago

I have never felt as much solidarity in this commit 6244c875c47061b49bf1516bdfcede1720f13776 anywhere else before in programming

image

dylanpdx commented 1 year ago

My brain is currently at half capacity haha, I'll (hopefully) put in a cleaner fix for this when I can, and maybe add some test cases for it as well

Delphox commented 1 year ago

Doesn't seem like the last fix worked... NSFW embeds are still small, including the last example one. I tried the ?a trick to make sure.

Delphox commented 1 year ago

Oh nevermind I tried changing the username in the url instead and it actually forced Discord to re-fetch it and it's fixed. So it was discord's cache messing it up and just adding ?aaa at the end isn't enough to force a true refetch. Thanks again!