Twitter is removing the Old UI, will this affect Nitter?

Kaleidosium commented 4 years ago

Here's the notice that they gave.

zedeus commented 4 years ago

That is bad news indeed. I have played around with the new public API that uses guest tokens just like the old video API, but I don't know how feasible it's going to be. Will definitely require a large rewrite of the nitter backend.

Kaleidosium commented 4 years ago

I remembered that twitter-scraper supports the new UI, if I remember correctly.

Kaleidosium commented 4 years ago

I'm going to guess this will massively setback all the other features until further notice, Why did you chose to scrape the old UI in the first place anyway?

zedeus commented 4 years ago

I wasn't even aware the new UI existed when I started working on nitter (april/may 2019). Thank you for mentioning twitter-scraper, haven't seen that before, should help me get started.

Kaleidosium commented 4 years ago

Actually nvm, they use the old UI.

zedeus commented 4 years ago

For those who are wondering, I have been working on rewriting part of the backend to use the new JSON-based API, without requiring a developer account. It's fast, loading a full timeline page with profile and photo gallery in just ~400 ms vs Twitter loading the same thing in 1.3-1.5 seconds.

It's not done yet, but I'm making a lot of progress and it should be ready for deployment before June 1st. Here's my current todo list.

Update: All tests are now passing, meaning feature parity has been reached with the old version. On top of that, they pass in ~25-27 seconds instead of 36-46.

High priority, before PR

[x] Clean up code, mass refactoring
[x] Replicate Twitter's short time format
[x] List members
[x] Rewrite thread sorting with better data
[x] Only show pinned tweet on first page
[x] Tweet tombstones (suspended, etc.)

Low priority, after PR

[ ] More aggressive user ID caching
[ ] Improve general error handling code to prevent crashes
[ ] Videos are apparently always available as mp4, new playback preferences?
[x] Show profile pictures in quotes and lists
[ ] Write proper Redis library and use it for caching
[ ] Store tokens in Redis
[x] Media attribution and tags
[ ] Use tweet geolocation/place fields
[ ] Make retweet headers clickable
[ ] Support "Top" search tab
[ ] Hidden tweets
[ ] Broadcasts. This is non-trivial and not already supported
[ ] Bold search queries

Done

[x] Timelines
[x] Multi-timelines
[x] Search
[x] Lists
[x] RSS for the above
[x] Quote media
[x] Preview media in quotes
[x] Promo cards
[x] Parse profile bio mentions/hashtags
[x] Handle request error codes, show suspended
[x] Move search bar below tabs
[x] Consider using hover card API endpoint for profiles, it uses the "old" API with separate limits but same tokens, equally as fast if not faster, allows fetching multiple profiles at a time

zedeus commented 4 years ago

After a few hotfixes, https://nitter.net is now running Nitter v2, and it's going very well so far. Despite an incredible amount of traffic (10-30 tweets per second), it's as fast as a local instance, serving tweets in 2-300 ms and timelines in 3-600 ms. I did some fairly heavy stress tests on some endpoints a couple weeks ago so it should hold up, but I am a little worried.

roytam1 commented 4 years ago

After a few hotfixes, https://nitter.net is now running Nitter v2, and it's going very well so far.

and now getting "Instance Down" message.

Perflyst commented 4 years ago

Just use another instance for the moment https://github.com/zedeus/nitter/wiki/Instances#list-of-public-nitter-instances

zedeus commented 4 years ago

https://nitter.net temporarily broke either due to a full disk causing requests to fail thus repeatedly trying to fetch tokens, or simply because the token strategy is too aggressive for the massive amount of traffic leading to a rate limit. thankfully it didn't last for long, so it's back now.

zedeus commented 4 years ago

Old Twitter is gone and nitter lives on, closing.

ghost commented 2 years ago

@zedeus, It seems like with latest changes on Twitter, Nitter now does NOT show pinned tweets anymore:

profile page: https://nitter.net/app4soft
pinned tweet: https://nitter.net/app4soft/status/1484644315909525505

Something is broken again?

zedeus commented 2 years ago

They started returning "429 too many requests" when using the official web client's bearer token with the timeline endpoint, so I switched it over to using the search endpoint instead. Unfortunately this doesn't include pinned tweets, nor does the alternative bearer token I've been experimenting with. Not sure what to do about this.

zedeus commented 2 years ago

@Symbian9 Nitter now supports pinned tweets again, ended up going with the alternative token and just fetching the pinned tweet separately, works nicely and is still very fast.

ghost commented 2 years ago

Nitter now supports pinned tweets again

Thanks for the hotfix!

ghost commented 2 years ago

@zedeus As an addition, I see that some quoted tweets inside pinned tweet not shown on profile page, BUT shown in single tweet page — is it related or separate issue should be created?

profile page (with pinned tweet): https://nitter.net/solvespace
single tweet page: https://nitter.net/SolveSpace/status/1484163030887841794

zedeus commented 2 years ago

That's a new issue, I'll look into it. Thank you for the example!

zedeus commented 2 years ago

Done, let me know if you find any other issues

ghost commented 2 years ago

That's a new issue, I'll look into it. Thank you for the example! Done, let me know if you find any other issues

Thanks! Yes, I will let you know ;)

I'm just curious does some tweets are "shadow hidden" by Twitter, because I already sew that some separate tweets (not pinned & without any quoted tweets) also not shown in Nitter, but shown on Twitter in a manner that almost nobody else see it.

May it be some sort/new form of "shadow ban"?

zedeus commented 2 years ago

I don't know, it's impossible to check and Nitter can't do anything about it. The tweets you posted work just fine in Nitter. You can't scroll far enough down the timeline to find them since you have so many tweets, but that applies to Twitter as well (this is where it stops loading, same as Nitter):

Searching for them works fine:

ghost commented 2 years ago

FTR, Has you seen next tweet when scrolled my Twitter page?

https://nitter.net/app4soft/status/1484835492399132676

Because on Nitter it is not shown on my page (i.e. https://nitter.net/app4soft, I checked yesterday just after tweeted) and only one of my followers on Twitter tells me that he sew it on Twitter (as notification from followed acount):

https://nitter.net/app4soft/status/1484900838858539008

Searching for them works fine:

Nope, that exact tweet was not "shadow hidden" on Nitter & Twitter ;)

https://nitter.net/app4soft/status/1470923298720686082

It was sad rumor about this one:

https://nitter.net/app4soft/status/1470927148722663425

Few days ago Twitter asked me to pass "robot or human" — I passed it successfully & Twitter verified me as a human... but since then my page lost organic reach. Quoted tweet: https://nitter/app4soft/status/1470107369036296204

But since than I'm checking after tweet something on Twitter does it also shown on Nitter — and there are already few times I not seen some of my tweets on Nitter (which also on Twitter almost has no any reactions, so looks "shadow hidden").

ghost commented 2 years ago

@zedeus, here are another examples of "Tweet not found" on Nitter, BUT visible on Twitter:

zedeus commented 2 years ago

That's because nitter.net is hosted in Germany, where that account is censored.

ghost commented 2 years ago

That's because nitter.net is hosted in Germany

Ok, got it. Its working from another instance ;)

https://nitter.fdn.fr/NotWoofers/status/1495814589510041601#m

@zedeus, BTW, could be there some notification like:

This might be censored by instance hoster.

Try with another one instance instead.

zedeus commented 2 years ago

Unfortunately not, their API returns the same as if the tweet actually didn't exist

jeffalo commented 2 years ago

Hey, I don't know if this is useful but thought I should share:

Twitter still serves the old site to Googlebot and archive.org's user agents. Perhaps that could be used in places where the new website/api is insufficient?

zedeus / nitter