Closed devgaucho closed 1 year ago
This issue could also be called "R.I.P Twitter".
and Musk has posted again.
https://platform.twitter.com/embed/Tweet.html?id=1675260424109928449
and Musk has posted again.
I don't wanna post offtopic but i just loaded this tweet in normal twitter and loading the replies made me hit the new limit. This is comedy gold.
Maybe not of any use, but this extension is helping tweets load on chromium based browsers for logged in users. Perhaps this extension's source code could lead a clue? https://github.com/dimdenGD/OldTwitter
I don't see it mentioned but tweetdeck for some reason lets you to view posts and comment when above the limit, you cant see profiles and their contents tho, only your feed.
@0xAB51NTH since github has no rate limit you can ctrl+f tweetdeck and see it was mentioned many times... only a few posts above yours
I found an alternative in a post on the internet:
https://syndication.twitter.com/srv/timeline-profile/screen-name/[name]
If he wanted to hide the data, I think he made a mistake!
@mthmcalixto
If this doesn't get reversed within the next couple of weeks I'll switch to using embeds as an emergency measure, but for now I'm just going to wait it out.
I appreciate the support everyone, it's amazing to see how many people use my little pet project. Nitter isn't going down without a fight.
Also using syndication addr is already discussed here several times...
@mthmcalixto
If this doesn't get reversed within the next couple of weeks I'll switch to using embeds as an emergency measure, but for now I'm just going to wait it out. I appreciate the support everyone, it's amazing to see how many people use my little pet project. Nitter isn't going down without a fight.
Also using syndication addr is already discussed here several times...
Any fork that takes the data?
It seems like there's still a way to see tweets without an account.
When posting a Twitter link on discord, there used to be a preview but it stopped working as soon as they made the change, I guess discord crawler is considered as "not logged in". However, when posting a link preceded with "vx" (i.e. https://vxtwitter.com/status/xxxx), there's still a preview. I'm not sure how they achieved that though.
I found an alternative in a post on the internet:
https://syndication.twitter.com/srv/timeline-profile/screen-name/[name]
If he wanted to hide the data, I think he made a mistake!
This... this is beautiful 🥹
#!/usr/bin/python3
import requests
import re
import urllib
url = "https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk"
with urllib.request.urlopen(url) as response:
encoding = response.info().get_param('charset', 'utf8')
html = response.read().decode(encoding)
result = re.search('script id="__NEXT_DATA__" type="application\/json">([^>]*)<\/script>', html)[1]
print(result)
Returns
{
"props": {
"pageProps": {
"contextProvider": {
"features": {},
"scribeData": {
"client_version": null,
"dnt": false,
"widget_id": "embed-0",
"widget_origin": "",
"widget_frame": "",
"widget_partner": "",
"widget_site_screen_name": "",
"widget_site_user_id": "",
"widget_creator_screen_name": "",
"widget_creator_user_id": "",
"widget_iframe_version": "bb06567:1687853948269",
"widget_data_source": "screen-name:elonmusk",
"session_id": ""
},
"messengerContext": {
"embedId": "embed-0"
},
"hasResults": true,
"lang": "en",
"theme": "light"
},
"lang": "en",
"maxHeight": null,
"showHeader": true,
"hideBorder": false,
"hideFooter": false,
"hideScrollBar": false,
"transparent": false,
"timeline": {
"entries": [
{
"type": "tweet",
"entry_id": "tweet-1675390796718014464",
"sort_index": "1675390796718014464",
"content": {
"tweet": {
"id": 0,
"location": "",
"conversation_id_str": "1675187969420828672",
"created_at": "Sun Jul 02 06:27:47 +0000 2023",
"display_text_range": [
0,
73
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": [],
"media": []
},
"favorite_count": 35676,
"favorited": false,
"full_text": "In yet another exercise in irony, this post achieved a record view count!",
"id_str": "1675390796718014464",
"in_reply_to_name": "elonmusk",
"in_reply_to_screen_name": "elonmusk",
"in_reply_to_status_id_str": "1675187969420828672",
"in_reply_to_user_id_str": "44196397",
"lang": "en",
"permalink": "/elonmusk/status/1675390796718014464",
"possibly_sensitive": false,
"quote_count": 6310,
"reply_count": 7417,
"retweet_count": 3055,
"retweeted": false,
"text": "In yet another exercise in irony, this post achieved a record view count!",
"user": {
"blocking": false,
"created_at": "Tue Jun 02 20:12:29 +0000 2009",
"default_profile": false,
"default_profile_image": false,
"description": "",
"entities": {
"description": {
"urls": []
},
"url": {}
},
"fast_followers_count": 0,
"favourites_count": 26835,
"follow_request_sent": false,
"followed_by": false,
"followers_count": 146178794,
"following": false,
"friends_count": 341,
"has_custom_timelines": false,
"highlightedLabel": {
"badge": {
"url": "https://pbs.twimg.com/profile_images/1488548719062654976/u6qfBBkF_bigger.jpg"
},
"description": "Twitter",
"userLabelType": "BusinessLabel",
"userLabelDisplayType": "Badge"
},
"id": 0,
"id_str": "44196397",
"is_translator": false,
"listed_count": 125551,
"location": "",
"media_count": 1605,
"name": "Elon Musk",
"normal_followers_count": 146178794,
"notifications": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1576183471",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg",
"protected": false,
"screen_name": "elonmusk",
"show_all_inline_media": false,
"statuses_count": 27556,
"time_zone": "",
"translator_type": "none",
"url": "",
"utc_offset": 0,
"verified": false,
"withheld_in_countries": [],
"withheld_scope": "",
"is_blue_verified": true
}
}
}
},
{
"type": "tweet",
"entry_id": "tweet-1675365001836593154",
"sort_index": "1675365001836593154",
"content": {
"tweet": {
"id": 0,
"location": "",
"conversation_id_str": "1675365001836593154",
"created_at": "Sun Jul 02 04:45:17 +0000 2023",
"display_text_range": [
0,
89
],
"entities": {
[...]
Actual full result : https://bin.socialspill.com/ariruqiq.json
The Musk strikes again. I'm genuinely surprised that Tesla and SpaceX are still afloat with such a moron of a CEO. At this point I start thinking that creating not so sophisticated Twitter alternative and deploying it to the cheapest DO droplet would be a viable business plan. I dunno, it's just frustration at this point. A whatever-billion-dollar company can't pay for it's hosting, bruh.
i don't particularly know how nitter scrapes, but i'm able to see tweets and profiles in my browser by setting the useragent to "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Actually I tried that half an hour ago and it didn't work either. But somehow, Google is still able to get new tweets to the news feed. Sus.
I found an alternative in a post on the internet:
https://syndication.twitter.com/srv/timeline-profile/screen-name/[name]
If he wanted to hide the data, I think he made a mistake!
That url is used in the embedded widgets, which continue to work (for now).
Tweet Documentation: https://developer.twitter.com/en/docs/twitter-for-websites/embedded-tweets/overview Example: https://platform.twitter.com/embed/Tweet.html?id=463440424141459456
Profile Timeline Documentation: https://developer.twitter.com/en/docs/twitter-for-websites/timelines/guides/profile-timeline Example: https://syndication.twitter.com/srv/timeline-profile/screen-name/TwitterDev
List Timeline Documentation: https://developer.twitter.com/en/docs/twitter-for-websites/timelines/guides/list-timeline Example: https://syndication.twitter.com/srv/timeline-list/screen-name/TwitterDev/slug/national-parks
The Musk strikes again. I'm genuinely surprised that Tesla and SpaceX are still afloat with such a moron of a CEO. At this point I start thinking that creating not so sophisticated Twitter alternative and deploying it to the cheapest DO droplet would be a viable business plan. I dunno, it's just frustration at this point. A whatever-billion-dollar company can't pay for it's hosting, bruh.
Tesla is on its way
The Musk strikes again. I'm genuinely surprised that Tesla and SpaceX are still afloat with such a moron of a CEO.
There are competent people (executives etc.)between him and day-to-day operation of those two companies. With Twitter it seem he is shouting into senior engineers's ears.
i don't particularly know how nitter scrapes, but i'm able to see tweets and profiles in my browser by setting the useragent to "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Actually I tried that half an hour ago and it didn't work either. But somehow, Google is still able to get new tweets to the news feed. Sus.
Twitter can also just allow the traffic only from the well known IPs of google bot. It's publicly listed
i don't particularly know how nitter scrapes, but i'm able to see tweets and profiles in my browser by setting the useragent to "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Actually I tried that half an hour ago and it didn't work either. But somehow, Google is still able to get new tweets to the news feed. Sus.
Twitter can also just allow the traffic only from the well known IPs of google bot. It's publicly listed
I mean... It's totally an option for them
__NEXT_DATA__
Yep! got it working like this also.
Grab the content of the script tag, do some decoding and return it as JSON response to my request...
Dirty prototype in node-red you can import in to visually see whats going on:
[{"id":"6e6bc092885be623","type":"json","z":"f6f2187d.f17ca8","name":"","property":"payload","action":"","pretty":false,"x":1070,"y":340,"wires":[["8b176c7d37cb03ea","fc591f610fe3a4e1"]]},{"id":"7e5d901ca520f39b","type":"function","z":"f6f2187d.f17ca8","name":"function 1","func":"const parsedPayload = msg.payload\nconst firstPayload = parsedPayload[0];\nconst decodedPayload = he.decode(firstPayload);\nmsg.payload = decodedPayload\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"he","module":"he"}],"x":900,"y":340,"wires":[["6e6bc092885be623"]]},{"id":"8b176c7d37cb03ea","type":"debug","z":"f6f2187d.f17ca8","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1240,"y":340,"wires":[]},{"id":"fc591f610fe3a4e1","type":"http response","z":"f6f2187d.f17ca8","name":"","statusCode":"","headers":{},"x":1170,"y":440,"wires":[]},{"id":"081c3f47b9f93a7b","type":"html","z":"f6f2187d.f17ca8","name":"","property":"payload","outproperty":"payload","tag":"script#__NEXT_DATA__","ret":"html","as":"single","x":690,"y":340,"wires":[["7e5d901ca520f39b"]]},{"id":"87fa9b636188d38b","type":"http request","z":"f6f2187d.f17ca8","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://syndication.twitter.com/srv/timeline-profile/screen-name/{{{req.params.user}}}","tls":"","persist":true,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":430,"y":340,"wires":[["081c3f47b9f93a7b"]]},{"id":"10385e3c19480452","type":"inject","z":"f6f2187d.f17ca8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":220,"y":340,"wires":[["87fa9b636188d38b"]]},{"id":"fd695068b87f4dc5","type":"http in","z":"f6f2187d.f17ca8","name":"","url":"/twitter/:user","method":"get","upload":false,"swaggerDoc":"","x":240,"y":440,"wires":[["87fa9b636188d38b"]]}]
Google needs to start applying their own rules to these big sites.
If the average site displays different content to Google and to users, it gets penalised/removed from search results. They must do the same thing to Twitter and Reddit. Hit them where it hurts.
They must do the same thing to Twitter and Reddit. Hit them where it hurts.
They were crying because subreddit blackouts made search results worse, that's surely never gonna happen.
I found an alternative in a post on the internet:
https://syndication.twitter.com/srv/timeline-profile/screen-name/[name]
If he wanted to hide the data, I think he made a mistake!
I think one of the staff or Elon is reading our comments lol
bird.makeup still working
It is trivial to completely defeat scrapers with no funding that power nitter. Nitter operates entirely at Twitter's pleasure.
I found an alternative in a post on the internet:
https://syndication.twitter.com/srv/timeline-profile/screen-name/[name]
If he wanted to hide the data, I think he made a mistake!
Now appears to be gone as well. Not working anymore.
Now appears to be gone as well. Not working anymore.
It started working again for me just now. EDIT: It stopped again. It fluctuates :\
That particular endpoint seems to be appearing / disappearing in the last few hours. I'm guessing someone's actively working to remove it.
RIP syndication 🪽
I found an alternative in a post on the internet: https://syndication.twitter.com/srv/timeline-profile/screen-name/[name] If he wanted to hide the data, I think he made a mistake!
This... this is beautiful 🥹
#!/usr/bin/python3 import requests import re import urllib url = "https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk" with urllib.request.urlopen(url) as response: encoding = response.info().get_param('charset', 'utf8') html = response.read().decode(encoding) result = re.search('script id="__NEXT_DATA__" type="application\/json">([^>]*)<\/script>', html)[1] print(result)
Returns
{ "props": { "pageProps": { "contextProvider": { "features": {}, "scribeData": { "client_version": null, "dnt": false, "widget_id": "embed-0", "widget_origin": "", "widget_frame": "", "widget_partner": "", "widget_site_screen_name": "", "widget_site_user_id": "", "widget_creator_screen_name": "", "widget_creator_user_id": "", "widget_iframe_version": "bb06567:1687853948269", "widget_data_source": "screen-name:elonmusk", "session_id": "" }, "messengerContext": { "embedId": "embed-0" }, "hasResults": true, "lang": "en", "theme": "light" }, "lang": "en", "maxHeight": null, "showHeader": true, "hideBorder": false, "hideFooter": false, "hideScrollBar": false, "transparent": false, "timeline": { "entries": [ { "type": "tweet", "entry_id": "tweet-1675390796718014464", "sort_index": "1675390796718014464", "content": { "tweet": { "id": 0, "location": "", "conversation_id_str": "1675187969420828672", "created_at": "Sun Jul 02 06:27:47 +0000 2023", "display_text_range": [ 0, 73 ], "entities": { "user_mentions": [], "urls": [], "hashtags": [], "symbols": [], "media": [] }, "favorite_count": 35676, "favorited": false, "full_text": "In yet another exercise in irony, this post achieved a record view count!", "id_str": "1675390796718014464", "in_reply_to_name": "elonmusk", "in_reply_to_screen_name": "elonmusk", "in_reply_to_status_id_str": "1675187969420828672", "in_reply_to_user_id_str": "44196397", "lang": "en", "permalink": "/elonmusk/status/1675390796718014464", "possibly_sensitive": false, "quote_count": 6310, "reply_count": 7417, "retweet_count": 3055, "retweeted": false, "text": "In yet another exercise in irony, this post achieved a record view count!", "user": { "blocking": false, "created_at": "Tue Jun 02 20:12:29 +0000 2009", "default_profile": false, "default_profile_image": false, "description": "", "entities": { "description": { "urls": [] }, "url": {} }, "fast_followers_count": 0, "favourites_count": 26835, "follow_request_sent": false, "followed_by": false, "followers_count": 146178794, "following": false, "friends_count": 341, "has_custom_timelines": false, "highlightedLabel": { "badge": { "url": "https://pbs.twimg.com/profile_images/1488548719062654976/u6qfBBkF_bigger.jpg" }, "description": "Twitter", "userLabelType": "BusinessLabel", "userLabelDisplayType": "Badge" }, "id": 0, "id_str": "44196397", "is_translator": false, "listed_count": 125551, "location": "", "media_count": 1605, "name": "Elon Musk", "normal_followers_count": 146178794, "notifications": false, "profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1576183471", "profile_image_url_https": "https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg", "protected": false, "screen_name": "elonmusk", "show_all_inline_media": false, "statuses_count": 27556, "time_zone": "", "translator_type": "none", "url": "", "utc_offset": 0, "verified": false, "withheld_in_countries": [], "withheld_scope": "", "is_blue_verified": true } } } }, { "type": "tweet", "entry_id": "tweet-1675365001836593154", "sort_index": "1675365001836593154", "content": { "tweet": { "id": 0, "location": "", "conversation_id_str": "1675365001836593154", "created_at": "Sun Jul 02 04:45:17 +0000 2023", "display_text_range": [ 0, 89 ], "entities": { [...]
Actual full result : https://bin.socialspill.com/ariruqiq.json
guilty? 🤦♂️
@sunlightspectre
One wonders what will happen when these self-inflicted stab-wounds deplete 'profit' rather then generate it?.. Removing access isn't going to make people want to use a site more. And it certainly won't inflate user numbers, which I'd imagine is the end goal.
C'est la vie.
I'm glad I got to experience the internet as it was.
Totally agreed. Login-walling even to a public post (set public by the user) is the most stupidest move I ever seen. Because it is no longer public, this won't be visible to search engine indexing/crawling. This means any user searching for X won't be seeing any twitter posts on, say google search results. Tweeters will now have a hard time gaining an audience, especially artists.
@sunlightspectre
One wonders what will happen when these self-inflicted stab-wounds deplete 'profit' rather then generate it?.. Removing access isn't going to make people want to use a site more. And it certainly won't inflate user numbers, which I'd imagine is the end goal. C'est la vie. I'm glad I got to experience the internet as it was.
Totally agreed. Login-walling even to a public post (set public by the user) is the most stupidest move I ever seen. Because it is no longer public, this won't be visible to search engine indexing/crawling. This means any user searching for X won't be seeing any twitter posts on, say google search results. Tweeters will now have a hard time gaining an audience, especially artists.
googlebot have free pass to twitter
@devgaucho can that be used to bypass login walls as well as archiving?
Guys the syndication.twitter.com embed method works again
Guys the syndication.twitter.com embed method works again
Thanks for the information.!
List Timeline Documentation: https://developer.twitter.com/en/docs/twitter-for-websites/timelines/guides/list-timeline Example: https://syndication.twitter.com/srv/timeline-list/screen-name/TwitterDev/slug/national-parks
This is probably a dumb question, but what if a list has a space in the name? I can't get my list to show up this way. I tried a dash (-) an underscore and a regular space/ %20 and it all shows up as the list not existing. Not really sure about the rules about case sensitivity as well. Is there a maybe a way to extract the proper list name and user name from the lists' ID?
List Timeline Documentation: https://developer.twitter.com/en/docs/twitter-for-websites/timelines/guides/list-timeline Example: https://syndication.twitter.com/srv/timeline-list/screen-name/TwitterDev/slug/national-parks
This is probably a dumb question, but what if a list has a space in the name? I can't get my list to show up this way. I tried a dash (-) an underscore and a regular space/ %20 and it all shows up as the list not existing. Not really sure about the rules about case sensitivity as well.
When you hover go into a list on the website, there's an ID showing, simply replace the id instead of trying to find a working list-name. You of course have to match the screen-name associated to that list.
simply replace the id instead of trying to find a working list-name.
Are you sure? I tried it with the example national parks links, where the list id shows up as 715919216927322112 (i assume this is the id, am i correct? The lists url is https://twitter.com/i/lists/715919216927322112 so i assume the number is the id) but https://syndication.twitter.com/srv/timeline-list/screen-name/TwitterDev/slug/715919216927322112 doesn't embed the list. I'm pretty sure im being an idiot here, but i don't get it. EDIT: Ok i just tried it on a random list instead of the national-parks one used in example and copying the ID does work, thank you so much <3
Yup it doesn't work for this one but I tried for one of my list and it weirdly it works
For example I clicked on this list on the website : and then created this link : https://syndication.twitter.com/srv/timeline-list/screen-name/JeandronMathieu/slug/1434116675968831497 And it works. But I agree, sometimes it doesn't. Not sure why.
Tried in on a different list and it worked just as you said, sorry for being dumb, thank you Write. Weird it doesn't work for the list used in the example, though Edit: Aand it's broken again, it loads the list correctly (as in, it reads the list name and author) but doesn't load the tweets, as if the list was empty. Elon must be playin whack a mole with us.
Also it seems you can quite easily get your embeded link with the use of this page : https://publish.twitter.com/?query=https%3A%2F%2Ftwitter.com%2Fi%2Flists%2F1434116675968831497&widget=Timeline and just pasting your list url.
https://cdn.discordapp.com/attachments/874455648883052544/1125145584798548119/image.png
"Parse HTML with regex" I feel attacked
Has anyone built a tool to get an rss from a syndication link?
Has anyone built a tool to get an rss from a syndication link?
@TempUser13 dvikan has linked to rss-bridge in this issue 🙈 https://github.com/zedeus/nitter/issues/919#issuecomment-1615253896
Has anyone built a tool to get an rss from a syndication link?
@TempUser13 dvikan has linked to rss-bridge in this issue see_no_evil #919 (comment)
Looks like this instance (rss-bridge.org) is modified to be able to parse the syndication (git.twitter-fix-2.be59a1e). I am running my own instance of rss-bridge with latest docker image (git.master.d8bc015). This is not working for syndication (yet).
Looks like this instance (rss-bridge.org) is modified to be able to parse the syndication (git.twitter-fix-2.be59a1e). I am running my own instance of rss-bridge with latest docker image (git.master.d8bc015). This is not working for syndication (yet).
the twitter issues might resolve itself by its own so im waiting a bit before merging to master.
do you want it merged asap?
do you want it merged asap?
Thanks for offering, but I am not in a hurry. I switched a few of my RSS subscriptions now to rss-bridge.org instead of my own instance.
Having this fix merged would just be of interest for me, if also media content would be available via syndication, which seems not to be the case. Most of my subscriptions contain media, so this workaround will not work for me, regardless if it is on your or my instance.
the twitter issues might resolve itself by its own [...]
Fingers crossed!
I wonder, will guys from https://github.com/DIYgod/RSSHub/issues/12732 switch to syndication too? I used their instance for more than a year already. Currently some of feeds are working (probably cached), others are not.
https://techcrunch.com/2023/06/30/twitter-now-requires-an-account-to-view-tweets/
the nitter crawler will need to be recreated...