search

RSS-Bridge / rss-bridge

The RSS feed for websites missing it
https://rss-bridge.org/bridge01/
The Unlicense
7.27k stars 1.03k forks source link

[TwitterBridgeV2/TwitterBridge] Twitter no longer support free access to the Twitter API, starting February 9 #3239

Closed csisoap closed 1 year ago

csisoap commented 1 year ago

As stated on Twitter Developer account,

Starting February 9, we will no longer support free access to the Twitter API, both v2 and v1.1. A paid basic tier will be available instead

We’ll be back with more details on what you can expect next week.

Source: https://twitter.com/TwitterDev/status/1621027418680229888

scottsweb commented 1 year ago

I believe something just changed on the API side. Bridge is returning errors now. 

Type: HttpException
Code: 401
Message: https://api.twitter.com/1.1/users/show.json?screen_name={username} resulted in `401 Unauthorized`
File: lib/contents.php
Line: 194
CeruleanSky commented 1 year ago

Tweetdeck.twitter.com just uses 1.1 endpoints afaik so if it works past the 9th the endpoints it uses to get data could be used.
Also https://github.com/zedeus/nitter/issues/783 says they won't likely be affected especially after it finishes its migration to graphql api which is described as a simple RPC Api with a graphql route. RSSHub also continues to work as of right now https://github.com/DIYgod/RSSHub/tree/master/lib/v2/twitter/web-api

headers:

Authorization: Bearer xxxx 
X-Twitter-Auth-Type: OAuth2Session
X-Csrf-Token: xxxx
X-Twitter-Client-Version: Twitter-TweetDeck-blackbird-chrome/4.0.220811153004 web/

cookies:


auth_token=xxxx
co=ch
daa=1
ads_prefs=""xxxx""
dnt=1
ct0=xxxx
twid=u"%"xxxx
night_mode=2
guest_id=v1"%"xxxx
tweetdeck_version=legacy
d_prefs=xxxx
eu_cn=1
lang=en
_twitter_sess=xxxx```
dvikan commented 1 year ago

As far as I can tell, also the regular TwitterBridge stopped working today.

@arnd-s

quickwick commented 1 year ago

It's not February 9th yet, no matter what time-zone you're in, but this seems about par for the course for Twitter these days. Unless something changes on the Twitter side, this is probably the end of my involvement with the TwitterBridgeV2. There's no way I'm paying for Twitter API v2.0 access, and the Twitter API v1.1 isn't an option for me (even if free access continues somehow) because it dropped support for Sensitive Content tweets (hence my creation of the TwitterBridgeV2 in the first place).

I guess now's a good time to look at Nitter RSS feeds.

jutsh65 commented 1 year ago

I just installed TwitterBridge last week. It worked great for a few days, then this morning around 6am PT I started seeing the following error when querying a timeline by username:

Exception: Failed to make api call: /1.1/users/show.json\nHTTP Status: 401\nErrormessage: 401 Unauthorized in /var/www/html/rss-bridge/lib/error.php:24\nStack trace:\n#0 /var/www/html/rss-bridge/lib/error.php(42): returnError()\n#1 /var/www/html/rss-bridge/bridges/TwitterBridge.php(643): returnServerError()\n#2 /var/www/html/rss-bridge/bridges/TwitterBridge.php(219): TwitterBridge->makeApiCall()\n#3 /var/www/html/rss-bridge/actions/DisplayAction.php(134): TwitterBridge->collectData()\n#4 /var/www/html/rss-bridge/index.php(24): DisplayAction->execute()\n#5 {main}

So far it does not seem intermittent but continuous.

dvikan commented 1 year ago

I've created a client that speaks to the api that is still publicly accessible. I've not taken a look at all the bridge configuration.

Testers needed: https://rss-bridge.org/twitter_test/

Pepepy commented 1 year ago

That seems to work for now. Is this change available in the github repository?

arnd-s commented 1 year ago

Yeah, looks like they deactivated the api call to get guest tokens for authentication.

I'm currently lacking time do look into a way to circumvent this or to adjust the necessary calls inside rss-bridge.

em92 commented 1 year ago

Testers needed: https://rss-bridge.org/twitter_test/

HttpException: https://twitter.com/i/api/graphql/hc-pka9A7gyS3xODIafnrQ/UserByScreenName?variables=%7B%22screen_name%22%3A%22odootechnical%22%2C%22withHighlightedLabel%22%3Atrue%7D resulted in `403 Forbidden` in lib/contents.php line 194

index.php(7): RssBridge->main()
lib/RssBridge.php(15): RssBridge->run()
lib/RssBridge.php(88): DisplayAction->execute()
actions/DisplayAction.php(133): TwitterBridge->collectData()
bridges/TwitterBridge.php(231): TwitterClient->fetchUserTweets()
lib/TwitterClient.php(21): TwitterClient->fetchUserInfoByScreenName()
lib/TwitterClient.php(65): getContents()
lib/contents.php(194)

Query string: action=display&bridge=TwitterBridge&context=By+username&u=odootechnical&format=Html
Version: dev.2022-06-14 (git.twitter-fix-1.1412903)
Os: Linux
PHP version: 7.4.33
triatic commented 1 year ago

For those who haven't got an API key yet, it might be worth getting one on the off chance that free developer accounts are grandfathered in or at least have a grace period before payment is required.

quickwick commented 1 year ago

For those who haven't got an API key yet, it might be worth getting one on the off chance that free developer accounts are grandfathered in or at least have a grace period before payment is required.

I have a free developer account, and it stopped working yesterday. I just checked the developer portal, it still shows my existing Project at the Essential Tier, with a listed Cost of "free", and 14,320 tweets pulled out of the 500k limit. But I get errors when I try to use the API with that developer account.

jutsh65 commented 1 year ago

For those who haven't got an API key yet, it might be worth getting one on the off chance that free developer accounts are grandfathered in or at least have a grace period before payment is required.

I have a free developer account, and it stopped working yesterday. I just checked the developer portal, it still shows my existing Project at the Essential Tier, with a listed Cost of "free", and 14,320 tweets pulled out of the 500k limit. But I get errors when I try to use the API with that developer account.

Mine is still working, but I only pull user tweet timelines to convert to RSS (using rssbox). Using this as my coal mine canary.

triatic commented 1 year ago

Mine is still working, but I only pull user tweet timelines to convert to RSS (using rssbox). Using this as my coal mine canary.

Same here, my Twitter API v2 token is still pulling in tweets, for now.

LinAGKar commented 1 year ago

I've been getting the same 401s for a few days, but now it just started working again, with no change made to the instance itself. The one thing that has changed on my end is that I got a new public IP from my ISP.

arnd-s commented 1 year ago

I tested my instance and can confirm, that TwitterBridge is for the moment working again, without any changes.

jutsh65 commented 1 year ago

I tested my instance and can confirm, that TwitterBridge is for the moment working again, without any changes.

I'm not getting the 403 error, but now a 500 error and a truncated rss feed containing one item with the title "Bridge returned error 500! (19396)" and the text: "was unable to receive or process the remote website's content!" .... "Error message: Could not parse guest token".

My IP address has not change though. I'll try setting rss-bridge up so it queries through a VPN and see what that does.

FYI/UPDATE: Updating my rss-bridge to the latest rather than the release fixed this.

dvikan commented 1 year ago

https://rss-bridge.org/twitter_test/ fixed the 403 errors. Notice that it runs the TwitterBridge and not the V2.

The code is here: https://github.com/dvikan/rss-bridge/tree/twitter-fix-1

quickwick commented 1 year ago

TwitterBridgeV2 is back to working for me. My guess is someone (maybe Musk himself) flipped the "turn off API access" switch too early, and has flipped it back. We'll see what happens on the 9th and after.

triatic commented 1 year ago

$100 per month for read & write API access: https://twitter.com/TwitterDev/status/1623467617037221889

Free access now to end on 13 February.

Mynacol commented 1 year ago

They delayed it yet again: https://twitter.com/TwitterDev/status/1625234161010343941

delaying the launch of our new API platform by a few more days

No specific date this time.

My version 1.1 API bridge is still running fine.

dvikan commented 1 year ago

For the TwitterBridge I've got a working patch[0] which fetches tweets by username using the unofficial api just like nitter[1] does. The other options are not done. They were pain to reverse engineer. And the docs on these apis are not easy to discover.

[0] https://rss-bridge.org/twitter_test/

[1] https://github.com/zedeus/nitter/issues/389

quickwick commented 1 year ago

For the TwitterBridge I've got a working patch[0] which fetches tweets by username using the unofficial api just like nitter[1] does. The other options are not done. They were pain to reverse engineer. And the docs on these apis are not easy to discover.

[0] https://rss-bridge.org/twitter_test/

I tried your twitter_test using a few of the handles I follow, and it's not showing any of their tweets from the last couple months, just stuff from Nov/Dec last year. However, I'm seeing the same behaviour on nitter, which makes me think it's an unofficial API issue, and not a problem with your implementation.

pin-grid-array commented 1 year ago

This might be useful to somebody:

https://pypi.org/project/twitter-api-client/

"Complete implementation of the undocumented Twitter API. Includes tools to scrape, automate, and search Twitter."

No tokens needed, just a username and password.

triatic commented 1 year ago

Twitter API v2 Free launched today, as expected it only offers write access. API v2 Basic is $100 per month.

API v2 Essential is being deprecated over the next 30 days.

https://twitter.com/TwitterDev/status/1641222782594990080

User123698745 commented 1 year ago

The Twitter V1 Bridge now returns error 0 (see #3382). They must have changed the V1 API again.

We should probably now switch to the unofficial api (used by Nitter). The implementation by @dvikan seems to work.

https://rss-bridge.org/twitter_test/ fixed the 403 errors. Notice that it runs the TwitterBridge and not the V2.

The code is here: https://github.com/dvikan/rss-bridge/tree/twitter-fix-1

triatic commented 1 year ago

Curiously, API v2 Essential is still operational despite the deadline. Not available for new signups though.

dvikan commented 1 year ago

The twitter branch [0] now has the latest changes from upstream and is fixed [1].

The only thing that is working is fetching tweets by username. Because I gave up how to scrape those other things. It was a bit tiresome.

I think maybe it's okay to merge it because it's better that atleast tweets by username is working rather than nothing.

I shall get to work on it right now.

[0] https://github.com/dvikan/rss-bridge/tree/twitter-fix-1

[1] https://rss-bridge.org/twitter_test/?action=display&bridge=TwitterBridge&context=By+username&u=elonmusk&format=Html

edit: pr which fixes only tweets by username: https://github.com/RSS-Bridge/rss-bridge/pull/3385

edit2: pr is merged to master

madking2099 commented 1 year ago

Your fix does indeed seem to have fixed the name look up. Thank you for both looking into this and for the fix of the twitter bridge that restores functionality.

The twitter branch [0] now has the latest changes from upstream and is fixed [1].

The only thing that is working is fetching tweets by username. Because I gave up how to scrape those other things. It was a bit tiresome.

I think maybe it's okay to merge it because it's better that atleast tweets by username is working rather than nothing.

I shall get to work on it right now.

[0] https://github.com/dvikan/rss-bridge/tree/twitter-fix-1

[1] https://rss-bridge.org/twitter_test/?action=display&bridge=TwitterBridge&context=By+username&u=elonmusk&format=Html

edit: pr which fixes only tweets by username: #3385

edit2: pr is merged to master

Pepepy commented 1 year ago

Hello everyone, Updated the entire repository to get the changes to the twitter bridge and now I get the following error:

Bridge returned error 0! (19488) 2023-05-11 09:23:20 Exception HttpException: cURL error Maximum (5) redirects followed: 47 (https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://twitter.com in lib/contents.php line 252

0 index.php(7): RssBridge->main()

1 lib/RssBridge.php(15): RssBridge->run()

2 lib/RssBridge.php(82): DisplayAction->execute()

3 actions/DisplayAction.php(137): TwitterBridge->collectData()

4 bridges/TwitterBridge.php(222): TwitterBridge->getApiKey()

5 bridges/TwitterBridge.php(531): getContents()

6 lib/contents.php(106): _http_request()

7 lib/contents.php(252)

Query string: action=display&bridge=TwitterBridge&context=By username&u=elonmusk&format=Html

Version: dev.2022-06-14 (git.master.1b45a53) OS: Linux PHP version: 8.0.28

Can someone tell me what is wrong or what I have to do to make it work? thank you so much.

Mynacol commented 1 year ago

It seems your update procedure didn't fully go through. The git commit 1b45a53 is not the current master.

Tintwo commented 1 year ago

Like @Pepepy I have the same issue with version dev.2023-03-22 (git.master.c628f99) since yesterday [edit] But when I check the commit tree, I guess I need to update because there was some other fixes on twitter bridge ?

Mynacol commented 1 year ago

Again, commit c628f99 is before the merged fix in https://github.com/RSS-Bridge/rss-bridge/pull/3385. The cURL error Maximum (5) redirects followed error message affects any version before this PR. Just checkout the current master, currently at commit e99e026fa8e88e54586837847eb23ca5683ddcb1.

Pimmetje commented 1 year ago

https://github.com/RSS-Bridge/rss-bridge/issues/3239#issuecomment-1543655542 I also have the 500 error with the curl redirects. 8 may it worked. So i guess twitter broke it.

dvikan commented 1 year ago

#3239 (comment) I also have the 500 error with the curl redirects. 8 may it worked. So i guess twitter broke it.

@Pimmetje on master latest commit?

Pepepy commented 1 year ago

Thank you so much. Apparently my git pull didn't work correctly yesterday. I did a full install again on the master branch and it is running without problems.

Thanks a lot

Mynacol commented 1 year ago

Closing, the by-username functionality was restored in the master branch.

quentinus95 commented 1 year ago

@Mynacol would it be possible to release a new GitHub release with associated Docker images?

Pimmetje commented 1 year ago

#3239 (comment) I also have the 500 error with the curl redirects. 8 may it worked. So i guess twitter broke it.

@Pimmetje on master latest commit?

Latest release

Mynacol commented 1 year ago

@dvikan should do that. I joined as moderator just a week ago 😄

csisoap commented 1 year ago

It seem Twitter killed my token and i can't use TwitterV2 ever since. When i go to the developer dashboard, my Essential plan is gone and replace it with Free plan with 1,000 tweet per month. Damn it suck.

triatic commented 1 year ago

Same, my Essential v2 API account was disabled yesterday, Twitter finally flipped the off switch, 6 weeks after the date they originally set.