search

ciderpunx / twitrssme

TwitRSS.me: Tool to make Twittter timelines and searches into RSS feeds
https://twitrss.me
GNU General Public License v2.0
359 stars 59 forks source link

404 on /twitter_user_to_rss #59

Closed alexlamsl closed 4 years ago

alexlamsl commented 5 years ago

Looks like the same as #52 - started about 10 hours ago.

Try https://twitrss.me/twitter_user_to_rss/?user=BillGates for example.

anarchitectures commented 5 years ago

Noticed the same issue. Many of my twitter feeds are now either unreachable or unscrapable.

kazux commented 5 years ago

same issue here

kav2k commented 5 years ago

The request that pulled Twitter data (through scraping with Yahoo's YQL) now returns:

"message": "html table is no longer supported. See https://policies.yahoo.com/us/en/yahoo/terms/product-atos/yql/index.htm for YQL Terms of Use"

He's dead, Jim..

ciderpunx commented 5 years ago

Hi folks.

Yes it looks like twitter are now rate limiting TwitRSS.me's IP address. TBH I'm slightly surprised it has taken them so long!

You should be able to run your own version of the code with no problems assuming you're not looking at loads of users/searches.

I can still see the mobile version of twitter from that IP, but I'd need to rewrite the code to deal with that version of Twitter - not likely to happen until the weekend.

Sorry for the annoyance, but y'know, blame Twitter.

kazux commented 5 years ago

i was able to fix my issue by using the advanced search mode if you want to get feeds from a specific user it will be like this

https://twitrss.me/twitter_search_to_rss/?term=from%253ABillGates

decorative-help commented 5 years ago

Does anyone know other services (cannot get the Twitter api key)?

publicate.it

miksh7 commented 5 years ago

I fixed it by changing the user name from low to upper case, but the next day the uppercase is blocked while lowcase is okay. Looks like Twitter's blocking logic is case-sensitive.

piccolbo commented 5 years ago

What was the original reason not to use the API? I am sure there are some draconian limits on it, but just to be sure.

On Wed, Oct 17, 2018 at 5:25 AM miksh7 notifications@github.com wrote:

I fixed it by changing the user name from low to upper case, but the next day the uppercase is blocked while lowcase is okay. Looks like Twitter's blocking logic is case-sensitive.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ciderpunx/twitrssme/issues/59#issuecomment-430607438, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3RW77LjrpwggaTci1FNchr1ReDn-IEks5ulyGygaJpZM4XjLcN .

piccolbo commented 5 years ago

Never mind, I see it's premium/enterprise.

On Wed, Oct 17, 2018 at 8:11 AM Antonio Piccolboni antonio@piccolboni.info wrote:

What was the original reason not to use the API? I am sure there are some draconian limits on it, but just to be sure.

On Wed, Oct 17, 2018 at 5:25 AM miksh7 notifications@github.com wrote:

I fixed it by changing the user name from low to upper case, but the next day the uppercase is blocked while lowcase is okay. Looks like Twitter's blocking logic is case-sensitive.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ciderpunx/twitrssme/issues/59#issuecomment-430607438, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3RW77LjrpwggaTci1FNchr1ReDn-IEks5ulyGygaJpZM4XjLcN .

cookieguru commented 5 years ago

@AlexeyVolkov:

publicate.it

Doesn't seem to work, and is loaded with CORS errors

kimadagem commented 5 years ago

@kazux, that worked for me. Thank you.

kiwi810 commented 5 years ago

@kazux that worked thanks, however now i think i see EVERY single post the user is posting even replies to other users rather than the ones he broadcasts on his/her own account. Not that it bothers me but it kinda shows you what kind of person he/she really is when you see them reply rudely to other people on twitter lol wow im gobsmacked at the things ive been reading.

garas commented 5 years ago

Until problem resolved, could you respond with single static RSS item like

<item>
  <title>Twitter are rate limiting requests from TwitRSS.me</title>
  <description>Twitter are rate limiting requests from TwitRSS.me, meaning it is effectively broken until I can think of a way round it. You can still run your own instance</description>
  <pubDate>Wed, 17 Oct 2018 00:00:00 +0000</pubDate>
  <link>https://github.com/ciderpunx/twitrssme/issues/59#issuecomment-430535935</link>
</item>

I didn't notice that feed is down, and Feedly didn't show sign until checked myself today.

p-ho commented 5 years ago

Would it be viable to fetch Twitter via proxies? Just did a quick check: The two perl-scripts seem to run fine with torsocks for example (with increased timeout).

jessicachinafile commented 5 years ago

@kazux @kiwi810 to avoid all the replies and retweets you can just add the following to the end of the url: +exclude:replies+exclude:retweets The Bill Gates feed is not a good example since he doesn't appear to reply to people :) . But I tested it with other feeds and it appears to be working.

kiwi810 commented 5 years ago

@p-ho What format for the feed did you use with tor?

@jessicachinafile thanks will try that out

p-ho commented 5 years ago

@kiwi810 What exactly do you mean by format? As for the URLs I've just used the "straightforward" ones such as https://twitter.com/billgates

ariddell commented 5 years ago

Perhaps a rotating proxy service could be persuaded to donate to twitrss? For example, https://scrapinghub.com/crawlera quotes 150K monthly requests at $25/mo.

ciderpunx commented 5 years ago

Hi,

A small update. I've added a second script mobile_twitter_to_rss.pl that replicates the behaviour of twitter_user_to_rss.pl, except using the mobile version of the twitter site, which I don't think is rate limited.

There are a couple of limitations cf: https://github.com/ciderpunx/twitrssme/commit/9f48b09d03a11d09ac89f09384e60bb76f8e8178

If people were able to help me by throwing some requests at it and seeing what breaks, that would be handy :-)

In the long run, I'll probably do some refactoring so that the choice of using the normal or mobile twitter version is passed as a param (mobile by default, normal if you need images and can deal with the reliability issues you might face).

mhoran commented 4 years ago

This didn't work for me at first, but after the changes in #74 things are a bit better.

nivekalara237 commented 4 years ago

try this https://charlieharvey.org.uk/page/twitter_rss

glye commented 4 years ago

@nivekalara237 I get internal server error with queries on that page.

ralyodio commented 4 years ago

You need to use a proxy rotator. A good one is https://www.scraperapi.com/?fp_ref=anthony24

ciderpunx commented 4 years ago

Thanks for the tip, at present it does not look like twitter are blocking by IP, so a proxy rotator isn't needed, but may well be helpful in future.

fossdd commented 3 years ago

Hey boys, not sure if this issue is fixed. But why not connect to Twitter over tor.

Wikipedia says to Tor Tor is free and open-source software for enabling anonymous communication by directing Internet traffic through a free, worldwide, volunteer overlay network consisting of more than seven thousand relays[6] in order to conceal a user's location and usage from anyone conducting network surveillance or traffic analysis.

So you're connecting to nodes over the whole world with their IP addresses and not yours.

A way is using the tor socks proxy. Install the tor package on your OS, run tor. And connect to the socks proxy withlocalhost:9050`.

I saw you're using docker for the service. Not sure if they support socks. Or maybe implement socks support in this service itself?