Nyaa feed doesn't work since Cloudflare implementation

[Schuuby]

Error showing:

2016-02-25 18:31:01 [Error] http.cpp:89 taiga::HttpClient::OnError | HTTP error #6: Couldn't resolve host name (8;URL=)
Connection mode: 9

[erengy]

There seems to be two issues here:

1. Nyaa is using the non-standard HTTP header Refresh: 8;URL=/cdn-cgi/l/chk_jschl?pass=(...) to redirect Taiga to another location. I have previously (and reluctantly) added support for this header in 795a0feddee18e8533c90c620495c13a1f5f522b, but the code needs to be updated to support uppercase URL=.
2. Fixing the first issue and following the redirection doesn't work. In fact, it kinda makes things worse. Cloudflare returns 302 Moved Temporarily with Location: /, which then makes Taiga fall into a redirection loop. Posing as a web browser by sending a different user-agent string doesn't work.

From what I can gather, Cloudflare's automatic browser check requires some JavaScript code to be run on the client. I don't think that it's possible for Taiga (and similar applications) to bypass this.

---

Apparently, Nyaa has enabled a security setting called "I'm Under Attack" mode:

When IUAM is turned on, CloudFlare will add an additional layer of protections to stop malicious HTTP traffic from being passed to your server. While a number of additional checks are performed in the background, an interstitial page is presented to your site’s visitors for 5 seconds while the checks are completed. Think of it as a challenge where the tests are automatic and visitors never need to fill in a CAPTCHA.

After verified as legitimate by the automated tests, visitors are able to browse your site unencumbered. Javascript and cookies are required for the tests, and to record the fact that the tests were correctly passed.

Again, I don't think that it's possible to bypass this, at least without acting maliciously.

[robflop]

Again, I don't think that it's possible to bypass this, at least without acting maliciously.

Man, that sounds bad. Would you suggest permanently switching to the feed you mentioned to me in a forum post (http://feeds.feedburner.com/anime-rss-atom-feeds?format=xml) then?

[erengy]

In theory, yes, switching to another RSS feed is the way to go. However:

• Most of the download links on other RSS feeds (including TokyoTosho) are still provided by Nyaa, so changing this setting doesn't help. Cloudflare's protection is enabled on Nyaa's download links too.
• As far as I'm aware, the only RSS feed that allows searching is Nyaa's. TokyoTosho and others don't have this feature.

@robflop You can customize your TokyoTosho RSS feed from here, but the default feed provided by Taiga is probably what you're after anyway.

[VivaLaPanda]

Nyaa says that IUAM has been off for a while. I talked to him on IRC. Here is some data; Recent bits of our conversation: http://puu.sh/nqAD5/1c7a2d1b2b.png Logs: http://pastebin.com/raw/cBGnz2GM

[VivaLaPanda]

Working DL: http://pastebin.com/raw/qHiu87Nv Failed DL: http://pastebin.com/raw/AzzaFw7Z

[erengy]

Their RSS feed now uses protocol-relative URLs for the channel link and item guids:

<link>//www.nyaa.se/</link>

<guid>//www.nyaa.se/?page=view&#38;tid=789763</guid>

Taiga's unable to download the .torrent files because it doesn't support protocol-relative URLs.

[VivaLaPanda]

What would need to be done to add support? Would it be a major change?

[erengy]

The URL parser of Taiga needs to be updated. I've already done this, I just haven't had the time to test it out yet.

That said, using relative URLs in RSS documents is against the specification:

In all link and url elements, the first non-whitespace characters in a URL MUST begin with a scheme defined by the IANA Registry of URI Schemes such as "ftp://", "http://", "https://", "mailto:" or "news://". These elements MUST NOT contain relative URLs.

I think Nyaa has already fixed this on their end.

[pickfire]

It really upsets me when Nyaa is using cloudflare, I can't automate the download now.

[reconman]

Cloudflare is blocking again.

[reconman]

Boku no Hero Academia was not listed as torrent so I switched the feed to tokyotosho. Then the episode was listed but trying to download resulted in an 403 error.

Seems like it only happens when I'm connected through a VPN.

My solution was adding a direct route to nyaa.se in my routing table so it bypasses the VPN.

[RozeDoyanawa]

I'm having the same issues since yesterday night. "Last update, 1 day 2 hours ago" "Bad response for 'https://www.nyaa.eu/?page=rss&term=HorribleSubs%201080p': 503 Service Temporary Unavailable." URL works fine in browser after allowing cloudfare scripts. URL does NOT work in Vuze.

//Regards

[reconman]

@RozeDoyanawa Are you using a VPN and if yes have you set up a direct route to nyaa.se?

[erengy]

@RozeDoyanawa Using nyaa.se instead of nyaa.eu works for me at the moment (you can change the URL from Settings → Torrents → Discovery).

[RozeDoyanawa]

@erengy Thanks for the .se tips, works for me now too. @reconman I'm not using any VPN.