Fanfiction Reader App

[lucena3]

Stories not updating and not showing when browsing, someone please fix.

[darthShadow]

Fanfiction.net has blocked unofficial apps from accessing the site till some undetermined time in the future: https://twitter.com/FictionPress/status/1340504620343250946

[lucena3]

I guess we'll have to wait

[lucena3]

The app is working again

[genious7]

Just a small update.

The issue is being caused by a CloudFlare captcha and not by ads as I had originally assumed. I'll have to check the request headers and compare them to the ones in a web browser or in the official app.

[darthShadow]

This still works occasionally btw, it's just that it fails when it reaches beyond some number of stories, which leads me to believe that it's being throttled after some pre-defined number of consequent requests and presented with the captcha page after that.

The only other app that works similarly and is also currently working is https://github.com/JimmXinu/FanFicFare which has had to add a captcha solver/bypass dependency https://github.com/VeNoMouS/cloudscraper. I am not sure if something similar is available for Java/Android.

[darthShadow]

Hopefully one of these helps:

• https://github.com/Marplex/CloudflareBypass
• https://github.com/zhkrb/cloudflare-scrape-Android
• https://github.com/ninja-beans/cloudflare-iuam-solver

[PrincessGrouchy]

This still works occasionally btw, it's just that it fails when it reaches beyond some number of stories, which leads me to believe that it's being throttled after some pre-defined number of consequent requests and presented with the captcha page after that.

The only other app that works similarly and is also currently working is https://github.com/JimmXinu/FanFicFare which has had to add a captcha solver/bypass dependency https://github.com/VeNoMouS/cloudscraper. I am not sure if something similar is available for Java/Android.

The fanfiction reddit bot also has added cloudscraper to get around cloudflare. https://github.com/FanfictionBot/reddit-ffn-bot/commit/6293286ee206d8c7e609efea8b5ffa9d14cab944?branch=6293286ee206d8c7e609efea8b5ffa9d14cab944&diff=split ...not that it can be imported as-is for this project😅

[Luminatrix-eng]

@genious7 I have over 600 stories on your Fanfiction App and the stories aren't updating and I can't download new stories.

Is there a possible solution in the works? I suspect @darthShadow is right about the downloads being throttled after a number of requests.

[Luminatrix-eng]

@genious7 I don't know much about android apps but just wanted to give my input. I tried using VPN to change my IP to bypass Cloudflare captcha and then tried updating stories which did not work. So it's not necessarily an issue of Cloudflare singling any specific IP and challenging that IP.

This site talks about the issue in detail which might help here: https://greasyfork.org/zh-CN/scripts/10182-fanfiction-qomplete/discussions/70993

According to this discussion, apparently Cloudflare doesn't get triggered if the "Rate limit chapter downloading down to 1 per second (faster for first few chapters) to not hit cloudflare rate limits" I wonder if this might be an alternative solution? It might take twice as long as normal to update stories but it could be something that people would be happy with if it means being able to use the app again.

There's also a discussion here which might have a solution as well: https://github.com/JimmXinu/FanFicFare/issues/614

I hope this helps.

[RavenBlack86]

Please fix the app. I miss reading my stories.

[Luminatrix-eng]

@RavenBlack86 I understand where you're coming from. I miss my stories too. My library hasn't been updated in a month!

Just to shed some clarity here though, the app was never broken to begin with. It's because the fanfiction site admins decided to be anal and adopt a very strict level of selfish gain by blocking third party reader apps, in order to force people to either use their own reader app or to force people to read stories online via the browser which would run ads whenever someone visited a page.

I'm trying to learn how to code android apps in order to help contribute but I'm limited right now by my lack of knowledge. I did my part by doing what I'm good at, which is research. And that was by posting the results of my research so that @genious7 has an easier time finding a potential fix. All we can do is be patient.

[Jwinter99e]

It wasn't really clear for those with android. But I was wondering if there is a rough estimate on how long it would take?

[Luminatrix-eng]

It wasn't really clear for those with android. But I was wondering if there is a rough estimate on how long it would take?

That is undetermined for now. @genious7 seems to be hard at work in fixing this issue. Please be patient in the interim.

I know it's frustrating but the problems that we are all experiencing came from the Admins of Fanfiction_dot_net website who decided to make changes that don't necessarily benefit most people. Just be patient, people are trying to find a solution in their spare time for free. And that deserves respect.

[hallowknight99]

@genious7 Api email dev@fictionpressm.com to become a tester which I belive this is to skip the cloudflare thing. Here is the source where I got api update https://mobile.twitter.com/FictionPress/status/1352168846682034177

[genious7]

Ok, small update.

None of the linked cloudflare solutions work, since they cannot solve the h-captcha challenge that cloudflare is currently using (they only solve the old reCaptcha challenges). Fanficare seems to have the same issue.

As far as I can tell, the "new Captcha" challenge goes as follows: when opening the web site, the browser receives a 403 error with the following javascript code. The code at the top contains the unique identifiers for the session. The m.fanfiction.net/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1 link goes to a very, very obfuscated js code. Based on this article, that page then leads onto another obfuscated javascript page. As far as I can tell, there aren't any elegant solutions (with HTTP connections & JSoup) to the cloudflare captcha issue as I'm not going to dive into that obfuscated mess (which is likely to change).

My best attempt so far consists on using a webview to run the javascript content, recover the html from the webview, and parse it from there. A proof of concept seems to be working, although I have to investigate a bit more regarding running a webview in the background and any potential issues with the android lifecycle.

[hassan852]

found this script today looks promising https://github.com/Paranoid-Dev/ParanoidFFD

[genious7]

found this script today looks promising https://github.com/Paranoid-Dev/ParanoidFFD

Not really. That script uses Undetected Chromedriver as a dependency. As seen in the function find_chrome_executable(), it works by opening the web site in a hidden chrome window (which requires the device to have chrome installed.

That concept (as-is) wont work on android since you can't run a hidden chrome window as they do. That being said, I'm attempting something conceptually similar using a hidden WebView instead of a hidden chrome browser.

[hassan852]

Good luck

[canerolu2019]

Hmm I wonder how long this will take to fix the issue

[Luminatrix-eng]

Ok, small update.

None of the linked cloudflare solutions work, since they cannot solve the h-captcha challenge that cloudflare is currently using (they only solve the old reCaptcha challenges). Fanficare seems to have the same issue.

As far as I can tell, the "new Captcha" challenge goes as follows: when opening the web site, the browser receives a 403 error with the following javascript code. The code at the top contains the unique identifiers for the session. The m.fanfiction.net/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1 link goes to a very, very obfuscated js code. Based on this article, that page then leads onto another obfuscated javascript page. As far as I can tell, there aren't any elegant solutions (with HTTP connections & JSoup) to the cloudflare captcha issue as I'm not going to dive into that obfuscated mess (which is likely to change).

My best attempt so far consists on using a webview to run the javascript content, recover the html from the webview, and parse it from there. A proof of concept seems to be working, although I have to investigate a bit more regarding running a webview in the background and any potential issues with the android lifecycle.

Seems like you're doing an amazing effort. Thank you.

On a side note, I can't fathom why Fanfiction.net admins are doing their best to piss off 85% of their visitors with this move to block just about every type of third party app that downloads stories. This is a dangerous legal territory because Fanfiction Admins don't own the rights to any stories belonging to writers who upload them or material taken from the original authors like JK Rowling, JRR Tolkien, CS Lewis, George Martin etc. This monopoly is further complicated by the fact, they are pushing ads on their website for these stories. It's just moronic to piss off the fanbase with this move.

[canerolu2019]

Ok, small update. None of the linked cloudflare solutions work, since they cannot solve the h-captcha challenge that cloudflare is currently using (they only solve the old reCaptcha challenges). Fanficare seems to have the same issue. As far as I can tell, the "new Captcha" challenge goes as follows: when opening the web site, the browser receives a 403 error with the following javascript code. The code at the top contains the unique identifiers for the session. The m.fanfiction.net/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1 link goes to a very, very obfuscated js code. Based on this article, that page then leads onto another obfuscated javascript page. As far as I can tell, there aren't any elegant solutions (with HTTP connections & JSoup) to the cloudflare captcha issue as I'm not going to dive into that obfuscated mess (which is likely to change). My best attempt so far consists on using a webview to run the javascript content, recover the html from the webview, and parse it from there. A proof of concept seems to be working, although I have to investigate a bit more regarding running a webview in the background and any potential issues with the android lifecycle.

Seems like you're doing an amazing effort. Thank you.

On a side note, I can't fathom why Fanfiction.net admins are doing their best to piss off 85% of their visitors with this move to block just about every type of third party app that downloads stories. This is a dangerous legal territory because Fanfiction Admins don't own the rights to any stories belonging to writers who upload them or material taken from the original authors like JK Rowling, JRR Tolkien, CS Lewis, George Martin etc. This monopoly is further complicated by the fact, they are pushing ads on their website for these stories. It's just moronic to piss off the fanbase with this move.

Well at least we should appreciate the fact that author is trying its best to fix the app. I mean showing fingers to fanfic admins seems kinda hypocritical atm. I get it what they did is pretty lame but we need to have a little hope that author is trying to fix the app. We should appreciate it.

[erd00073]

Ok, small update. None of the linked cloudflare solutions work, since they cannot solve the h-captcha challenge that cloudflare is currently using (they only solve the old reCaptcha challenges). Fanficare seems to have the same issue. As far as I can tell, the "new Captcha" challenge goes as follows: when opening the web site, the browser receives a 403 error with the following javascript code. The code at the top contains the unique identifiers for the session. The m.fanfiction.net/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1 link goes to a very, very obfuscated js code. Based on this article, that page then leads onto another obfuscated javascript page. As far as I can tell, there aren't any elegant solutions (with HTTP connections & JSoup) to the cloudflare captcha issue as I'm not going to dive into that obfuscated mess (which is likely to change). My best attempt so far consists on using a webview to run the javascript content, recover the html from the webview, and parse it from there. A proof of concept seems to be working, although I have to investigate a bit more regarding running a webview in the background and any potential issues with the android lifecycle.

Seems like you're doing an amazing effort. Thank you.

On a side note, I can't fathom why Fanfiction.net admins are doing their best to piss off 85% of their visitors with this move to block just about every type of third party app that downloads stories. This is a dangerous legal territory because Fanfiction Admins don't own the rights to any stories belonging to writers who upload them or material taken from the original authors like JK Rowling, JRR Tolkien, CS Lewis, George Martin etc. This monopoly is further complicated by the fact, they are pushing ads on their website for these stories. It's just moronic to piss off the fanbase with this move.

It is specifically about the ad revenue they are loosing every time you access material on their website without using a browser or one of their "approved" apps. Nothing more, nothing less. I half keep expecting to try to access the site one day and find that the owner has banned the use of browser adblock software like lots of commercial sites try to do.

The FFN site owner doesn't really care if you are pissed off about it because people like you and me that use third party apps to access content are only a small part of his userbase and the rest of the users don't care as it doesn't affect them.

[Luminatrix-eng]

Ok, small update. None of the linked cloudflare solutions work, since they cannot solve the h-captcha challenge that cloudflare is currently using (they only solve the old reCaptcha challenges). Fanficare seems to have the same issue. As far as I can tell, the "new Captcha" challenge goes as follows: when opening the web site, the browser receives a 403 error with the following javascript code. The code at the top contains the unique identifiers for the session. The m.fanfiction.net/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1 link goes to a very, very obfuscated js code. Based on this article, that page then leads onto another obfuscated javascript page. As far as I can tell, there aren't any elegant solutions (with HTTP connections & JSoup) to the cloudflare captcha issue as I'm not going to dive into that obfuscated mess (which is likely to change). My best attempt so far consists on using a webview to run the javascript content, recover the html from the webview, and parse it from there. A proof of concept seems to be working, although I have to investigate a bit more regarding running a webview in the background and any potential issues with the android lifecycle.

Seems like you're doing an amazing effort. Thank you. On a side note, I can't fathom why Fanfiction.net admins are doing their best to piss off 85% of their visitors with this move to block just about every type of third party app that downloads stories. This is a dangerous legal territory because Fanfiction Admins don't own the rights to any stories belonging to writers who upload them or material taken from the original authors like JK Rowling, JRR Tolkien, CS Lewis, George Martin etc. This monopoly is further complicated by the fact, they are pushing ads on their website for these stories. It's just moronic to piss off the fanbase with this move.

Well at least we should appreciate the fact that author is trying its best to fix the app. I mean showing fingers to fanfic admins seems kinda hypocritical atm. I get it what they did is pretty lame but we need to have a little hope that author is trying to fix the app. We should appreciate it.

I really appreciate that the author of this great app is trying to fix it. I know thousands of others would also appreciate it as well.

I'm just venting a little in regards to the fanfiction admins and their bizarre actions.

[canerolu2019]

Ok, small update. None of the linked cloudflare solutions work, since they cannot solve the h-captcha challenge that cloudflare is currently using (they only solve the old reCaptcha challenges). Fanficare seems to have the same issue. As far as I can tell, the "new Captcha" challenge goes as follows: when opening the web site, the browser receives a 403 error with the following javascript code. The code at the top contains the unique identifiers for the session. The m.fanfiction.net/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1 link goes to a very, very obfuscated js code. Based on this article, that page then leads onto another obfuscated javascript page. As far as I can tell, there aren't any elegant solutions (with HTTP connections & JSoup) to the cloudflare captcha issue as I'm not going to dive into that obfuscated mess (which is likely to change). My best attempt so far consists on using a webview to run the javascript content, recover the html from the webview, and parse it from there. A proof of concept seems to be working, although I have to investigate a bit more regarding running a webview in the background and any potential issues with the android lifecycle.

Seems like you're doing an amazing effort. Thank you. On a side note, I can't fathom why Fanfiction.net admins are doing their best to piss off 85% of their visitors with this move to block just about every type of third party app that downloads stories. This is a dangerous legal territory because Fanfiction Admins don't own the rights to any stories belonging to writers who upload them or material taken from the original authors like JK Rowling, JRR Tolkien, CS Lewis, George Martin etc. This monopoly is further complicated by the fact, they are pushing ads on their website for these stories. It's just moronic to piss off the fanbase with this move.

Well at least we should appreciate the fact that author is trying its best to fix the app. I mean showing fingers to fanfic admins seems kinda hypocritical atm. I get it what they did is pretty lame but we need to have a little hope that author is trying to fix the app. We should appreciate it.

I really appreciate that the author of this great app is trying to fix it. I know thousands of others would also appreciate it as well.

I'm just venting a little in regards to the fanfiction admins and their bizarre actions.

yeah i get it. The reason they did this i suppose it was a major site upgrade thats why

[thinkpad4]

@genious7 How goes the fix? Any progress yet?

[lucena3]

How's fixing the app going so far?

[kstout24]

Out of curiosity, has anyone run into the new captcha challenge whilst in incognito mode on their browser?

[genious7]

If you turn javascript and cookies off whilst on incognito, you'll have a higher chance of running into the captcha. The Cloudflare anti-scrapping uses javascript (as well as some other factors) to determine if you're using a computer browser (in which case no captcha appears) or a third party app (in which case the captcha appears). The only way to side-step the captcha is to have some sort of javascript engine on the app itself, which is what I'm trying to do through a WebView.

The easiest way to see the captcha screen is by trying to open the web site on one of those proxy websites that allow you to disable scripts. I've included a random one as an example (not that I advertise that particular proxy or anything, it's just the first one that came up in Google to me).

Regarding the progress update, I've been busy with a work-related deliverable I had due last week, so progress on the fix has been somewhat slow, although I'll try to finish it this week.

[canerolu2019]

If you turn javascript and cookies off whilst on incognito, you'll have a higher chance of running into the captcha. The Cloudflare anti-scrapping uses javascript (as well as some other factors) to determine if you're using a computer browser (in which case no captcha appears) or a third party app (in which case the captcha appears). The only way to side-step the captcha is to have some sort of javascript engine on the app itself, which is what I'm trying to do through a WebView.

The easiest way to see the captcha screen is by trying to open the web site on one of those proxy websites that allow you to disable scripts. I've included a random one as an example (not that I advertise that particular proxy or anything, it's just the first one that came up in Google to me).

Regarding the progress update, I've been busy with a work-related deliverable I had due last week, so progress on the fix has been somewhat slow, although I'll try to finish it this week.

I wish you success on fixing the app. I'm sure you can do it. We believe in you dude.

[Luminatrix-eng]

If you turn javascript and cookies off whilst on incognito, you'll have a higher chance of running into the captcha. The Cloudflare anti-scrapping uses javascript (as well as some other factors) to determine if you're using a computer browser (in which case no captcha appears) or a third party app (in which case the captcha appears). The only way to side-step the captcha is to have some sort of javascript engine on the app itself, which is what I'm trying to do through a WebView.

The easiest way to see the captcha screen is by trying to open the web site on one of those proxy websites that allow you to disable scripts. I've included a random one as an example (not that I advertise that particular proxy or anything, it's just the first one that came up in Google to me).

Regarding the progress update, I've been busy with a work-related deliverable I had due last week, so progress on the fix has been somewhat slow, although I'll try to finish it this week.

Good luck dude and thank you for trying to fix the app.

[genious7]

I just uploaded a "fix" that routes the app's requests through a hidden web browser to bypass Cloudflare's javascript check. It seems to be working for most of the app itself, although downloads are still failing with a Cloudflare 403 "forbidden" error, despite using essentially the same code to bypass the cloudflare captcha.

[hallowknight99]

@genious7 Good work getting the main part working