Unable to download anything

[morgan-rhea]

For some reason, it can’t collect the fiction information and therefore can’t download the stories. I’ve tried it with different fanfictions and it still continues to fail to collect fiction information. Any info regarding what happened to it?

[StarWolf3000]

Sometimes this is only temporary. Fanfiction.net (FictionPress) will sometimes change something in the site configuration or the pages' source. This can sometimes span several days even.

[sfu69]

It seems like fanfiction.net stopped the curl command on some applications which scrap data from ffnet. There is a discord bot with the same issue as here.

[StarWolf3000]

So the only solution is to rewrite the scraper? Or add user-agent headers of common browsers to the curl call?

[sfu69]

Doesn't seem to be a user-agent ban according to another developer. Yet another developer found a solution by circling through proxies.

IP based curl ban, firewall most likely.

[StarWolf3000]

The random proxy approach might work for a first attempt. cURL itself has a proxy setting with CURLOPT_PROXY, and with array_random() one is picked from a list of predefined proxies each time a request is made. Maybe coupled this with random user-agents might even FF see it as legit normal users.

Of course this whole approach will slow down the process in the end.

[p0ody]

Yeah StarWorlf, you are right, i just tried using some random proxy and it works. It looks slower though lol.

I'll need to find some way to get a list of random proxys.

Thanks

[sfu69]

https://twitter.com/FictionPress/status/1334301179682013185?s=20

Here is the official reasoning behind it

[LanzCorporalAssWipe]

By stfu69's comment. We play in part to the problem since we constantly uses ff2ebook to download fanfics....... 😳😳😳 whistles innocently

[StarWolf3000]

I don't think ff2ebook creates significant more traffic than the average user.

[LanzCorporalAssWipe]

Unless a shit ton of people downloads a popular recently updated followed/favorited story. There's a hell of a lot of categories in FFN. I'm just saying it's a possibility. I for one constantly use it on newly updated stories out of fear that at any given time the authors might delete their story or completely rewrite it by either editing the existing story or deleting and uploading the new one. I have atleast more than a dozen stories that have been deleted by their authors and a few that have been edited.

[Kawa-oneechan]

I can understand wanting to defend against bad bots. But AO3 and FimFiction have to defend themselves too. The difference? They both have on-site epub downloads.

Either FFN can't be bothered to do so (yet will bother to block stuff), or they don't want to miss out on the ad revenue.

Thing is, if you offer on-site epub downloads the users would still see the ads while on their way to get the goods. Just... fewer in total?

Just my two few cents as someone who just wants to read his dumb stories in white on black, page by page, and automatically bookmarked. You know, on an epub reader app with a nice UX.

[StarWolf3000]

If FFN would offer on-site epub downloads, I would use them.

[p0ody]

TBH, with epub caching (the site is not pulling data when the epub is already up to date on the server), it doesn't use that much data.

Anyway, i've updated the website to use proxy from a list, choosing the fastest first. I hope it will work for a while.

[StarWolf3000]

It doesn't seem to work (stuck at "Collecting Fiction infos...") and I'm also getting a TypeError: error is undefined in file /js/ajax.functions.js in line 83 in the browser console.

If I "Force Update", then some chapters will download (if it has a lot), not all, and the same error occurs in line 133 instead. No file is generated. The error message in the HTTP response for those missing chapters reads:

error: {
    code_1: 0,
    message_1: "Couldn't find chapter(num_of_chapter) text."
}

where num_of_chapter is the chapter nothing was retrieved from.

[p0ody]

My cron job were not set properly, so the proxy list was never updated.

Now it should work better (Hopefully haha)

[Silent-Hunter]

This is still happening to me, although I will admit that this fic has 98 chapters. Maybe that's too many?

[sfu69]

FFnet put their website behind cloudflare. It's a huge RIP for any downloaders. None will work.

[LanzCorporalAssWipe]

FFnet put their website behind cloudflare. It's a huge RIP for any downloaders. None will work.

God fucking damn it. I just tried it on FF2EBOOK and another site that I thought was going to be my salvation http://ffnet-downloader.fr/ And they don't work anymore. FUCK! I even tried ficsave for fucks sake! WHY CAN'T WE EVER HAVE NICE THINGS!!!!!😭😭😭😭😭😭😭

[Silent-Hunter]

It still works, I just have to try it a whole bunch of times and occasionally it gets fiction infos. And even less occasionally, it actually gets all the chapter infos and finishes.

Would it be possible to make this appear to be a real person using a browser? Especially if I host it myself?