Can't download from Webnovel

[seba11998]

Until last night I didn't have any problems, but since a few hours ago all novels return the same error. Be it for update or for new download:	Status	Title	Author	Comment	URL
Error	Naruto: The Strongest Kakashi	Iampoorguy	'volumeItems'	https://www.webnovel.com/book/naruto-the-strongest-kakashi_19348537506221105
Error	New God in marvel	DarknessAuthor	'volumeItems'	https://www.webnovel.com/book/new-god-in-marvel_21731452906433805
Error	One Piece Gold List: The Strongest Teacher!	Novelette_Seeker	'volumeItems'	https://www.webnovel.com/book/22964706206517905
Error	One Piece Starting with Uchiha Itachi's Template	theTripleX_HSA	'volumeItems'	https://www.webnovel.com/book/21070649705116205
Error	One Piece: I am Psychopath	VILLIAN_MAKER	'volumeItems'	https://www.webnovel.com/book/one-piece-i-am-psychopath_21563658606881805
Error	Reborn in Konoha as the Anbu Torturer	nyawdao2	'volumeItems'	https://www.webnovel.com/book/reborn-in-konoha-as-the-anbu-torturer_21747821906636805
Error	Stand User in Marvel Universe	nyawdao	'volumeItems'	https://www.webnovel.com/book/stand-user-in-marvel-universe_19351028205076305
Error	Star Wars: Skywalker	Auri_EL	'volumeItems'	https://www.webnovel.com/book/20290550605883805
Error	The Cursed Gamer	Master4thWall	'volumeItems'	https://www.webnovel.com/book/the-cursed-gamer_21047143105971305
Error	The Holy Chronicles of Baator	Trafford	'volumeItems'	https://www.webnovel.com/book/the-holy-chronicles-of-baator_13700263506926405
Error	The Legend of the Kyubi	GenoXX	'volumeItems'	https://www.webnovel.com/book/the-legend-of-the-kyubi_15934687606573405

[JimmXinu]

Instead of giving the chapter list in clear text JSON, the site is now giving some long base64 in a JSON packet that looks like this:

{
  "code":0,
  "data":{
    "content":"R2HNZ.....BNBEJPvc1lBS6rSUxYcQ==",
    "encryptKeyPool":"ujh.....BM4Xf6hC/l3x6u6/tw==",
    "encryptType":1,
    "encryptVersion":1652956258
  },
  "msg":"Success"
}

...except encryptKeyPool is ~80 chars long and content is 200k long.

The names suggest there's now some encryption going on. Presumably there's code in the browser or javascript loaded by the page that decrypts it.

This may or may not be fixable in FFF, depending on what's going on. I'm not going to be very interested in supporting it in FFF if it ends up requiring a bunch of crypto libraries.

[JimmXinu]

Crawling through the javascript, it looks like the site decrypts it in javascript. I believe the relevant JS files are:

www.yueimg.com/en/js/6198.1e36c847.js
www.yueimg.com/common/fock/2022040601/893458732457837.js

At least, they were for the three stories I checked.

Perhaps some one better with JS can translate the decode routines into python.

I did look briefly in js2py: It produces a huge unreadable mess--and makes no bones about it from the leading comment: # Don't look below, you will not understand this Python code :) I don't.

For all I know, it works perfectly well--but I'm not interested in including it (and the necessary dependencies) in FFF. And that's before even looking at whether it's py2 compatible.

TL;DR: Might be fixable, but I personally lack the ambition--especially since they could well just change it again.

[tom221-e]

hello, someone fix this problem?

[jcotton42]

@tom221-e Per the above discussion it looks like Webnovel is really determined to break FanFicFare. I wouldn't count on a fix coming, or working for long.

[tom221-e]

Hello, could you add the option to download the cache stories?

[jcotton42]

@tom221-e if you're referring to the browser cache feature that won't help here. The browser cache feature is for getting around things like Cloudflare, but that's not the issue here. FFF can fetch the page just fine, and it would be the same coming out of the cache.

[tom221-e]

But it is possible to download them using WebToEpub. Perhaps by cache they can continue downloading and updating the stories.

[JimmXinu]

The fact WebToEpub works is not really relevant.

WebToEpub runs in browser, it has access to the rendered HTML pages after all the javascript has run and, if necessary, the full javascript engine. The browser cache only retains files as fetched from the network, not what they render to.

Frankly, WebToEpub working is further reason for me to not try to fix this.

[CodeyAce95]

i know this is off topic but iuse webtoepub for fanfiction.net but still use this for everything else so does anybody know of a similiar solution for webnovel.com p.s. i still use this program for other websites i just want to get my webnoels updated

[JimmXinu]

It's been ~2 months, so I'm removing webnovel.com from FFF to avoid users expecting it to work.

[tom221-e]

hello it seems this project solved the problem and could webnovel.com (https://github.com/gmastergreatee/Fanfiction-Manager) import it to fanficfare?

[JimmXinu]

That in written in javascript and runs in electron, which is basically a frame work on top of a browser and therefore has a javascript engine to run the decrypt code.

[JimmXinu]

Site removed in version 4.14.3