search

JimmXinu / FanFicFare

FanFicFare is a tool for making eBooks from stories on fanfiction and other web sites.
Other
739 stars 157 forks source link

Royal Road: Cannot update stories that have become stubs #942

Closed zozo1237 closed 1 year ago

zozo1237 commented 1 year ago

Some background:

A stub on Royal Road is when an author removes parts of a story (Usually at the start of a story). This is commonly done to comply with exclusivity contracts that the author has entered into. Stories that are stubs almost always still have new content being added to them by their author.

Why this is a problem for FanFicFare:

When updating a story FanFicFare checks the chapter count of the story and does a straightforward comparison of chapter counts. FanFicFare makes the assumption that if the new chapter count is greater then the old chapter count then the story has updated. This means that FanFicFare won't update the story and doesn't have the desired functionally of appending new chapters to an already archived story.

Why this problem is bigger then Royal Road:

The straightforward comparison of chapter counts is a dead simple and highly effective way to track if a story has updated but it has quite a few edge cases. Here's just a couple of issues it causes:

What's the fix?

The issue I'm ostensibly reporting here is the Royal Road stub issue - it's something that I just encountered and prompted me to report it. There are lots of quick ways you to patch this edge case - inducing checking for the stub tag that Royal Road automatically adds - but the root cause of the issue is clearly the simplistic comparison of chapter counts that FanFicFare does to track story updates.

In the short term I'd like to see a fix for the Royal Road stub issue specifically. In the long term though I'd hope to see FanFicFare gain the ability to deal with arbitrary changes to story content. This would reduce the possibility of edge cases, alert the user to changes to the stories they care about, and overall provide a better user experience.

Thanks for your time!

zozo1237 commented 1 year ago

At the time of writing there are 319 stub stories on Royal Road - inducing the most followed story on the platform. (Beware Of Chicken with over 29 thousand followers)

JimmXinu commented 1 year ago

I don't read Royal Road and I didn't write the adapter for it, so I don't know much about that site specifically.

FFF does use chapter count alone to decide whether to run the update or not. But when updating an epub for a site with unique chapter URLs, it does use the chapter URLs.

So in your case of an author note chapter that's removed, FFF will discard the epub chapter if it doesn't appear in the TOC anymore and download missing chapters that are in the TOC. You can also use 'Update Always' mode.

When the chapter URL doesn't change, the only way for FFF to check every chapter for changes in text/author notes etc is to download them all. At which point you may as well just do an 'Overwrite Always' mode update--it would actually cost less in terms of processing.

I don't think there's any way for FFF to intelligently detect and handle the cases you describe.

For example, how is FFF supposed to differentiate between a 'stub' story vs chapters removed/rewritten/replaced by the author? Both could appear as some chapter URL(s) removed from TOC and others added. Presumably in the 'stub' case, you'd want to keep the now missing chapter and but replace it in the rewrite case.

FFF's fundamental paradigm doesn't have a way to keep chapters that aren't in the TOC.

A nontechnical issue is the FanFicFare is meant for fanfiction. If an author has removed their content to make money on it, I'm not interested in helping work around that.

zozo1237 commented 1 year ago

Thanks for the quick response!

FFF does use chapter count alone to decide whether to run the update or not. But when updating an epub for a site with unique chapter URLs, it does use the chapter URLs.

I didn't know that, thanks for the info!

So in your case of an author note chapter that's removed, FFF will discard the epub chapter if it doesn't appear in the TOC anymore and download missing chapters that are in the TOC.

Yeah that's what I thought it would do, that's somewhat problematic.

At least for FF.Net if I understand correctly.

I could use 'Overwrite Always' - but I can't manually keep track of all the ~700 stories I have archived, if content gets removed I want to know about it before overwriting my archive.

For example, how is FFF supposed to differentiate between a 'stub' story vs chapters removed/rewritten/replaced by the author? Both could appear as some chapter URL(s) removed from TOC and others added. Presumably in the 'stub' case, you'd want to keep the now missing chapter and but replace it in the rewrite case.

I'd utilize a heuristic looking at per-chapter content. I've had good success with Levenshtein Distance in the past.

FFF's fundamental paradigm doesn't have a way to keep chapters that aren't in the TOC.

That's a valid point, it would be a major rework and it makes sense to avoid feature creep.

A nontechnical issue is the FanFicFare is meant for fanfiction. If an author has removed their content to make money on it, I'm not interested in helping work around that.

You should know then that fanfiction fundamentally violates US and UK copyright and - almost certainty - doesn't meet the US standard of fair use or the UK standard of fair dealing.

Regardless I'd argue that the ability to preserve cultural works is of utmost priority, but I understand your point of view.

Despite all of the above, thank you for both for taking the time to respond to me and for the time you spend building and maintaining such a useful archival tool!

Lagicrus commented 1 year ago

Out of interest, wouldn't a rough solution be checking if it has the STUB tag on the novel homepage and checking what the most recent chapter is by storing the latest chapter ID as those are unique if that isn't done already?

I admit that, yes, it possibly gets a bit dicey storing chapters that the Author has moved to Amazon to make money. But on the flip side, having a story just breaking due to a chapter count issue feels a bit of an unbalanced cost, which I assume only fixes itself when the live chapters equal the count before the stub.