AO3 is slow on CLI

[Twilight666]

I have been using FFF for a while and lately I have noticed that if I am trying to download some stories that have many chapters it takes way much longer than it did before.

So if I download http://archiveofourown.org/works/41934561 it takes ~4 minutes to download. It used to be much less, especially since use_view_full_work was set to true by default, so all it had to do is download 1 page.

On the other hand I tried to download a similar sized FFnet story (https://www.fanfiction.net/s/8642442/1/Variant-Strain)... after going though all it's chapters to put them in the cache, and it took less than a minute. So it doesn't look like it's the file building that is the problem

In fact it looks like the issue is in downloading the story data from AO3.

On the other hand I noticed that the above story had an image inside it and I thought that it might have been that, but I activated no_image_processing and nothing changed. I also downloaded http://archiveofourown.org/works/41552274 that is only 10 chapters but had a couple of big images and it took less than a minute. Also downloaded https://archiveofourown.org/works/15830529 that had a similar chapter length but no images and still took ~4 minutes

So it looks like the problem is just chapter number, and not how long it would take to download the story

[JimmXinu]

This is a significant slow down caused by change a2e9d29cf6b3a64f37ce043724b27d7578d87e9e AO3: Re-soup full_work on every chapter to avoid problems with soup changes. Found with always_reload_first_chapter:true MR discussion: https://www.mobileread.com/forums/showthread.php?p=4300015&postcount=8086

I expected that change to have some slowdown, but this is excessive. I'll see what I can figure out.

[JimmXinu]

Test versions up.