search

yellowbluesky / PixivforMuzei3

Best Pixiv plugin for Muzei 3
GNU General Public License v3.0
174 stars 15 forks source link

Actually fetching the artworks from Pixiv is very slow #89

Open yellowbluesky opened 4 years ago

yellowbluesky commented 4 years ago

It takes upwards of three minutes to fetch a single artwork. Slow on both wifi at home and on 4G

No hypotheses yet.

Could it to do with the API endpoints?

I don't think it's to do with a596ee154605fc7bbdfc2700c29990423167ee85, as that affects only authenticated update modes, and it's slow even on ranking updates modes.

yellowbluesky commented 4 years ago

Problem resolved itself? I tested this last night, on both home wifi and 4G, speeds are normal

Will close this issue in a few days if problem does not reoccur

yellowbluesky commented 4 years ago

It appears to just have been a transient issue with Pixiv's network.

Kyle2142 commented 3 years ago

As long as I've used this app, downloading has been very slow. Other plugins like danbooru download images at multiple megabytes per second, while this one seems to be less than 200KB/s most of the time. Not sure what else I can tell you to help debugging

yellowbluesky commented 3 years ago

Hey Kyle,

The app works perfectly fine for me, with download speeds of up to 4 MB/s when downloading feed images. I'll take a look at the code again soon, it's been a while for me now, see if there's anything I can update

Kyle2142 commented 3 years ago

I will also mention that your app seems to load images differently from others, notably the danbooru one I mentioned. The danbooru one populates the image meta data first in the browse view, and seems to load the images asynchronously after that. Your app seems to try load them synchronously in background, often resulting in none being downloaded (from what I see)

yellowbluesky commented 3 years ago

There's a good reason why my apps behaviour differs from danboorus or other Muzei sources

With most boorus, no particular authentication is required to download images. Thus the Danbooru Muzei source submits an image url for Muzei to download at its own leisure.

With Pixiv, you cannot access any image without certain fields in the HTTP header being set and present.

The referee field must be set to come from another Pixiv Web page, I use Daily Ranking. You can see this for yourself, attempting to directly open a Pixiv image url will result in an error. For example, opening this image https://i.pximg.net/img-original/img/2020/12/13/00/00/05/86261533_p0.jpg directly causes a 403 error, while opening the image from the artwork page is fine (right click on image and "Open image in new tab") https://www.pixiv.net/en/artworks/86261533

All the authenticated feed modes require an access token in the HTTP header.

There is no way to instruct Muzei to insert the correct referer and access tokens into the HTTP header, so my app downloads images by itself.

I'm at work right now, I'll get back to you with an answer for why no images get loaded for you

Kyle2142 commented 3 years ago

Thank you for the insight. Out of interest, how does https://github.com/james58899/MuzeiPixivSource do it? Images load quickly (not stable, but it reaches 5MB/s and usually stays around 3MB/s) and seemingly in the same way that cardboard museum does. I know its open source so I can look myself but I'm no java/kotlin programmer. I want to see your app get better. I really don't know why downloading is slow compared to the above, I could install a debug version that logs extra if you'd like.

yellowbluesky commented 3 years ago

I'll havew a squizz at James' app on the weekend.

rsmyst commented 2 years ago

Just droppin by to say that this issue is still present. James' App downlads images very quickly but for whatever reason this source takes a long time or doesn't download at all

misternem0 commented 2 years ago

I'm not certain if it is related, but I sometimes get partially downloaded images that are considered successful and used by Muzei for background art.