search

Volvagia356 / pixiv-downloader

Script to download artworks from pixiv
BSD 3-Clause "New" or "Revised" License
12 stars 3 forks source link

Pixiv revised URL again #3

Open Theonik opened 9 years ago

Theonik commented 9 years ago

It would appear that the URLs used by Pixiv for new single illustrations are now changed to a new format.

Theonik commented 9 years ago

This also applies to manga illustrations.

Volvagia356 commented 9 years ago

Seems like they started using the new URL on Work #46270953 and the URL is based on the upload time. I hadn't checked out how the URL format is like for MyPixiv-only works, but will do so later.

Theonik commented 9 years ago

Yes, it seems the new format is more in line with how they deal with the Ugoira URL. Preview URL is also brought in-line with Ugoira as well so you can't use that as a cheap trick to DL those. Thanks for the help!

Volvagia356 commented 9 years ago

Argh, I was thinking of using the preview URLs to determine if the work is an Ugoira, looks like I can't do that anymore. I wonder how does the iOS app know though, since there doesn't seem to be any indication in the data returned by the API about whether it's an Ugoira or not.

Do you plan to make a patch for the new URL format?

Theonik commented 9 years ago

I have a partially working fix for my unstable branch of my fork. I'd done a lot of changes and I need to refactor the URL system quite badly now since I had to rush it to not lose any art on my feed. The changes are pretty big though so I thought not to do a pull request.

For Ugoira, my kind of 'hack' is to do a head request on the full URL and if that fails try the Ugoira. It's not too slow but it's not optimal, the 'bad' part is really when you already have a lot of the pictures where the extra delay is noticeable.

The actual app might very well be using that to choose since they only need to display the thumbnail until you click. I plan to do some extensive digging/network sniffing to confirm though. There should be more data somewhere as Ugoira also needs to send frametimes to the app. (since the animation doesn't have to have a fixed framerate each frame should also contain the time the frame should stay on screen which isn't in the ZIP)

Edit: Also working on solving a bug on my end in that it seems that the date implementation was changed since I wrote my Ugoira stuff. Might have been a problem from then but I never got any errors that I saw. Basically, the date where an illustration is stored is not necessarily the date that the PixivGUI and consecutively the API reports, it corresponds with the filing system. So for some multi-image illustrations it will produce wrong URL that way. It's a lot more rare with Ugoira I think due to only being one file.

Volvagia356 commented 9 years ago

You mean it's possible for different pages in multi-image illustrations to be sorted into different folders on the image server?

Theonik commented 9 years ago

Yes and no. For some illustrations it might be possible to have 2 copies it seems. So page 0 and 4 could be on a folder corresponding to the date given by the API/GUI. After some testing however I think that in those cases they will also exist on a folder together, which you can only determine by parsing the preview URL.

Also, it seems that work #46255594 also uses the new URL system. Maybe a couple more do. They probably were testing it and then rolled it back fairly quickly before giving it a full run with work #46270953

Volvagia356 commented 9 years ago

Hmm, I'm thinking about preparing a Wiki or something to document the mobile app's API.

Anyway, I think I'l be making a fix for this my own way, since it looks like yours has drifted quite a bit from mine. What do you think?