Downloading rankings and animations throws API error

[Razzorxi]

Hi, I noticed today that the downloader breaks on the daily rankings and on users with animations in their gallery, downloading users without any animations in their gallery still works fine. I don't know since when it is broken, it still worked when I last updated my collection around a month ago. I assume Pixiv changed something in their API again.

Daily ranking:

H:\pixiv>pixiv.py -r 10:17:22 10/02/21 Traceback (most recent call last): File "H:\pixiv\pixiv.py", line 380, in sys.exit(main()) File "H:\pixiv\pixiv.py", line 346, in main download_by_ranking(user) File "H:\pixiv\pixiv.py", line 247, in download_by_ranking data_list = user.get_ranking_illustrations(per_page=100, mode='daily') File "H:\pixiv\api.py", line 518, in get_ranking_illustrations return self.parse_result(r) File "H:\pixiv\api.py", line 206, in parse_result raise Pixiv_Get_Error(r.url, data['errors']) api.Pixiv_Get_Error: Failed to get data: https://public-api.secure.pixiv.net/v1/ranking/all?mode=daily&page=1&per_page=100&include_stats=True&include_sanity_level=True&image_sizes=px_128x128%2Cpx_480mw%2Csmall%2Cmedium%2Clarge&profile_image_sizes=px_170x170%2Cpx_50x50

User with animations (NSFW user):

H:\pixiv>pixiv.py 4752417 10:18:58 10/02/21 Artists 4752417 Traceback (most recent call last): File "H:\pixiv\pixiv.py", line 380, in sys.exit(main()) File "H:\pixiv\pixiv.py", line 340, in main download_by_user_id(user, ids) File "H:\pixiv\pixiv.py", line 241, in download_by_user_id download_illustrations(user, data_list, save_path, add_user_folder=True) File "H:\pixiv\pixiv.py", line 216, in download_illustrations illustrations = PixivIllustModel.from_data(data_list, user) File "H:\pixiv\model.py", line 107, in from_data illust = cls.create_illust_from_data(data, user) File "H:\pixiv\model.py", line 84, in create_illust_from_data illust.extract_common_information(illust, data) File "H:\pixiv\model.py", line 55, in extract_common_information r = self.user.get_illustration(illust.id) File "H:\pixiv\api.py", line 411, in get_illustration return self.parse_result(r) File "H:\pixiv\api.py", line 206, in parse_result raise Pixiv_Get_Error(r.url, data['errors']) api.Pixiv_Get_Error: Failed to get data: https://public-api.secure.pixiv.net/v1/works/93052615.json?profile_image_sizes=px_170x170%2Cpx_50x50&image_sizes=px_128x128%2Cpx_480mw%2Csmall%2Cmedium%2Clarge&include_stats=true&include_sanity_level=true

User without animations:

H:\pixiv>pixiv.py 17429 10:20:57 10/02/21 Artists 17429 Start download, total illustrations 198 100%|████████████████████████████████████████████████████████████████████████████████| 198/198 [01:04<00:00, 3.09it/s] 10:22:08 10/02/21

[bebound]

It seems the old API expired. Thanks for your report. I'll migrate to new API, (and it may takes some time).

[bebound]

All API is failed. I'm working on this...

[bebound]

I create a new branch unstable, you can try it. PixivPy need to be installed before using it

[Razzorxi]

I tested it, logging in doesn't seem to work for me, after entering the refresh_token it just exits.

Please login Please open https://app-api.pixiv.net/web/v1/login?code_challenge=&code_challenge_method=S256&client=pixiv-android, and following steps in https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362 to get refresh_token Please input refresh_token: Traceback (most recent call last): File "H:\pixiv-unstable\pixiv.py", line 382, in sys.exit(main()) File "H:\pixiv-unstable\pixiv.py", line 333, in main user = PixivApi() File "H:\pixiv-unstable\api.py", line 52, in init self.login_required() File "H:\pixiv-unstable\api.py", line 171, in login_required self.login() File "H:\pixiv-unstable\api.py", line 156, in login self.aapi.auth(refresh_token=refresh_token) File "C:\Python38\lib\site-packages\pixivpy3\api.py", line 121, in auth raise PixivError('[ERROR] auth() failed! check refresh_token.\nHTTP %s: %s' % (r.status_code, r.text), pixivpy3.utils.PixivError: [ERROR] auth() failed! check refresh_token. HTTP 400: {"has_error":true,"errors":{"system":{"message":"Invalid refresh token","code":1508}},"error":"invalid_grant"}

I tried it with the session file from my master copy, and it worked fine the first time I ran it, but it takes a very long time to get into the menu on consecutive runs, throwing me a rate limit error before going into the menu.

pixiv.py {'error': {'user_message': '', 'message': 'Rate Limit', 'reason': '', 'user_message_details': {}}} Traceback (most recent call last): File "H:\pixiv-unstable\pixiv.py", line 276, in artist_folder_scanner data_list = user.get_user_illustrations(user_id, per_page=per_page) File "H:\pixiv-unstable\api.py", line 302, in get_user_illustrations return self.get_all_user_illustrations(user_id) File "H:\pixiv-unstable\api.py", line 297, in get_all_user_illustrations if not data['next_url']: KeyError: 'next_url' There is no new illustration need to download ############################ Pixiv Downloader 2.4 ########################### Which do you want to: 1 download by user id 2 download by ranking 3 download by history ranking 4 update exist 5 remove repeat e exit

As for downloads, it doesn't error out on animations anymore, but it also only downloads the first frame of them.

[bebound]

Please update to the latest version, it should start up quicker.

I’ll look for the login and animation problem tomorrow. On Oct 17, 2021, 23:57 +0800, bebound/pixiv @.***>, wrote:

but it takes a very long time to get into the menu on consecutive

[Razzorxi]

Thanks, startup is now as fast as before.

[bebound]

In latest version, ugoira is download as zip and login function is fixed

[Razzorxi]

Login works fine now, but throws me an error when I want to download directly after logging in. I can download when I run the script a second time.

Please login
Please open https://app-api.pixiv.net/web/v1/login?code_challenge=<challenge_replaced>&code_challenge_method=S256&client=pixiv-android
, and following steps in https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362 to get code
Please input code: <code_replaced>
access token: <access_token_replaced>, refresh token: <refresh_token_replaced>
############################ Pixiv Downloader 2.4 ###########################
Which do you want to:
         1 download by user id
         2 download by ranking
         3 download by history ranking
         4 update exist
         5 remove repeat
         e exit
:1
################### download by user id ####################

Input the artist's id:(separate with space)4752417
Artists 4752417
Traceback (most recent call last):
  File "H:\pixiv-unstable\pixiv.py", line 381, in <module>
    sys.exit(main())
  File "H:\pixiv-unstable\pixiv.py", line 370, in main
    options[choose](user)
  File "H:\pixiv-unstable\pixiv.py", line 241, in download_by_user_id
    data_list = user.get_all_user_illustrations(user_id)
  File "H:\pixiv-unstable\api.py", line 301, in get_all_user_illustrations
    data = self.aapi.user_illusts(user_id, offset=offset)
  File "C:\Python38\lib\site-packages\pixivpy3\aapi.py", line 119, in user_illusts
    r = self.no_auth_requests_call('GET', url, params=params, req_auth=req_auth)
  File "C:\Python38\lib\site-packages\pixivpy3\aapi.py", line 43, in no_auth_requests_call
    self.require_auth()
  File "C:\Python38\lib\site-packages\pixivpy3\api.py", line 51, in require_auth
    raise PixivError('Authentication required! Call login() or set_auth() first!')
pixivpy3.utils.PixivError: Authentication required! Call login() or set_auth() first!

Download seems to work fine, I don't mind ugoira being zips. Thanks.

[bebound]

Thank you, fixed.

[Razzorxi]

Yes, everything works now, thanks. Closing it unless I notice anything else.

[Razzorxi]

Hello, I didn't notice it earlier but I'm running into rate limits when updating my full illustration folder. I'm getting them consistently after it checked the fourth artist for new illustrations. My folder consists of over 1.2k artist right now, so splitting it into batches of 4 for updating doesn't seem viable for me.

Traceback (most recent call last):
  File "H:\pixiv\pixiv.py", line 276, in artist_folder_scanner
    data_list = user.get_user_illustrations(user_id, per_page=per_page)
  File "H:\pixiv\api.py", line 312, in get_user_illustrations
    return self.get_all_user_illustrations(user_id)
  File "H:\pixiv\api.py", line 307, in get_all_user_illustrations
    if not data['next_url']:
KeyError: 'next_url'
{'error': {'user_message': '', 'message': 'Rate Limit', 'reason': '', 'user_message_details': {}}}
Traceback (most recent call last):
  File "H:\pixiv\pixiv.py", line 276, in artist_folder_scanner
    data_list = user.get_user_illustrations(user_id, per_page=per_page)
  File "H:\pixiv\api.py", line 312, in get_user_illustrations
    return self.get_all_user_illustrations(user_id)
  File "H:\pixiv\api.py", line 307, in get_all_user_illustrations
    if not data['next_url']:
KeyError: 'next_url'```

[bebound]

I update the logic, hope this works.

[Razzorxi]

Hi, I get an auth error when going through my full folder, I assume because it takes too long and the api session runs out. I split up my folders into sets of 100 and it works then. I still get some rate limits on artists with a bigger collection. I'm also getting 404s on some animations.

Connection error: 404 => 93259919_ugoira0.png download error, retry
https://i.pximg.net/img-zip-ugoira/img/2021/10/06/14/25/34/93259919_ugoira0.png reach max retries, canceled

[bebound]

I add one second interval for each API in update folder, this may reduce the probability of rate limit.

404 error should be fixed now.

[Razzorxi]

Seems to work fine now, I'm only trying batches of 100 artists at a time, but I don't get any errors at the moment.