OAuth required for some artists, works don't get downloaded/claims no images

nnm711 commented 3 years ago

Prerequisites

[x] Did you read FAQ section in readme.md?
[x] Did you test with the latest releases or commit ?
[x] Did you search for existing issues in Issues?

Description

Hi, first of all, thank you for this utility, a lot of great work and very configurable. After Pixiv changed its API in February, I am having trouble with OAuth logins.

I have to use the cookie method to log in. To download works, I am using batch downloading from list.txt(method 4.). For most artists it works fine, however, out of 185 artists, for ~35 the utility attempts to use OAuth method and fails because of no credentials in config.

2021-02-17 10:34:15,626 - PixivUtil20210215-beta1 - INFO - Member ID (4991228): 0 Empty Username or Password, remove cookie value and relogin, or add username/password to config.ini., hasDumpPage=N, pageData=None

If I do provide username/password, the OAuth login, it of course fails due to the new API changes:

2021-02-17 10:36:08,625 - PixivUtil20210215-beta1 - DEBUG - 400: {"has_error":true,"errors":{"system":{"message":"Invalid grant_type parameter or parameter missing","code":1508}},"error":"invalid_grant"}

Digging a bit more into past issues here, I found this thread and this workaround to get "refresh_token". Plugging that into config alongside cookie (without cookie or login credentials the utility immediately exits), seem to pass the OAuth login, but PixivUtil finds no images and just happily continues down the list.

2021-02-17 16:37:38,734 - PixivUtil20210215-beta1 - INFO - [33m[1mProcessing Member Id: 4991228[0m
2021-02-17 16:37:38,756 - PixivUtil20210215-beta1 - INFO - Member Url: https://www.pixiv.net/ajax/user/4991228/profile/all
2021-02-17 16:37:40,693 - PixivUtil20210215-beta1 - DEBUG - b'{"error":false,"message":"","body":{"illusts":[],"manga":[],"novels":[],"mangaSeries":[],"novelSeries":[],"pickup":[],"bookmarkCount":{"public":{"illust":1,"novel":0},"private":{"illust":0,"novel":0}},"externalSiteWorksStatus":{"booth":false,"sketch":true,"vroidHub":false},"request":{"showRequestTab":false,"postWorks":{"artworks":[],"novels":[]}}}}'
2021-02-17 16:37:40,711 - PixivUtil20210215-beta1 - INFO - Using OAuth to retrieve member info for: 4991228
2021-02-17 16:37:40,711 - PixivUtil20210215-beta1 - DEBUG - Getting member information: 4991228
2021-02-17 16:37:40,711 - PixivUtil20210215-beta1 - INFO - Login to OAuth using refresh token.
2021-02-17 16:37:43,759 - PixivUtil20210215-beta1 - DEBUG - 200: **_SNIP_ - JSON with all login data**
2021-02-17 16:37:45,520 - PixivUtil20210215-beta1 - DEBUG - reply: {"user":{"id":4991228,"name":"bobapple","account":"bobapple","profile_image_urls":{"medium":"https:\/\/i.pximg.net\/user-profile\/img\/2021\/01\/28\/12\/00\/32\/20083931_72c938277d324c3fc2a261393b34e8fd_170.jpg"},"comment":"\u65e5\u672c\u8a9e\u306f\u3061\u3087\u3063\u3068\u308f\u304b\u308a\u307e\u3059\u3002\r\n\r\nEnglish speaker who understands a little Japanese.","is_followed":false},"profile":{"webpage":null,"gender":"male","birth":"","birth_day":"","birth_year":1995,"region":"United States","address_id":48,"country_code":"US","job":"","job_id":0,"total_follow_users":237,"total_mypixiv_users":1,"total_illusts":0,"total_manga":0,"total_novels":0,"total_illust_bookmarks_public":1034,"total_illust_series":0,"total_novel_series":0,"background_image_url":null,"twitter_account":"","twitter_url":null,"pawoo_url":null,"is_premium":true,"is_using_custom_profile_image":true},"profile_publicity":{"gender":"public","region":"public","birth_day":"public","birth_year":"public","job":"public","pawoo":true},"workspace":{"pc":"","monitor":"","tool":"","scanner":"","tablet":"","mouse":"","printer":"","desktop":"","music":"","desk":"","chair":"","comment":"","workspace_image_url":null}}
2021-02-17 16:37:47,178 - PixivUtil20210215-beta1 - INFO - No image found for: 4991228
2021-02-17 16:37:47,248 - PixivUtil20210215-beta1 - INFO - Member_id: 4991228 completed: no images were found.

Affected PixivID examples(all nsfw):

4054206 4081073 4991228

Steps to Reproduce

put only cookie into authentication part of config.ini
put affected IDs into list.txt and load in the utility
use batch download from list in the utility (press 4, no tags filter)
utility will ask for OAuth and fails/ with also "refresh_token" in config OAuth is successful, but finds no images

Expected behavior:

Utility should download works of the artist.

Actual behavior:

Utility will ask for OAuth some artists, and fails to download their works. Even if OAuth login is successful, the utility finds no images, while the artist has images when using browser.

Versions

PixivUtil20210215-beta1

Thank you for your time, sorry to bother you.

Nandaka commented 3 years ago

check your pixiv website settings? ensure it is allow R-18 mode and update the cookies. I tried with different artist (14095911 also nsfw) and it can download just fine.

nnm711 commented 3 years ago

Yes, that was it, I am sorry. I made a new account in February to avoid my main account getting banned, and understandably it was set up with R-18 content disabled. It works fine now, thanks. Dumb from me to read FAQ, but not come to the idea it's new account's settings causing it. (I already triggered some protection for couple hours once, by downloading ~10gb within 24 hours, started getting error 500s inside utility, and I could only load the main page, without images, the rest of site were throwing "unknown exception" errors. I thought I was fine with 10s download_delay, but whole day of letting it run racked up a lot of data transfer)

Nandaka / PixivUtil2