Pixiv changes broke PixivUtil

[charzho]

Prerequisites

• [x] Did you read FAQ section? Yes
• [x] Did you test with the latest releases or commit ? Yes

Description

Pixiv appears to have changed their website once again.

Steps to Reproduce

1. Download an image or artist's ID

Expected behavior: Download should work

Actual behavior: Return OAuth or key error

Versions

You can get this information from executing PixivUtil2.py --help.

[Nandaka]

@Hecatom can you login using pixiv apps?

[whinette]

@Hecatom that's the exact message I got on my banned account. \ Can you login into the mobile app? You may not and have the same error (if using https://github.com/alphasp/pxview, the official app just can't log in).

[Nandaka]

Changed the logic to use web-rpc rather than OAuth to get the member information. Tested for Option 1 and 2.

This require reference image id (automatically resolved), else it will fallback to OAuth (you will see in the console message). Need more testing if it can return the correct data

[NHOrus]

Well, Option 4, it uses OAuth for member information for me. Nothing about OAuth in logs for image id.

[NHOrus]

Beta 3 doesn't write things into log beyond startup info.

And I was wrong, non-OAuth requests happen, too.

[Nandaka]

Weird, I try option 4 and it only uses webrpc. https://github.com/Nandaka/PixivUtil2/blob/master/PixivBrowserFactory.py#L400 Tried to search OAuth, and it return nothing. https://github.com/Nandaka/PixivUtil2/blob/master/PixivBrowserFactory.py#L408

Also, no issue for the log file, it does write the logging (maybe need to exit so it will flush to disk?)

[Nandaka]

updated: https://github.com/Nandaka/PixivUtil2/files/3117161/pixivutil20190424-beta3.zip compiled one.

[NHOrus]

Weird, I try option 4 and it only uses webrpc. https://github.com/Nandaka/PixivUtil2/blob/master/PixivBrowserFactory.py#L400 Tried to search OAuth, and it return nothing. https://github.com/Nandaka/PixivUtil2/blob/master/PixivBrowserFactory.py#L408

Also, no issue for the log file, it does write the logging (maybe need to exit so it will flush to disk?)

Well, last one is user error: log level WARNING instead of INFO. Will look and report things.

[IPods-and-spree-candy]

Hello new here just made an account in order to chime in I was able to recover my account after it was suspended, I changed the password in order to login and and then replied to the staff's message saying I was unaware that I was causing a problem after about a day I got a reply saying that my account was no longer suspended, created a throw away account in order to protect my main account if you can't login with the new account through the program due to the human check thing login on a browser and copy the cookie value should be the one that starts with your member id and paste it in the config file

[IPods-and-spree-candy]

Oh also unsure if it's a new bug or not but when I try to download by member id it will download 1 or 2 images and finish when there are a lot more then 1 or 2 images under the member or it will say no image found.

[Nandaka]

need the log file and config.ini.

Looks ok from my side.

[ikkion]

pixivutil20190424-beta3.zip has no PixivUtil2.exe in it is this correct?

[Nandaka]

woops, I think missed out the full package. pixivutil20190424-beta3.zip

[Hecatom]

@Hecatom can you login using pixiv apps?

@Hecatom that's the exact message I got on my banned account. Can you login into the mobile app? You may not and have the same error (if using https://github.com/alphasp/pxview, the official app just can't log in).

Sorry for not replaying before I dont use pixiv apps, so i cant check right now.

But after i got that error with the pixivutil i had to reset my pass because it kept giving me loging errors on the main site.

[todesschnitzel]

Looks like my account got flagged or something else... couldn't log in and received an email 8h ago....

We confirmed the act against the terms of use.

■Terms of use https://www.pixiv.net/terms/?page=term&lang=en https://www.pixiv.net/terms/?page=guideline&lang=en

Article 13. Prohibited Acts 14.Putting burden on the server over the normal extend or damaging the service operation or network system.

Please stop a lot of requests. If it does not improve, delete the account.

Please respond to this by submitting a support ticket below. https://www.pixiv.net/support.php

This mail is translated by Google.　https://translate.google.com/

After resetting my password, site login is working again... looks like they are after heavy users... and i even have premium for a few years... So, we will have to dial back on the amount of simultaneous downloads.

[whinette]

What was your typical daily use?

[todesschnitzel]

Too much, i think... But around 500-600 IDs a day... sometimes more than that, if i missed a day or something else happened... I also ran multiple instances... most of the time 2, but sometimes 4 and more. Downloaddelay in the config.ini was 5s.

[MartinChikov]

I've been using the app for over an year at this point, with various Internet speeds, for 24/7 periods of time (never more than one instance at a time however) and I've had never hit the download limit or had any problems, until 2 days ago.

When OAuth broke, I compiled the first commit - the result was that I would be able to download around 100 images (and since I away from home and had REALLY slow internet - I'm talking 1 Mbps, this process would take around 1:30 hours) and then it would break.

The following day, beta-1 came out. This version seemed to work for more than 100 images and I decided to let it work throughout the whole night. When I checked in the morning, I had downloaded around 1500 images (I guess my internet speed was better during the night), but everything seemed to work fine and I let the script continue doing its job.

A few hours later, I got the same email as @todesschnitzel saying my account was suspended. At first, I couldn't log in as well, but after changing my password I managed to and then proceeded to write an email to the support team, apologising for the inconvenience. This morning the suspension was removed, but in case of a repeat, the account will be permanently banned.

So, given the fact, I somehow managed to hit the download limit with 1 Mbps, I would have to guess the problem was with the OAuth requests, rather than the amount of images. The curious thing to me is that I managed to download 1500+ images throughout the night, without triggering anything. It's also possible, that they implemented these new download limits just yesterday morning and the trigger happened afterwards.

[whinette]

I have four new users I haven't downloaded anything from yet for a total of 570 submissions (so at least 570 images, not accounting for multipages @ manga). I'll do the test right now and see what happens.

[todesschnitzel]

this was also logged under my account... https://www.pixiv.net/security_history.php

I think they logged this for every downloaded image thorugh PixivUtil2, or more so PixivUtil2 logged in for every downloaded image and pixiv logged it.

Like @MartinChikov said, it could be that pixiv isn't happy to have this many log-ins in one account and that the downloading isn't the main problem, if at all.

[Nandaka]

@todesschnitzel can you clear the session in the web and try to use beta3 => https://github.com/Nandaka/PixivUtil2/releases/tag/v20190424-beta3

It should skip the oAuth (tested with option 1 and 2)

[Revemohl]

I made a new account, manually added the 1300 or so users I'm following on my main one because I don't know any better way to do so, got beta3, changed downloaddelay to 10, then let option 8 catch up on the past few days worth of submissions since everything broke and everything went well for the something around 400 posts it downloaded. The log doesn't say anything about OAuth either, so that could be some hope.

[todesschnitzel]

@Nandaka I already use the beta3. Also cleared the session and now it seems to work fine. (also only tested option 1 and 2) At the time of the logged pixiv OAuth logins, i used beta1 i think....

[reyaz006]

So does OAuth have anything to do with possibility of account suspension? If I use beta3 (if there are no mistakes in implementation), should it no longer cause a ban?

[Nandaka]

most likely, because it try to relogin multiple time :P

[Phenrei]

I see the same as @todesschnitzel during downloads made with beta 1 and after using beta 3 and resetting my password due to a similar email and lockout, I have been downloading fine. I will be turning off my delay and trying another round of downloading this evening. Interestingly I didn't get a spam of "new login" emails from the time of these logs.

Edit: I will add my downloads were with the same account and using option 8.

[ikkion]

When downloadning from Pixiv Fanbox (option f2) Beta3 will do a OAuth login for every processed page. If you set Max Page to 10 it will login ten times if there are ten pages or more.

[Nandaka]

beta4 add the handling for refresh token, so technically it shouldn't create new login everytime it do the user query from app-api

[ikkion]

When running a longer list using option 4 and Beta4 eventually it starts throwing this message, that it gets stuck on endlessly repeating , that might be linked to this specific user id whose gallery seems to be empty (I will remove this user from my list but it should not react like this):

2019-04-29 20:03:04,473 - PixivUtil20190429-beta4 - INFO - Member Url: https://www.pixiv.net/ajax/user/8716793/profile/all 2019-04-29 20:03:04,473 - PixivUtil20190429-beta4 - DEBUG - {"error":false,"message":"","body":{"illusts":[],"manga":[],"novels":[],"mangaSeries":[],"novelSeries":[],"pickup":[],"bookmarkCount":{"public":{"illust":0,"novel":0},"private":{"illust":0,"novel":0}}}} 2019-04-29 20:03:04,473 - PixivUtil20190429-beta4 - INFO - Using OAuth to retrieve member info for: 8716793 2019-04-29 20:03:04,473 - PixivUtil20190429-beta4 - ERROR - Error at processing Artist Info: (<type 'exceptions.NameError'>, NameError("global name '_config' is not defined",), <traceback object at 0x0415F7B0>) 2019-04-29 20:03:04,473 - PixivUtil20190429-beta4 - ERROR - Traceback (most recent call last): File "PixivUtil2.py", line 413, in process_member File "PixivBrowserFactory.pyc", line 508, in getMemberPage File "PixivBrowserFactory.pyc", line 411, in getMemberInfoWhitecube NameError: global name '_config' is not defined

EDIT: Beta4 seems to react like this to every user id with a empty gallery

[PixivAddict]

on startup beta 4 is giving me

[Nandaka]

on startup beta 4 is giving me

You are using windows xp or 32-bit machine?

[PixivAddict]

Win8 64 bit

[Latias4Ever]

I can verify Beta4 works, transferred my Auth info to the new config.ini, was able to download a couple of submissions just fine. My security history also has nothing besides my browser session. I haven't tried to download hundreds of IDs, though.

[Nandaka]

@PixivAddict weird, usuall dll error only happen if using older windows.

I'm compiling the script to exe using Win10 64bit. Looking around, it says something related to the included crypt32.dll or environment variables? I don't have older machine, so I cannot really test it.

Can you open the console and run cmd -> set -> search for SystemRoot? It should be set to your windows root.

[reyaz006]

Having the same issue as @PixivAddict and I'm on Windows 8.1 x64 - on Beta3 and Beta4. SystemRoot entry is there for me, no problem with it.

Beta1 did not have such problem.

[Nandaka]

is replacing the crypt32.dll from the local windows version fix the issue? Beta4 require new library(requests) to handle oauth relogin using refresh_token.

[tkiapril]

I just installed a fresh copy on macOS Mojave with python2 virtualenv and pip install -e ., and can confirm this issue still exists.

[tkiapril]

Sorry, also should have mentioned that I was trying FANBOX downloads (option f1). Logs below (ids censored for obvious reasons):

PixivDownloader2 version 20190429-beta4
https://github.com/Nandaka/PixivUtil2/releases
Donate at https://bit.ly/PixivUtilDonation
Reading config.ini ...
done.
Using default DB Path: db.sqlite 
Creating database... done.
Only process members where the last update is >= 7 days ago 
Using Username: 
Trying to log in with saved cookie 
Login successful. 
My User Id: 
PixivDownloader2 version 20190429-beta4
https://github.com/Nandaka/PixivUtil2/releases
Donate at https://bit.ly/PixivUtilDonation
1. Download by member_id
2. Download by image_id
3. Download by tags
4. Download from list
5. Download from bookmarked artists (/bookmark.php?type=user)
6. Download from bookmarked images (/bookmark.php)
7. Download from tags list
8. Download new illust from bookmarked members (/bookmark_new_illust.php)
9. Download by Title/Caption
10. Download by Tag and Member Id
11. Download Member Bookmark (/bookmark.php?id=)
12. Download by Group Id
------------------------
f1. Download from supported artists (FANBOX)
f2. Download by artist id (FANBOX)
------------------------
d. Manage database
e. Export online bookmark
m. Export online user bookmark
r. Reload config.ini
p. Print config.ini
x. Exit
Input: f1
Max Page = 10000
Getting supported artists from https://www.pixiv.net/ajax/fanbox/support 
Found 4 supported artist(s) 
[]
Processing , page 1 
Getting posts from https://www.pixiv.net/ajax/fanbox/creator?userId=
Using OAuth to retrieve member info for: 
Traceback (most recent call last):
  File "/Users/tki/PixivUtil2/PixivUtil2.py", line 2302, in main
    np_is_valid, op_is_valid, selection = main_loop(ewd, op_is_valid, selection, np_is_valid, args)
  File "/Users/tki/PixivUtil2/PixivUtil2.py", line 2059, in main_loop
    menu_fanbox_download_supported_artist(op_is_valid, args)
  File "/Users/tki/PixivUtil2/PixivUtil2.py", line 1809, in menu_fanbox_download_supported_artist
    processFanboxArtist(artist_id, end_page)
  File "/Users/tki/PixivUtil2/PixivUtil2.py", line 1818, in processFanboxArtist
    result_artist = __br__.fanboxGetPostsFromArtist(artist_id, next_url)
  File "/Users/tki/PixivUtil2/PixivBrowserFactory.py", line 602, in fanboxGetPostsFromArtist
    self.getMemberInfoWhitecube(artist_id, pixivArtist)
  File "/Users/tki/PixivUtil2/PixivBrowserFactory.py", line 411, in getMemberInfoWhitecube
    if _config.useProxy:
NameError: global name '_config' is not defined
press enter to exit.```

[reyaz006]

is replacing the crypt32.dll from the local windows version fix the issue?

No, same error.

But if I remove bcrypt.dll, it works. But I really don't want to use it like this without knowing it's okay.

[Nandaka]

bcrypt.dll or CRYPT32.dll

If the file is deleted I think it will uses the version provided by windows. Those related to the https/ssl encryption I think.

This is what provided in the zip file, which matches with my windows version.

I tried to delete those files and I can download just fine. Maybe I can skip those file from the bundled zip file.

[reyaz006]

I don't think you should package any DLL that is being shipped with Windows, because it will most certainly not work for different OS versions. Whatever it was that you tried to solve by adding those DLL, should be solved in a different way. E.g. there may be newer versions of redistributable packages available for all Windows versions, or you may need to refrain from using API that is only available in Windows 10, for the tool to work in previous OS versions.

[Nandaka]

nah, those are automatically packaged by the py2exe so I usually just leave it there.

I'll update the compile script to delete those dll.

[PixivAddict]

nice removing bcrypt works my system root was the same as in the image, "C:\Windows" btw

now that its working im seeing the same issue @ikkion had with empty galleries giving an error

[ghost]

Other option is working fine on beta 4 except for option 4, is not working again.

I got this error around 4 PM (GMT+7) (before that time is working fine):

2019-04-30 16:46:01,540 - PixivUtil20190429-beta4 - INFO - Member Url: https://www.pixiv.net/ajax/user/18061751/profile/all
2019-04-30 16:46:01,540 - PixivUtil20190429-beta4 - DEBUG - {"error":false,"message":"","body":{"illusts":[],"manga":[],"novels":[],"mangaSeries":[],"novelSeries":[],"pickup":[],"bookmarkCount":{"public":{"illust":1,"novel":1},"private":{"illust":0,"novel":0}}}}
2019-04-30 16:46:01,549 - PixivUtil20190429-beta4 - INFO - Using OAuth to retrieve member info for: 18061751
2019-04-30 16:46:01,549 - PixivUtil20190429-beta4 - ERROR - Error at processing Artist Info: (<type 'exceptions.NameError'>, NameError("global name '_config' is not defined",), <traceback object at 0x0507F940>)
2019-04-30 16:46:01,549 - PixivUtil20190429-beta4 - ERROR - Traceback (most recent call last):
  File "PixivUtil2.py", line 413, in process_member
  File "PixivBrowserFactory.pyc", line 508, in getMemberPage
  File "PixivBrowserFactory.pyc", line 411, in getMemberInfoWhitecube
NameError: global name '_config' is not defined

[NHOrus]

Same error with option f1

2019-04-30 22:31:14,907 - PixivUtil20190429-beta4 - INFO - Using OAuth to retrieve member info for: 7968
2019-04-30 22:31:14,910 - PixivUtil20190429-beta4 - ERROR - Unknown Error: global name '_config' is not defined
Traceback (most recent call last):
  File "PixivUtil2.py", line 2293, in main
  File "PixivUtil2.py", line 2053, in main_loop
  File "PixivUtil2.py", line 1808, in menu_fanbox_download_supported_artist
  File "PixivUtil2.py", line 1817, in processFanboxArtist
  File "PixivBrowserFactory.pyc", line 602, in fanboxGetPostsFromArtist
  File "PixivBrowserFactory.pyc", line 411, in getMemberInfoWhitecube
NameError: global name '_config' is not defined
2019-04-30 22:31:29,674 - PixivUtil20190429-beta4 - INFO - EXIT: -1

[Nandaka]

Try https://github.com/Nandaka/PixivUtil2/releases/tag/v20190501

[NHOrus]

2019-05-01 16:20:39,486 - PixivUtil20190501 - ERROR - Unknown Error: Could not find a suitable TLS CA certificate bundle, invalid path: E:\conf\pixiv\pixivutil\library.zip\certifi\cacert.pem
Traceback (most recent call last):
  File "PixivUtil2.py", line 2302, in main
  File "PixivUtil2.py", line 2059, in main_loop
  File "PixivUtil2.py", line 1809, in menu_fanbox_download_supported_artist
  File "PixivUtil2.py", line 1818, in processFanboxArtist
  File "PixivBrowserFactory.pyc", line 602, in fanboxGetPostsFromArtist
  File "PixivBrowserFactory.pyc", line 419, in getMemberInfoWhitecube
  File "PixivOAuth.pyc", line 92, in login
  File "requests\api.pyc", line 116, in post
  File "requests\api.pyc", line 60, in request
  File "requests\sessions.pyc", line 533, in request
  File "requests\sessions.pyc", line 646, in send
  File "requests\adapters.pyc", line 416, in send
  File "requests\adapters.pyc", line 228, in cert_verify
IOError: Could not find a suitable TLS CA certificate bundle, invalid path: E:\conf\pixiv\pixivutil\library.zip\certifi\cacert.pem

Not yet.

[bluerthanever]

I got this while using the 20190501 release...

Traceback (most recent call last): File "pixivutil2.py", line 2306, in main np_is_valid, op_is_valid, selection = main_loop(ewd, op_is_valid, selection, np_is_valid, args) File "pixivutil2.py", line 2063, in main_loop menu_fanbox_download_supported_artist(op_is_valid, args) File "pixivutil2.py", line 1809, in menu_fanbox_download_supported_artist processFanboxArtist(artist_id, end_page) File "pixivutil2.py", line 1818, in processFanboxArtist result_artist = br.fanboxGetPostsFromArtist(artist_id, next_url) File "F:\PixivUtil2-20190501\PixivBrowserFactory.py", line 602, in fanboxGetPostsFromArtist self.getMemberInfoWhitecube(artist_id, pixivArtist) File "F:\PixivUtil2-20190501\PixivBrowserFactory.py", line 421, in getMemberInfoWhitecube login_response = json.loads(oauth_response.text) NameError: global name 'oauth_response' is not defined

[Nandaka]

IOError: Could not find a suitable TLS CA certificate bundle, invalid path: E:\conf\pixiv\pixivutil\library.zip\certifi\cacert.pem

You running from scripts? need to install certifi too apparently, I thought it was included in requests lib. Going to update the requirements.txt

[bluerthanever]

https://github.com/Nandaka/PixivUtil2/blob/356d86cfe51cd9b1c20423a768fc23f0fd3e179c/PixivBrowserFactory.py#L420-L422

I think codes here should be like this?

if login_response.status_code == 200: login_response = json.loads(login_response.text) self._config.refresh_token = login_response["response"]["refresh_token"] self._config.writeConfig(path=self._config.configFileLocation)