search

Nandaka / PixivUtil2

Download images from Pixiv and more!
http://nandaka.devnull.zone/
BSD 2-Clause "Simplified" License
2.37k stars 257 forks source link

Pixiv changes broke PixivUtil #483

Closed charzho closed 5 years ago

charzho commented 5 years ago

Prerequisites

Description

Pixiv appears to have changed their website once again.

Steps to Reproduce

  1. Download an image or artist's ID

Expected behavior: Download should work

Actual behavior: Return OAuth or key error

Versions

You can get this information from executing PixivUtil2.py --help.

Nandaka commented 5 years ago

yap, forgot to change it. I think i need to change my IDE from pyscripter to vscode. The error shows rights away.

bluerthanever commented 5 years ago

(´ω｀) I'm using notepad++ to view your code though.... haha.

NHOrus commented 5 years ago

I was running compiled program. With 20190501b IOError: Could not find a suitable TLS CA certificate bundle, invalid path: E:\conf\pixiv\pixivutil\library.zip\certifi\cacert.pem remains when I try to do fantia download

And in general. Trying fresh deploy instead of overwriting old file.
Fresh deploy haven't done anything to fix that problem.

aksskl commented 5 years ago

Gallery downloads work, but when pixivutil reaches a member that has an empty gallery it gets stuck on this error.

2019-05-01 21:10:55,407 - PixivUtil20190501b - INFO - Member Url: https://www.pixiv.net/ajax/user/6523466/profile/all
2019-05-01 21:10:55,407 - PixivUtil20190501b - DEBUG - {"error":false,"message":"","body":{"illusts":[],"manga":[],"novels":[],"mangaSeries":[],"novelSeries":[],"pickup":[],"bookmarkCount":{"public":{"illust":1,"novel":0},"private":{"illust":0,"novel":0}}}}
2019-05-01 21:10:55,411 - PixivUtil20190501b - INFO - Using OAuth to retrieve member info for: 6523466
2019-05-01 21:10:55,411 - PixivUtil20190501b - DEBUG - Getting member information: 6523466
2019-05-01 21:10:55,411 - PixivUtil20190501b - INFO - Login to OAuth using refresh token.
2019-05-01 21:10:55,421 - PixivUtil20190501b - ERROR - Error at processing Artist Info: (<type 'exceptions.IOError'>, IOError('Could not find a suitable TLS CA certificate bundle, invalid path: Z:\\pixivutil\\library.zip\\certifi\\cacert.pem',), <traceback object at 0x03EF3F80>)
2019-05-01 21:10:55,421 - PixivUtil20190501b - ERROR - Traceback (most recent call last):
  File "PixivUtil2.py", line 413, in process_member
  File "PixivBrowserFactory.pyc", line 504, in getMemberPage
  File "PixivBrowserFactory.pyc", line 415, in getMemberInfoWhitecube
  File "PixivOAuth.pyc", line 92, in login
  File "requests\api.pyc", line 116, in post
  File "requests\api.pyc", line 60, in request
  File "requests\sessions.pyc", line 533, in request
  File "requests\sessions.pyc", line 646, in send
  File "requests\adapters.pyc", line 416, in send
  File "requests\adapters.pyc", line 228, in cert_verify
IOError: Could not find a suitable TLS CA certificate bundle, invalid path: Z:\pixivutil\library.zip\certifi\cacert.pem
Nandaka commented 5 years ago

can you update the library.zip and add this file in the certifi folder: https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem

Nandaka commented 5 years ago

updated the release packages in https://github.com/Nandaka/PixivUtil2/releases/tag/v20190501 might need to delete the old folder (backup first! at least your config.ini, list.txt, and db.sqlite) and extract the new archive.

ikkion commented 5 years ago

Setup pixivutil20190501b-repack in both old folder and new clean folder. It just crashes during startup. Log almost empty.

2019-05-02 06:42:29,549 - PixivUtil20190501b - INFO - Setting log level to: DEBUG
2019-05-02 06:42:29,549 - PixivUtil20190501b - INFO - No default cookie jar available, creating...
NHOrus commented 5 years ago

Sadly, the same error.

2019-05-02 09:56:51,233 - PixivUtil20190501b - INFO - ###############################################################
2019-05-02 09:56:51,233 - PixivUtil20190501b - INFO - Starting with argument: [E:\conf\pixiv\pixivutil\PixivUtil2.exe].
2019-05-02 09:56:51,236 - PixivUtil20190501b - INFO - Setting log level to: INFO
2019-05-02 09:56:51,236 - PixivUtil20190501b - INFO - No default cookie jar available, creating... 
2019-05-02 09:56:51,237 - PixivUtil20190501b - INFO - Using default DB Path: E:\conf\pixiv\pixivutil\db.sqlite
2019-05-02 09:56:51,239 - PixivUtil20190501b - INFO - Using Username: nhorus
2019-05-02 09:56:51,239 - PixivUtil20190501b - INFO - Logging in...
2019-05-02 09:57:33,243 - PixivUtil20190501b - ERROR - Error at login(): (<class 'PixivException.PixivException'>, PixivException('Failed to get page: , please check your internet connection/firewall/antivirus.',), <traceback object at 0x04FA9288>)
2019-05-02 09:57:33,243 - PixivUtil20190501b - ERROR - Traceback (most recent call last):
  File "PixivBrowserFactory.pyc", line 253, in login
  File "PixivBrowserFactory.pyc", line 156, in open_with_retry
PixivException: 9005 Failed to get page: , please check your internet connection/firewall/antivirus., hasDumpPage=N

2019-05-02 09:57:33,243 - PixivUtil20190501b - ERROR - Error at doLogin(): (<class 'PixivException.PixivException'>, PixivException('Failed to get page: , please check your internet connection/firewall/antivirus.',), <traceback object at 0x04FA92B0>)
2019-05-02 09:57:33,243 - PixivUtil20190501b - ERROR - Traceback (most recent call last):
  File "PixivUtil2.py", line 2105, in doLogin
  File "PixivBrowserFactory.pyc", line 253, in login
  File "PixivBrowserFactory.pyc", line 156, in open_with_retry
PixivException: 9005 Failed to get page: , please check your internet connection/firewall/antivirus., hasDumpPage=N

2019-05-02 09:57:33,243 - PixivUtil20190501b - ERROR - Cannot Login!
2019-05-02 09:57:33,243 - PixivUtil20190501b - ERROR - Traceback (most recent call last):
  File "PixivUtil2.py", line 2299, in main
  File "PixivUtil2.py", line 2109, in doLogin
PixivException: 1005 Cannot Login!, hasDumpPage=N

2019-05-02 09:58:39,319 - PixivUtil20190501b - INFO - EXIT: 1005
2019-05-02 09:58:39,321 - PixivUtil20190501b - INFO - ###############################################################

Pixiv really messed with our ability to get funsies, this time..

ghost commented 5 years ago

can you update the library.zip and add this file in the certifi folder: https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem

i already added that file on library.zip, but still not working on option 4, just get this error (using recent build):

2019-05-02 15:05:55,183 - PixivUtil20190501 - INFO - Member Url: https://www.pixiv.net/ajax/user/18061751/profile/all
2019-05-02 15:05:55,183 - PixivUtil20190501 - DEBUG - {"error":false,"message":"","body":{"illusts":[],"manga":[],"novels":[],"mangaSeries":[],"novelSeries":[],"pickup":[],"bookmarkCount":{"public":{"illust":1,"novel":1},"private":{"illust":0,"novel":0}}}}
2019-05-02 15:05:55,198 - PixivUtil20190501 - INFO - Using OAuth to retrieve member info for: 18061751
2019-05-02 15:05:55,198 - PixivUtil20190501 - DEBUG - Getting member information: 18061751
2019-05-02 15:05:55,198 - PixivUtil20190501 - INFO - Login to OAuth using refresh token.
2019-05-02 15:05:55,338 - PixivUtil20190501 - ERROR - Error at processing Artist Info: (<class 'requests.exceptions.SSLError'>, SSLError(MaxRetryError("HTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by SSLError(SSLError(0, u'unknown error (_ssl.c:2947)'),))",),), <traceback object at 0x03F10300>)
2019-05-02 15:05:55,338 - PixivUtil20190501 - ERROR - Traceback (most recent call last):
  File "PixivUtil2.py", line 413, in process_member
  File "PixivBrowserFactory.pyc", line 508, in getMemberPage
  File "PixivBrowserFactory.pyc", line 419, in getMemberInfoWhitecube
  File "PixivOAuth.pyc", line 92, in login
  File "requests\api.pyc", line 116, in post
  File "requests\api.pyc", line 60, in request
  File "requests\sessions.pyc", line 533, in request
  File "requests\sessions.pyc", line 646, in send
  File "requests\adapters.pyc", line 514, in send
SSLError: HTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by SSLError(SSLError(0, u'unknown error (_ssl.c:2947)'),))
Nandaka commented 5 years ago

pixivutil20190501b-repack2.zip Not sure because of repackaging...

It is working on my PC. Ensure there is no https snooping, as it uses its own SSL store, so if you have custom SSL certificate that do mitm proxy, then it might not be working.

Nandaka commented 5 years ago

i already added that file on library.zip, but still not working on option 4, just get this error (using recent build):

It must be in the certifi folder, anyway latest build should have it already. image

Nandaka commented 5 years ago

@NHOrus I have tested the latest pack on new win10 installation (no proxy) and it is working as expected.

NHOrus commented 5 years ago

That's Win7 for me... Maybe it's time to update...

ikkion commented 5 years ago

I am running a list using pixivutil20190501b-repack2, so far it is working, but I have yet to encounter a user with empty gallery because I have been cleaning them from my list. I have not tested it on Fanbox yet. My OS is Win 7 64bit.

Nandaka commented 5 years ago

@NHOrus try to test it on different pc with different internet connection?

aksskl commented 5 years ago

Added the cacert.pem to certifi folder while using pixivutil20190501b, and also tried the new repack. Both appear to work with empty users, but first run it went through the oAuth process that got people suspended. Checked my pixiv account login history and there were two logins when a no image gallery was processed. One by pixivutil20190501b and one by the repack. (Top two in the below image)

pixivlogins

I think those logins were only to get the refresh token. Two logins because two different folders with separate config files. Subsequent runs did not show any more logins.

So yes it is working for me on win10. Empty galleries do not get stuck in a loop. I tested that using option 1. I will see what happens if I get to an empty gallery in option 5, but should work the same right?

Nandaka commented 5 years ago

@aksskl it should be works for other modes as the root cause for the infinite loop is due to missing cert file.

Technically, you can use the refresh_token from the other config, so it will see the connection from single 'mobile app'.

NHOrus commented 5 years ago

It works from linux on same internet connection. What worries me is the lack of failed URL in the error message.

Nandaka commented 5 years ago

based on the logging, it failed when it try to log in: https://github.com/Nandaka/PixivUtil2/blob/v20190501/PixivBrowserFactory.py#L274

NHOrus commented 5 years ago

I'm getting error 400 if I'm trying to access that page in browser.

matsumotokenji commented 5 years ago

Windows 7 here.

pixivutil20190501b repack failed to launch. however i manage to get it working by using pixivutil20190501b and cacert.pem from the repack (by placing the file to the correct path in library.zip file).

NHOrus commented 5 years ago

It's something with packaging: I got python27 for windows, pip install --user -r requrements.txt and ran pixivutil2 with no problems.

Nandaka commented 5 years ago

@NHOrus I have tried the repack2 on fresh win10 installation without python27 installed and I see no issue.

@matsumotokenji repack2 should have the cacert.pem included.

maslos commented 5 years ago

Just want to say thank you for fix ^^

PixivAddict commented 5 years ago

I have some members with no gallery in my db, been removing them one by one but for now I get this error, has it been addressed yet?

Member Url: https://www.pixiv.net/ajax/user/25/profile/all Using OAuth to retrieve member info for: 25 Traceback (most recent call last): File "PixivUtil2.py", line 413, in process_member File "PixivBrowserFactory.pyc", line 504, in getMemberPage File "PixivBrowserFactory.pyc", line 415, in getMemberInfoWhitecube File "PixivOAuth.pyc", line 92, in login File "requests\api.pyc", line 116, in post File "requests\api.pyc", line 60, in request File "requests\sessions.pyc", line 533, in request File "requests\sessions.pyc", line 646, in send File "requests\adapters.pyc", line 416, in send File "requests\adapters.pyc", line 228, in cert_verify IOError: Could not find a suitable TLS CA certificate bundle, invalid path: C:\p ixivutils\pixivutil(4)\library.zip\certifi\cacert.pem Error at processing Artist Info: (<type 'exceptions.IOError'>, IOError('Could no t find a suitable TLS CA certificate bundle, invalid path: C:\pixivutils\pixiv util(4)\library.zip\certifi\cacert.pem',), <traceback object at 0x0505A738>)

Nandaka commented 5 years ago

Try repack-2 https://github.com/Nandaka/PixivUtil2/issues/483#issuecomment-488590435

PixivAddict commented 5 years ago

ok thanks

bbappserver commented 5 years ago

Something is still broken in the Oauth code of the latest release. I successfully downloads some images but others produce: Mode 5

Member Url: https://www.pixiv.net/ajax/user/<id>/profile/all 
Using OAuth to retrieve member info for: <id>
Traceback (most recent call last):
  File "PixivUtil2.py", line 413, in process_member
    (artist, list_page) = PixivBrowserFactory.getBrowser().getMemberPage(member_id, page, bookmark, tags)
  File "PixivUtil2/PixivBrowserFactory.py", line 508, in getMemberPage
    self.getMemberInfoWhitecube(member_id, artist, bookmark)
  File "PixivUtil2/PixivBrowserFactory.py", line 421, in getMemberInfoWhitecube
    login_response = json.loads(oauth_response.text)
NameError: global name 'oauth_response' is not defined
Error at processing Artist Info: (<type 'exceptions.NameError'>, NameError("global name 'oauth_response' is not defined",), <traceback object at 0x107fcbf38>)
Nandaka commented 5 years ago

@bbappserver https://github.com/Nandaka/PixivUtil2/issues/483#issuecomment-489523025