search

Nandaka / PixivUtil2

Download images from Pixiv and more!
http://nandaka.devnull.zone/
BSD 2-Clause "Simplified" License
2.37k stars 257 forks source link

Pixiv changes broke PixivUtil #483

Closed charzho closed 5 years ago

charzho commented 5 years ago

Prerequisites

Description

Pixiv appears to have changed their website once again.

Steps to Reproduce

  1. Download an image or artist's ID

Expected behavior: Download should work

Actual behavior: Return OAuth or key error

Versions

You can get this information from executing PixivUtil2.py --help.

todesschnitzel commented 5 years ago

Getting the same error:

Unknown errors from previous operation Image: ........ ==> ,Error occurred at the OAuth process. Please check your Access Token to fix this. Error Message: invalid_request,

Deleting your login credentials also doesn't seem to fix the issue

PixivDownloader2 version 20190416

hakata-san commented 5 years ago

Image ID (ID-NUMBER): 1003 ,Error occurred at the OAuth process. Please check your Access Token to fix this. Error Message: invalid_request,, hasDumpPage=Y PixivDownloader2 version 20190416

Same step from other users and problem still remain.

Nandaka commented 5 years ago

try to copy the cookie value?

todesschnitzel commented 5 years ago

I tried it with the Cookie just now. No change. Same error.

Shadark commented 5 years ago

Same here, checked the cookie, is still the same but getting OAuth errors anyways.

Nandaka commented 5 years ago

crap, error when calling https://app-api.pixiv.net/v1/user/detail?user_id=<user_id> api, looks like now need to have separate key. Let see if I can remove this call.

Nandaka commented 5 years ago

I cannot get the oAuth works using this code, based on https://github.com/kokororin/pixiv-api-php/blob/master/PixivBase.php#L74

    def get_oauth_token(self, username, password):
        url = "https://oauth.secure.pixiv.net/auth/token"
        values = {'client_id' : 'bYGKuGVw91e0NMfPGp44euvGt59s',
                  'client_secret' : 'HP3RmkgAmEGro0gn1x9ioawQE8WMfvLXDz3ZqxpK',
                  'device_token' : 'af014441a5f1a3340952922adeba1c36',
                  'username' : username,
                  'password' : password,
                  'grant_type' : 'password'
                   }
        data = urllib.urlencode(values)
        request = urllib2.Request(url, data)
        # request.add_header("Referer", "https://www.pixiv.net")
        # request.add_header("Content-Type", "application/x-www-form-urlencoded")
        request.add_header("Authorization", "Bearer WHDWCGnwWA2C8PRfQSdXJxjXp0G6ULRaRkkd6t5B6h8")
        try:
            oauth_response = self.open(request).read()
            print(json.loads(oauth_response))
        except urllib2.HTTPError as ex:
            print(json.loads(ex.read()))

It return this for me:

{"has_error":true,"errors":{"system":{"message":"access_denied","code":1508}}}

any idea?

Nandaka commented 5 years ago

got it working, should be fixed if you get the latest source code. compiled version to follow up after testing.

kattjevfel commented 5 years ago

the fix works great on my end, thanks for your quick work! <3

Kilo19 commented 5 years ago

Latest commit gives me another error:

Input: 2 Image ids: 74335137 Processing Image Id: 74335137 No OAuth token available yet, retrieving... Image ID (74335137): 'str' object has no attribute 'format1' Stack Trace: (<type 'exceptions.AttributeError'>, AttributeError("'str' object has no attribute 'format1'",), <traceback object at 0x22DCBB98>)

whinette commented 5 years ago

Same as @Kilo19. \ I deleted the cookie and now I got this:

Logging in...
Server reply: {u'captcha': u'Authenticate your profile after keying in your pixiv ID, e-mail address or password.', u'lock': u"You've entered incorrect passwords too many times. Your account has been temporarily locked. Please try again later. "}

I have no issue with the website though.

The lock is quite short (less then 5mn), I still have the issue mentionned by @Kilo19

NHOrus commented 5 years ago

Meanwhile, I missed most of the events, but thank you.

whinette commented 5 years ago

I've fixed the typo and it works using a sub account. It is not worth a PR I guess ..

whinette commented 5 years ago

I got these error when tring to dl with my main account (option 8 or 1)

Processing Image Id: 74337141
No OAuth token available yet, retrieving...
Failed to get OAuth Token: {u'has_error': True, u'errors': {u'system': {u'message': u'104:\u7121\u52b9\u306a\u30e6\u30fc\u30b6\u30fc\u30a2\u30ab\u30a6\u30f3\u30c8\u3067\u3059\u3002', u'code': 1508}}}
Image ID (74337141): local variable 'msg' referenced before assignment
Stack Trace: (<type 'exceptions.UnboundLocalError'>, UnboundLocalError("local variable 'msg' referenced before assignment",), <traceback object at 0x7f6df57cbb48>)

Translating the error: 無効なユーザーアカウントです。 Weird ... I can log in with these credentials.

Nandaka commented 5 years ago

@whinette That message is similar with this error {"has_error":true,"errors":{"system":{"message":"access_denied","code":1508}}}, I think this is because too many invalid login? It should be ok after waiting for some times (not sure for how long it will reset).

Nandaka commented 5 years ago

also, current logic require you to save the username/password in the config.ini, maybe I can change it to remember based on what was entered when starting the application.

whinette commented 5 years ago

also, current logic require you to save the username/password in the config.ini, maybe I can change it to remember based on what was entered when starting the application.

Maybe I understand incorrectly, but I use this tool mainly unatended and having to login manually would break its usage for me :/

@whinette That message is similar with this error {"has_error":true,"errors":{"system":{"message":"access_denied","code":1508}}}, I think this is because too many invalid login? It should be ok after waiting for some times (not sure for how long it will reset).

I was banned (no info but I suppose excessive download/call on their site) about 6 month ago. I can still use the website but not the mobile app. If I understand correctly, their app use their OAuth and I'm banned from using it.

Nandaka commented 5 years ago

@whinette if you save it to config.ini, then you don't need to key in anything. https://github.com/Nandaka/PixivUtil2/commit/68d9a2565ccba50979aeaeb09210045ba3f0e328#diff-30eb981ce68adac654c4cdeb81b2924cR306 << this changes will allow it to remember the entered username/password in the memory when you don't save it in config.ini.

The app-api is used to resolve the member token (https://github.com/Nandaka/PixivUtil2/blob/master/PixivModelWhiteCube.py#L68) as I cannot find other non-app-api that can resolve the value.

So far I got this api endpoints, but no artist token available

Maybe I can use: https://www.pixiv.net/rpc/get_work.php?id=74318310 but this require the image_id.

Manifestoliving commented 5 years ago

Im having this error too.

Unknown errors from previous operation Image: ........ ==> ,Error occurred at the OAuth process. Please check your Access Token to fix this. Error Message: invalid_request,

I dont know how to apply the source code on program.

Can you upload the new version of pixivdownloader if you fix it?

Nandaka commented 5 years ago

try https://github.com/Nandaka/PixivUtil2/releases/tag/v20190423-beta1

Manifestoliving commented 5 years ago

Error occured. Maybe similiar with above problem?

Failed to get 0Auth Token: {"has_error":true,"errors":{"system":"Missing parameters. ₩u0022username₩u0022 and ₩u0022password₩u0022 required","code":1508}}} Member ID (1774701): 1508 Missing parameters. "username" and "password" required, hasDumpPage=Y

GerhardSauer commented 5 years ago

beta 1 works for me. Case 1: download by member id -> works Case 2: download latest from bookmarks -> works

hakata-san commented 5 years ago

Tried download by member id and download latest from bookmarks and works.

Hecatom commented 5 years ago

An error ocurred for me using the beta while trying to download by member id

Failed to get OAuth Token: {"has_error":true,"errors":{"system":{"message":"Missing parameters. \u0022username\u0022 and \u0022password\u0022 required","code":1508}}} Member ID (4486233): 1508 Missing parameters. "username" and "password" required, hasDumpPage=Y

Nandaka commented 5 years ago

Try to key in the username/password in the config.ini?

Hecatom commented 5 years ago

Try to key in the username/password in the config.ini?

That worked

RndUser0 commented 5 years ago

I had the same issue as Hecatom. After entering username and password in the config.ini, it works. Thank you, Nandaka.

ahelpfulperson commented 5 years ago

Putting username and password in config.ini works, but it feels unsafe to keep usernames and passwords in plain text.

Nandaka commented 5 years ago

the other choice is to always key in your username/password everytime you start the application

ahelpfulperson commented 5 years ago

I've been trying that, but it just spits out this error:

Failed to get OAuth Token: {"has_error":true,"errors":{"system":{"message":"Missing parameters. \u0022username\u0022 and \u0022password\u0022 required","code":1508}}} Member ID (14469137): 1508 Missing parameters. "username" and "password" required, hasDumpPage=Y

Nandaka commented 5 years ago

updated to remember the username/password from console input.

ahelpfulperson commented 5 years ago

I don't know how to update (Windows executable). Can you upload another release? Or can you tell me how to update?

NekoCoaster commented 5 years ago

You can download the latest beta release zip file here: https://github.com/Nandaka/PixivUtil2/releases/download/v20190423-beta1/pixivutil20190423-beta1.zip Personally, I just overwrite all the files in my older PixivUtil folder with the newer ones in the Zip File and has worked fine for me so far.

ahelpfulperson commented 5 years ago

Oh yes, I know how to do that. But that version you linked is the one that I'm already using, which is giving me the error above.

Nandaka made some changes to the code after that version but I cannot figure out how to update.

scottyseng commented 5 years ago

Yep, using the beta version and deleting old cookie from config.ini and putting in credentials again gives me the same OAuth Token error as above. I also have a Pixiv Premium account if that matters.

Nandaka commented 5 years ago

pixivutil20190424-beta2.zip << try this one?

NekoCoaster commented 5 years ago

Hmm, beta1 seems to work fine for me after I put in my login credentials in the config.ini

ahelpfulperson commented 5 years ago

Beta2 still gives the same error. Logging in works, because I get an email confirmation, but it gives me the error when I want to download something.

After the first log in, I used to be able to leave the username and password blank and PixivUtil would log in using the saved cookie, but now even if I enter my credentials it doesn't work.

ATGNKC39 - Using config.ini to save your credentials works, but I am concerned with the security of storing credentials in plain text.

scottyseng commented 5 years ago

Ah, using beta2, putting login into console (clearing cookie from config.ini) gives the same OAuth error when downloading. However, putting in credentials into config.ini (username / password) does work for me. Thanks for the fix (though dislike having credentials in plain text as mentioned above).

Nandaka commented 5 years ago

If you don't save the login credential in config,ini, you need to key in during the login process, as the oAuth require the username/password (you cannot leave it blank, even if the cookie still valid).

ahelpfulperson commented 5 years ago

Yes, I have been entering username and password each time as we were talking here.

I don't think it's a log in issue because I still get the email confirmation. It's just when I want to download the error pops up saying that it's missing the username/password parameters.

I'll attach screenshots below:

1

2

Nandaka commented 5 years ago

pixivutil20190424-beta2.1.zip updated

ahelpfulperson commented 5 years ago

Looks like everything's working now, thanks!

ikkion commented 5 years ago

Bevare. It seems that with the new login system Pixiv have also implemented fairly severe limits on downloading.

zoram999 commented 5 years ago

I confirm that. Years of using the app with no problems while avoiding using multiple instances, and now I've got an account suspension after about 100 downloads. (Granted, I've got a 4500+ followed on that account which means a lot of new posts every day)

I've just exported my bookmarked userIDs to keep for the worst case of losing the account...

whinette commented 5 years ago

Ok, that sucks (I have around 12000 follows). \ I've used this api to reimport the list of followed user on my subaccount: https://github.com/alphasp/pixiv-api-client \ I've made a dirty script (never wrote js until then, lucky me) to migrate the users exported from pixivutil2 (option e): https://gist.github.com/whinette/4bde849d69a16356abdd995efff2f6aa

Manifestoliving commented 5 years ago

Working after updating my username and password in config. Thank you for your work!

Revemohl commented 5 years ago

That's really disappointing, I guess a way around it could be increasing the randomized delay between downloads or having a waiting period after so many of them, but that would probably require a lot of testing to figure out the possible new limits. I hope there's still a small chance they're temporary.

Nandaka commented 5 years ago

@Revemohl update downloaddelay in config.ini

Hecatom commented 5 years ago

Dunno what they changed again that broke the downloader once more

Failed to get OAuth Token: {"has_error":true,"errors":{"system":{"message":"104:\u7121\u52b9\u306a\u30e6\u30fc\u30b6\u30fc\u30a2\u30ab\u30a6\u30f3\u30c8\u3067\u3059\u3002","code":1508}}} Member ID (15104121): 1508 104:無効なユーザーアカウントです。, hasDumpPage=Y

Tried the 2nd beta to see if it was fixed, but nope It gives me the same error i posed above.