upbit / pixivpy

Pixiv API for Python
https://pypi.org/project/PixivPy3/#files
The Unlicense
1.78k stars 148 forks source link

pixiv seems not allowing username/password login now #158

Open roytam1 opened 3 years ago

roytam1 commented 3 years ago

server returning JSON:

{"has_error":true,"errors":{"system":{"message":"The grant type is unauthorized for this client_id","code":1508}},"error":"invalid_grant"}
roytam1 commented 3 years ago

background information:

In pixiv android Version 5.0.234 in 2021-02-08, there is a line:

and for testing, I tried to install 5.0.220.1 apk and tried to login, and it pops out custom browser window and jumps to pixiv standard login form.

roytam1 commented 3 years ago

a cross reference is found: https://github.com/upbit/pixivpy/issues/140#issuecomment-708976828

kilosonc commented 3 years ago

I encountered the same problem

HTTP 400: {"has_error":true,"errors":{"system":{"message":"Invalid grant_type parameter or parameter missing","code":1508}},"error":"invalid_grant"}

I want to use set_auth, but I don't know which is access token, there're to many pairs in cookie

roytam1 commented 3 years ago

I want to use set_auth, but I don't know which is access token, there're to many pairs in cookie

they're not exist in cookies or IndexedDB or Local Storage.

evazion commented 3 years ago

In my project I had to revert back to using the Ajax API. The problem is that the grant_type=password authentication method is no longer supported. The Pixiv app now logs in through https://accounts.pixiv.net/login, but this page is protected by Google reCAPTCHA, which seems impossible to circumvent.

I think that as long as you have a refresh token, you can keep using it, but getting one is tricky since you have to login with the mobile app and sniff traffic to get it.

https://github.com/danbooru/danbooru/blob/39cc3ed5cf913499093d2f641d70d7682a14fa42/app/logical/pixiv_ajax_client.rb

emesh0620 commented 3 years ago

If password authentication is disabled, is there no choice but to scrape with BeautifulSoup from now on? Unbelievable

Mouwoo commented 3 years ago

Screenshot_20210210_155236 经测试该账号和密码可以pixiv官网正常登录

emesh0620 commented 3 years ago

Last year, you said that you are trying not to register with a leaked password, but you are also considering methods such as multi-factor authentication, so it is time to think about some fundamental measures. It may be. (Link to Japanese)

https://inside.pixiv.blog/2020/01/22/180000

kuronekouwu commented 3 years ago

Pixiv was changed grant_type password to authorization_code. But it's so hard to login because pixiv using login with OAuth 2.0 image

ZipFile commented 3 years ago

Pixiv now follows RFC7636. We won't be able to use (old) password login again, but there is a way to automate token retrieval without traffic sniffing. I've implemented a simple script to retrieve access/refresh tokens. Right now it requires single manual step inspecting dev console, but in general, entire flow could be fully automated using browser extensions: you can add request handler and extract code param from the url.

upbit commented 3 years ago

Not being able to login with password is really bad news.. Thank you all for these new methods. My current network is relatively poor, and the login method will be updated as soon as possible after verification.

Happy Chinese New Year!

farmerswalker commented 3 years ago

pixiv seems not allowing username/password login now #158😢😢 It's really sad..😢 Is there any solutions yet?

MFWT commented 3 years ago

群友看不到涩图了,上issues一看果然有蹊跷 根据我这边程序的推断,Pixiv登录方式具体的更换时间应该是(UTC+8)2021/2/8 22:30 - (UTC+8)2021/2/9 22:30之间 还是希望尽快恢复罢

Akise commented 3 years ago

Pixiv now follows RFC7636. We won't be able to use (old) password login again, but there is a way to automate token retrieval without traffic sniffing. I've implemented a simple script to retrieve access/refresh tokens. Right now it requires single manual step inspecting dev console, but in general, entire flow could be fully automated using browser extensions: you can add request handler and extract code param from the url.

这个方法有效耶, 可以用pyside2中内嵌的浏览器自动登录web版自动获取code, 登录后的phpssid保存起来下次用, 作者什么时候更新登录方法啊, 过年想改改pixiv爬虫代码结果登录都不行. @ZipFile

Xdynix commented 3 years ago

Pixiv now follows RFC7636. We won't be able to use (old) password login again, but there is a way to automate token retrieval without traffic sniffing. I've implemented a simple script to retrieve access/refresh tokens. Right now it requires single manual step inspecting dev console, but in general, entire flow could be fully automated using browser extensions: you can add request handler and extract code param from the url.

这个方法有效耶, 可以用pyside2中内嵌的浏览器自动登录web版自动获取code, 登录后的phpssid保存起来下次用, 作者什么时候更新登录方法啊, 过年想改改pixiv爬虫代码结果登录都不行. @ZipFile

@Akise 不需要作者更新登录方法啊,本来就支持用refresh_token登录。你人工获取一次后存在本地,够用很长时间。api.auth(refresh_token='refresh_token')

MFWT commented 3 years ago

Pixiv now follows RFC7636. We won't be able to use (old) password login again, but there is a way to automate token retrieval without traffic sniffing. I've implemented a simple script to retrieve access/refresh tokens. Right now it requires single manual step inspecting dev console, but in general, entire flow could be fully automated using browser extensions: you can add request handler and extract code param from the url.

这个方法有效耶, 可以用pyside2中内嵌的浏览器自动登录web版自动获取code, 登录后的phpssid保存起来下次用, 作者什么时候更新登录方法啊, 过年想改改pixiv爬虫代码结果登录都不行. @ZipFile

@Akise 不需要作者更新登录方法啊,本来就支持用refresh_token登录。你人工获取一次后存在本地,够用很长时间。api.auth(refresh_token='refresh_token')

refresh_token要怎样获取呢?群友们没有涩图看饿到嗷嗷叫了

Xdynix commented 3 years ago

Pixiv now follows RFC7636. We won't be able to use (old) password login again, but there is a way to automate token retrieval without traffic sniffing. I've implemented a simple script to retrieve access/refresh tokens. Right now it requires single manual step inspecting dev console, but in general, entire flow could be fully automated using browser extensions: you can add request handler and extract code param from the url.

这个方法有效耶, 可以用pyside2中内嵌的浏览器自动登录web版自动获取code, 登录后的phpssid保存起来下次用, 作者什么时候更新登录方法啊, 过年想改改pixiv爬虫代码结果登录都不行. @ZipFile

@Akise 不需要作者更新登录方法啊,本来就支持用refresh_token登录。你人工获取一次后存在本地,够用很长时间。api.auth(refresh_token='refresh_token')

你有几个账号? 都是人工获取? refresh_token能存活多久? 为什么不做成自动化?

一个账号,偶尔才用。目前观察到的来看存活至少一个月,上个月获取的还能用。ZipFile大佬的脚本都说了目前还需要人工介入。而操作浏览器获取refresh token的内容我认为有点超出pixiv api库的职能范畴了。

Xdynix commented 3 years ago

Pixiv now follows RFC7636. We won't be able to use (old) password login again, but there is a way to automate token retrieval without traffic sniffing. I've implemented a simple script to retrieve access/refresh tokens. Right now it requires single manual step inspecting dev console, but in general, entire flow could be fully automated using browser extensions: you can add request handler and extract code param from the url.

这个方法有效耶, 可以用pyside2中内嵌的浏览器自动登录web版自动获取code, 登录后的phpssid保存起来下次用, 作者什么时候更新登录方法啊, 过年想改改pixiv爬虫代码结果登录都不行. @ZipFile

@Akise 不需要作者更新登录方法啊,本来就支持用refresh_token登录。你人工获取一次后存在本地,够用很长时间。api.auth(refresh_token='refresh_token')

refresh_token要怎样获取呢?群友们没有涩图看饿到嗷嗷叫了

@MFWT 楼上ZipFile大佬的脚本https://github.com/upbit/pixivpy/issues/158#issuecomment-777815440

Akise commented 3 years ago

现在pixiv app端登录会打开个浏览器登录, 会遇到谷歌reCAPTCHA, 我一直使用pyside2内嵌的浏览器模拟登录pixiv, 小概率触发验证码. 或者用pyppeteer . @ZipFile @upbit @Xdynix @evazion

upbit commented 3 years ago

At present, the login method of the previous app is no longer available 😢 As @Xdynix said, integrating OAuth2 login in pixivpy is not reasonable, but we can consider using third-party libraries to simplify token acquisition.

In the short term, you can use the method provided by @ZipFile to manually obtain the token through the browser. In addition, using the refresh(refresh_token) method, you can retrieve the access_token to avoid expiration (default 3600 seconds), this may also be a solution to keep login.


目前,之前App的登录方法都不再可用😢 正如 @Xdynix 所说,在pixivpy中集成OAuth2登录是不合理的,不过我们可以考虑使用第三方库来简化access_token的获取。

在短期内,您可以使用 @ZipFile 提供的方法通过浏览器手动获取tokens。 另外,使用refresh(refresh_token)方法,可以重新获取access_token以避免过期(默认3600秒),这也许可以作为保持登录的另一个解决方案。


临时解决方法:

  1. 根据 @ZipFile 的方法取的refresh_token
  2. 采用 api.auth(refresh_token=_REFRESH_TOKEN) 方式登录 (refresh_token目前测试中未变化过,推测可以保存后长时间使用;如果失效可以用1的方法再获取一次)
    # api.login(_USERNAME, _PASSWORD)
    api.auth(refresh_token=_REFRESH_TOKEN)
Akise commented 3 years ago

不过我们可以考虑使用第三方库来简化access_token的获取。

什么方法? 半自动+手动? 我想找个全自动登录的思路. @upbit

Do you have a fully automatic login scheme? @ZipFile

upbit commented 3 years ago

不过我们可以考虑使用第三方库来简化access_token的获取。

什么方法? 半自动+手动? 我想找个全自动登录的思路. @upbit

Do you have a fully automatic login scheme? @ZipFile

全自动估计比较复杂(至少想到的得用headless的浏览器模拟)。半自动是指,可以把callback转向本地监听的server,从而自动获取token(避免过期),不过目前还没找到可行的库

ZipFile commented 3 years ago

I don't have much time to implement fully automated solution, so I shared bare minimum implementation so that everyone could recover ASAP.

Algorithm itself is not something unique, as it is described by aforementioned RFC. Important parts are urls and params used in Pixiv auth flow. OAuth PKCE works well on mobile because it relies on OS implementation of deep links. On desktop it's pain in the ass to deal with, so we have to use embedded/debug browser (QtWebEngine, Puppeteer, etc...) or extension with elevated permissions to extract the code.

emesh0620 commented 3 years ago

Many of you may be worried that you'll have to log in and set it up again, but you'll still be able to log in after several attempts, so even if you use Auth, it will last quite a while. But I can't say for sure because we don't examine it every few hours or days.

upbit commented 3 years ago

I tested the login method of refresh_token, and I can get access_token successfully (and refresh_token not changed):

  1. Within 3600 seconds, success
  2. About 1-2 hours, success
  3. About 3-4 hours, success
  4. More than 24 hours, success

Temporarily use the following method, you should be able to keep logged in:

api.auth(refresh_token=_REFRESH_TOKEN)
MFWT commented 3 years ago

楼上有人问RefreshToken管不管用,我在这说一句,管用 我回复一条issue的时候,是2/14 image 然后一直到现在,都能用(注意看ranking.csv修改时间) image

upbit commented 3 years ago

Update the semi-automatic script, get the code through chromedriver


更新了半自动脚本,通过chromedriver获取code

https://gist.github.com/upbit/6edda27cb1644e94183291109b8a5fde

Mapaler commented 3 years ago

我这几天自己研究后把我的脚本更新了,果然这边也有问题。

有需要的可以参考我的做法。 https://github.com/Mapaler/PixivUserBatchDownload/wiki/%E8%8E%B7%E5%8F%96APP%E7%99%BB%E9%99%86%E9%93%BE%E6%8E%A5

也是第一次用 pivix 私有链接登陆后保存auth数据,之后就每次只刷新token了。

具体网络请求代码可以参考 https://github.com/Mapaler/PixivUserBatchDownload/blob/f1d7286ce033a242d0278feac0fad99380c30a51/PixivUserBatchDownload.user.js#L379

reimu1234 commented 3 years ago

Update the semi-automatic script, get the code through chromedriver

更新了半自动脚本,通过chromedriver获取code

https://gist.github.com/upbit/6edda27cb1644e94183291109b8a5fde

请问手机端能拿到吗 pc 的试了拿到code后就过不去了 开了全局

upbit commented 3 years ago

Update the semi-automatic script, get the code through chromedriver 更新了半自动脚本,通过chromedriver获取code https://gist.github.com/upbit/6edda27cb1644e94183291109b8a5fde

请问手机端能拿到吗 pc 的试了拿到code后就过不去了 开了全局

如果能输出code,可以参考requests的代理配置,对95行的post请求增加代理

reimu1234 commented 3 years ago

Update the semi-automatic script, get the code through chromedriver 更新了半自动脚本,通过chromedriver获取code https://gist.github.com/upbit/6edda27cb1644e94183291109b8a5fde

请问手机端能拿到吗 pc 的试了拿到code后就过不去了 开了全局

如果能输出code,可以参考requests的代理配置,对95行的post请求增加代理

都试了才来打扰😂

upbit commented 3 years ago

试试换个代理?这个和ZipFile的原始脚本一样,都用requests直接请求的。或者试下 Mapaler 的GUI工具

ps: 手机端没试过,如果你用term跑python应该可以;纯浏览器会打开app,无法拦截code

Mapaler commented 3 years ago

@yzwik 手机端用浏览器获取code会打开P站APP,除非你把P站APP卸载掉,自己写一个注册pixiv协议的APP来获取code。 获取的登陆code只有几十秒的生存时间,需要尽快使用,不然就过期了。你电脑获取的话,需要马上转移到手机上使用。 或者你可以电脑上登陆了后,把刷新code拿去直接添加刷新code,刷新code的生存时间非常长。

txperl commented 3 years ago

Update the semi-automatic script, get the code through chromedriver 更新了半自动脚本,通过chromedriver获取code https://gist.github.com/upbit/6edda27cb1644e94183291109b8a5fde

请问手机端能拿到吗 pc 的试了拿到code后就过不去了 开了全局

如果能输出code,可以参考requests的代理配置,对95行的post请求增加代理

都试了才来打扰😂

可以通过 pixivpy bypass SNI 的方式,先获取 API 的真实 IP,然后直接请求,就不需要代理了。

参考脚本: https://github.com/txperl/PixivBiu/blob/24119157facdef4eb31ab6492eb32b583d01c7bf/app/core/biu/login_token.py

参考实现: https://github.com/txperl/PixivBiu/blob/24119157facdef4eb31ab6492eb32b583d01c7bf/app/core/biu/main.py#L164

reimu1234 commented 3 years ago

试试换个代理?这个和ZipFile的原始脚本一样,都用requests直接请求的。或者试下 Mapaler 的GUI工具

ps: 手机端没试过,如果你用term跑python应该可以;纯浏览器会打开app,无法拦截code

确实是代理的问题 换了个节点解决😂 感谢大佬帮助

TianyiShi2001 commented 3 years ago

如果是有reCAPTCHA的话那就不可能全自动了吧

不过cookie不是太快过期的话倒也不太影响使用

eggplants commented 3 years ago

https://gist.github.com/upbit/6edda27cb1644e94183291109b8a5fde

I have written a library/CLI based on this script to get tokens headlessly in selenium. Check: https://github.com/eggplants/get-pixivpy-token