pixiv login method is outdated

[Thesola10]

Further details have already been discussed on upbit/pixivpy#172 and I would like to know if it is possible to update this program to reflect the new login method

[akazukin5151]

Hi there. Yes, I've been aware of this issue. Pixivpy's solution isn't very satisfactory to me, but it might be the most straightforward. I'm not sure if I have time to fix, but I'm happy to accept contributions.

As an alternative I have examined other third party pixiv tools. Ideally I just want a username + password login, which none of the alternatives offer exactly.

• Pixivutil uses mechanize, but it stopped working for me recently too
• Pix-EzViewer (mobile app) worked well with just username and password, but it's now discontinued in favour of the...
• flutter version. This one uses a proper oauth method, opening a page to pixiv for authentication. I think that's the most ideal for security, but I don't know Dart, or its libraries, or webdev.

For now, if you (the reader) is willing to, I suggest using pixez-flutter app for Android and iOS. For koneko, I think it's best to just continue with pixivpy. Another possible consideration is using ucollage instead of (the rather hacky) lscat, which could be a good excuse to overhaul the code.

[akazukin5151]

pixivpy's method is a bit annoying and didn't work on Firefox, but I've managed to get it working ... only for the cloudflare captcha to block me all but one time. Oh well, that's not something koneko or pixivpy can fix. On the bright side, very little changes are required for this approach.

Anyway, can you please test this branch and see how well it works for you? Thanks

Edit: Instructions

• Use https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362 to get your refresh token
• Put the token in the file ~/.local/share/koneko/token
• pip uninstall koneko to remove outdated dependencies
• python koneko/__main__.py to quickly test it

[Thesola10]

When I start the app with or without a config file, I am prompted for my username/password, and at startup I get this message:

[ERROR] auth() but no password or refresh_token is set.
If this is a cloudflare captcha issue, just quit and retry
It is not a problem with koneko or pixivpy
Otherwise, please report to https://github.com/twenty5151/koneko/issues/4
Press 'q' and enter to exit

Am I missing something? How do I test the new login method?

[akazukin5151]

Oh, I haven't removed the username and password config logic yet. Just copy and paste the example config, using any string for the username and password fields.

[Thesola10]

Okay, I managed to obtain a token, and I'm getting this:

requests POST https://oauth.secure.pixiv.net/auth/token error: Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter.
If this is a cloudflare captcha issue, just quit and retry
It is not a problem with koneko or pixivpy
Otherwise, please report to https://github.com/twenty5151/koneko/issues/4

[akazukin5151]

Yep, this is the error message I get too. I tried a few more times and it succeeded only once. Maybe you'll have better luck? I think we'll have to consider alternative methods that other third-party apps are using...

[Thesola10]

yay! I had to use the refresh token

Also i'm using firefox

[Thesola10]

It would seem that the login method returns a pixiv: link, hopefully with the required code, so maybe registering as a link handler might help streamline the login process?

I managed to capture it, it looks like this: pixiv://account/login?code=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Is that useable?

[akazukin5151]

Oh, so that's why chromium prompted me to open a link with xdg-open when it finished, but I didn't pay attention. I suppose the program can handle pixiv:// URIs, but I'll need to research how to do it. (Can terminal programs even do that? I've never seen one before)

Is the login working for you consistently?

[Thesola10]

Here's how I did it, adapted from here

1. Create this file pixiv.desktop with these contents:

   [Desktop Entry]
   Type=Application
   Name=pixiv auth handler
   Exec=koneko url-login %u
   MimeType=x-scheme-handler/pixiv;

2. Register pixiv.desktop by moving it to ~/.local/share/applications or /usr/share/applications

3. update-desktop-database ~/.local/share/applications

4. Profit!

[akazukin5151]

That's amazing! Thanks for the help. I'm currently removing the username and password code and replacing it with the token. If you want to, you can make a PR to add the protocol handler, otherwise I'll get to it soon.

Is koneko url-login a new command? I suppose we could add a new command for the program and have it:

1. extract the code from the URI
2. pass it to the manual login method[1] script
3. get the refresh token
4. add it to the config

[1] I'm going to call this https://gist.github.com/ZipFile/c9ebedb224406f4f11845ab700124362 "manual login method"

[Thesola10]

Is koneko url-login a new command?

Yes, but preferably undocumented, as only the .desktop handler could call it properly. It could also be a separate script like koneko-url-login if you prefer.

Also, don't forget to remove pixiv.desktop from ~/.local/share/applications if the user (or another app) wants to use pixiv: links for something else.

[Thesola10]

Actually, here's a comprehensive list of what the new login flow for koneko will look like:

1. koneko silently copies pixiv-url.desktop to ~/.local/share/applications and runs:
   • xdg-mime default pixiv-url.desktop x-scheme-handler/pixiv
   • update-desktop-database ~/.local/share/applications
2. koneko opens the manual login URL and leaves a message instructing the user to continue logging in in the browser... and closes.
3. koneko url-login parses the pixiv:// URL and requests the refresh token from the obtained code
4. koneko url-login removes ~/.local/share/applications/pixiv-url.desktop just before exiting

The user is then expected to run koneko again, but this time with we already have a valid refresh token.

[akazukin5151]

Sounds good to me, thanks. From my understanding, once koneko url-login parses the code from the pixiv:// URL, the manual login method would be able to immediately generate the refresh token. Then it can already write it to the config file and immediately start koneko. So there's no need for a separate step for the user to "copy the token, run koneko again and paste it". Thoughts?

[Thesola10]

Indeed there is not, but what I mean is that the first koneko run does not prompt anything, it just opens the browser and quits, and koneko url-login does all the login stuff in the background. The user is expected to run koneko on the CLI again, at their convenience, once the whole process is done.

During the process the user sees nothing on the terminal.

[akazukin5151]

Hmm, I suppose leaving it to the user's choice is good, but I would at least like it to write the token to the config. Perhaps an interactive prompt that asks whether the user wants to manually copy and paste the token then run koneko again, or have the script write the token and launch koneko automatically? I mean, every other user facing app with a login will not request the user to restart upon logging in. And neither did koneko before the pixiv API changed. The login flow should be closer to a simple username + password flow: open app -> log in -> use app. Compare it to open app -> log in -> copy and paste something -> open app.

It would be doubly bizarre when we can easily automate the copy & paste & launch steps, so why burden the user with that? I'm genuinely curious; is it for transparency of the process, or something else?

During the process the user sees nothing on the terminal.

Even if the user is doing it manually, koneko should at least print out what to copy and where to paste, and the need to re-run the program.

the first koneko run does not prompt anything, it just opens the browser and quits

I agree that in the first run koneko would not prompt anything; the prompt should be in koneko url-login at the end of the process.

[Thesola10]

koneko url-login is NOT a prompt. The user will never see it. We tell the browser that it's the program to use to open pixiv: URLs through our .desktop file. Everything in koneko url-login happens in the background, without a terminal attached.

[Thesola10]

From the user's point of view, it's:

Run koneko → browser opens → complete login on browser → run koneko again, and this time it's logged in.

[akazukin5151]

Ah, the koneko url-login is invisible to the user, I see. In a GUI I suppose there would be a prompt, but it would be unconventional on the terminal.

Run koneko → browser opens → complete login on browser → run koneko again, and this time it's logged in.

So koneko url-login will still write the token into the config file? That sounds good.

[Thesola10]

The user is then expected to run koneko again, but this time with a valid refresh token.

Oh, that must have been the confusing part. I meant, this time, they already have a valid refresh token. sorry

[akazukin5151]

Can you try 4e3316d24bc1f22c27e7c518842981897e83cc0a and report if it works? I have a few problems but I think it's a problem with my setup not the code. I'll continue testing and report later. But at the very least koneko needs to give instructions on what if my problem happens to the user.

Just fetch the commit and run python setup.py develop then koneko

[akazukin5151]

So there are two problems for me, neither can be solved by koneko. The first is Firefox isn't able to open links in app (the "open link in ?" prompt). The obvious mitigation is to use another browser. Problem solved, right? Well, my second problem is that Chromium opens the pixiv:// links with ... Firefox, not koneko-url-login. Even if Chromium is (supposedly) set as the default browser.

Problem 1: Firefox isn't able to open the pixiv:// link Using a clean Firefox profile, I get Specified end-point doesn't exist -- either it failed to get to the pixiv:// link or couldn't open it. On further retries a blank page just appears. This persists even after disabling Firefox's default tracking protection (which shouldn't matter because it worked for me on Chromium WITH ublock origin). (But using the original script (python pixiv_auth.py login), I was now able to get to the point where code= appears.) Not such a big deal, just use another browser. If only the other browser works...

Problem 2: Chromium isn't properly opening the pixiv:// link either Chromium isn't properly opening the pixiv:// link at the end, it always opens in Firefox, which decides to do a websearch, exposing the code to the internet (good thing it has a short expiry time). However, if I manually run xdg-open "pixiv://<...>", it correctly passes it to koneko-url-login and everything works out.

Both problems can't be solved by koneko. Unless if you have any ideas how to debug my mess, the only solution is to adapt and offer the user to use the manual script themselves (manually getting the code from the Network tab and pasting it), then manually copy and pasting the refresh_token to koneko's config file.

Side note: if koneko-url-login exits with a failure, xdg resorts to opening the pixiv:// link with Firefox, which decides to do a websearch again... so we need to catch any and all errors in koneko-url-login

[Thesola10]

On my end, running with Firefox worked perfectly without manual intervention. May I recommend resetting your Firefox profile or using the Flatpak version of Firefox?

Didn't try Chrome, but it would seem your environment is having issues I cannot reproduce on my end?

Edit: "Specified end-point does not exist" isn't a documented Firefox error as far as I know, definitely try creating a new Firefox profile.

[akazukin5151]

May I recommend resetting your Firefox profile or using the Flatpak version of Firefox?

I already tried with "a clean Firefox profile", didn't work. Flatpak Firefox has better success; successfully prompted to open the link (with "System Handler"), and was able to select "koneko pixiv login handler" ... but there's no indication that koneko-url-login even opened. There was no configuration file generated. Tried adding os.system('notify-send test') in the first line of url_login.main, no notification seen. Furthermore, the .desktop file was not removed despite being in this line. I'll have a look at the logs...

(I set flatpak Firefox as the default browser; while xdg-open recognized it, apparently Python's webbrowser.open (https://docs.python.org/3/library/webbrowser.html) did not and opened the initial link in my base Firefox. Well, not a problem as I can copy and paste the link into flatpak Firefox. But it could be an indicator of a bigger problem causing the above)

Does the .desktop file actually get written to the right place or is there a catch (permissions maybe) on your end?

As "koneko pixiv login handler" is in the list of apps that can open the link, I think there's little doubt that the .desktop file is working. The permission is -rw-r--r--. Furthermore, in my original report, xdg-open on the pixiv:// link correctly passes the link to koneko-url-login, indicating it is working. Additionally I can see "koneko pixiv login handler" in rofi.

Also, is Chromium already running before you run koneko?

May I ask why you ask this question? I tried running koneko first, then pasting the link to Chromium. Interestingly, after logging in (and passing the captcha) it just ... does nothing. The loading icon animates for a while then disappears. In contrast if Chromium is already running, it successfully continues and prompts to open the pixiv:// link. If I use flatpak Firefox instead, everything is the same as above.

Thanks for the help

Edit: I'm going to ensure all programs are updated and try a reboot

[akazukin5151]

(After updating + rebooting:) Alright, Chromium works perfectly now. On Firefox I get "Invalid request.". On Flatpak Firefox, the behaviour is the same -- I select "koneko pixiv login handler" and nothing happens.

("Specified end-point does not exist" is just Temporary containers putting the page in a new container tab; ignore it)

[Thesola10]

Tried adding os.system('notify-send test') in the first line of url_login.main

Could be a good idea for the final product, but using a libnotify python module to reduce reliance on PATH and stuff

[akazukin5151]

Right, I was thinking that there should be some visual indicator that the handler script has finished writing the config file and koneko should be ran again

[akazukin5151]

I think the only functionality that you can test out is whether the notification works. The user ID should be fine