Open foobar546 opened 2 years ago
The API requires a login for any action (afaik). It used to not have this "horrible" oauth procedure, but iirc it was deprecated so now we're stuck with this.
Consider not using the API.
With regard to "1st WTF":
There's an option for that: extractor.oauth.browser Setting this to false allows you to copy and paste the URL manually to whichever browser you choose..
Not sure if not using the API is even an option at all. Haven't used the site in a while, but it used to be the case that an account was required for viewing original res artworks even in the browser, and there wasn't really any workaround, as far as I recall.
Account is required for 18+ marked content. Yes, the current process is painful, but you only have to do it once, the token does not expire. It is much preferable to cookies or whatever else. Using API over parsing HTML is also much preferable from a maintenance viewpoint. Parsing HTML is very fragile, if at all possible with such a javascript-heavy site.
I agree that the documentation could be expanded regarding the timeout, if this is currently not mentioned.
Account is required for 18+ marked content.
I tried this with content that can be accessed without account in the browser. But you might need an account to get the full sized images.
but you only have to do it once, the token does not expire.
But this happened to me, when it was trying to "refresh" the token.
Parsing HTML is very fragile, if at all possible with such a javascript-heavy site.
Using a website's private API is fragile too, as demonstrated by the twitter extractor.
This is how I can login to Pixiv using gallery-dl
on Windows 10. For me, this is the easiest way for login to Pixiv.
I was able to login without issues after creating config.json
file as following example from gallery-dl-example.conf.
First I got my pixiv token (refresh-token) following gppt: get-pixivpy-token readme.md. So after install I download chromedriver.exe (you should check your Chrome version before download) and I include the ChromeDriver location in my PATH environment variable as mentioned in WebDriver for Chrome: Getting started. Example of my path: C:\chromedriver_win32
After that I login with this code:
gppt login-headless -u <user-email> -p <password>
Then, copy your refresh_token
. Now, edit your configuration file config.json
in one of the locations mentioned in Configuration.
Windows:
%APPDATA%\gallery-dl\config.json
%USERPROFILE%\gallery-dl\config.json
%USERPROFILE%\gallery-dl.conf
(
%USERPROFILE%
usually refers to the user's home directory, i.e.C:\Users\<username>\
)
Example following gallery-dl-example.conf:
"pixiv":
{
"#": "override global archive setting for pixiv",
"archive": "~/gallery-dl/archive-pixiv.sqlite3",
"#": "set custom directory and filename format strings for all pixiv downloads",
"filename": "{id}_{title}_p{num}.{extension}",
"directory": ["Pixiv", "Works", "{user[name]}_{user[id]}"],
"refresh-token": "this_is_your_refresh-token",
"#": "transform ugoira into lossless MKVs",
"ugoira": true,
"postprocessors": ["ugoira-copy"],
"#": "use special settings for favorites and bookmarks",
"favorite":
{
"directory": ["Pixiv", "Favorites", "{user[id]}"]
},
"bookmark":
{
"directory": ["Pixiv", "My Bookmarks"],
"refresh-token": "this_is_your_refresh-token"
}
},
And now it works for me. In my terminal I got a message like this:
PS C:\Users\USERNAME> gallery-dl "https://www.pixiv.net/en/artworks/NUMBERS"
[pixiv][info] Refreshing access token
[postprocessor][warning] module 'ugoira-copy' not found
* .\gallery-dl\Pixiv\Works\ARTIST\NUMBERS_TITTLE_p0.png
I hope this results helpful for someone.
Note: don't ask me about uigora. I'm still figuring out about it.
Edit: Following this comment the problem with uigora is solved.
Uh that's very fancy. Since it's a pip thing, is there a chance gallery-dl could use this as a library?
Although support for something like Selenium etc. might be worthy to consider as a potential goal.. For now it maybe just Pixiv, but other sites might follow suit eventually..
The token died again. It's much easier to download images manually instead of trying to use gallery-dl on pixiv.
The token died again. It's much easier to download images manually instead of trying to use gallery-dl on pixiv.
Just set your token in configurations with config.json
or gallery-dl.conf
and then you won't have this issue again. Until now this works for me without any trouble so far.
@rodnm thanks! To clarify:
"bookmark": { "directory": ["Pixiv", "My Bookmarks"], "refresh-token": "this_is_your_refresh-token" }
Does refresh-token
have to be present under every subcategory?
Is having it just in the main category (Pixiv, in this case) not enough?
The token died again. It's much easier to download images manually instead of trying to use gallery-dl on pixiv.
After some clicks, I succeeded. And the problem with me is not the "Complicated" steps but I forgot to remove the username and password, and it was continuously saying "refresh-token required" but I certainly GOT the tokens already and updated it in gallery-dl.conf. At first I thought maybe it was the conf file didn't come into effect and I didn't know how to deal with that. Then I suddenly realized the usename and password is redundant(Maybe because I saw @mikf mentioned the user and pass login is not supported anymore), so I deleted those and it worked. BTW, this program is way much convenient to batch download images, etc. Unless you just want to download several images individually.
I tried using cookies, and it works. But whether I log in or not, it downloads all the artworks anyway. Now I'm confused.
{
"extractor": {
"pixiv": {
"cookies": {
"PHPSESSID": "xxx..."
}
}
}
}
Cookies are only needed to detect and download R-18 limit_sanity_level_360.png
works for https://www.pixiv.net/en/users/USERID/artworks
URLs. You currently don't need them for anything else, only a refresh-token
.
I tried to configure the refresh-token using gallery-dl oauth:pixiv
, but I received the following error:
[oauth][error] HttpError: '400 Bad Request' for 'https://oauth.secure.pixiv.net/auth/token'
Even if I don't configure cookies or a refresh-token, I can still download R18 or R18-G images from
https://www.pixiv.net/en/users/USERID/artworks
URLs. But I'm using a Japanese IP, and I'm not sure if this affects anything.
When trying to download a pixiv page that requires no login, gallery-dl still wants a login. It suggests you to use
gallery-dl oauth:pixiv
. When you do this, it starts chrome (1st WTF), then you enter your login details, and you're supposed to use developer tools (2nd WTF), and quickly grab some random network request (3rd WTF) and extract some code which isn't all that uninvolved (4th WTF), and if you're no fast enough it times out (5th WTF), which isn't mentioned anywhere (6th WTF), and then you need to logout and try again (7th WTF).I actually succeeded after many retries. But couldn't this be better? It should not require a login if not needed, and it should not require this horrible oauth procedure.