Authenticate with e621

[ButtNumz]

There is a global blacklist for users that aren't logged in, so without authentication some images cannot be downloaded.

[biznizz]

So wait, you mean that if one tries to download/rip a pool from there, it'll be blocked or something?

Yeesh, that place just gets dumber and dumber the more time goes on.

Maybe something can be done like adding user credentials in configuration in a later build?

[ButtNumz]

There are some workarounds that can be utilized as seen here But yes, images are (mostly) outright blocked from being seen without the use of an API key or authorization credentials such as a user/pass pair or cookies.

I'd prefer the latter since the use of the API is limited per account.

[mikf]

Authentication was already implemented before and works the same way as with Danbooru: username needs to be your login name and password the API key you get from "Manage API Access" in your user profile. This is now documented with https://github.com/mikf/gallery-dl/commit/5b676ea59dfc4d6d70732e656b15d1b41c1ce562.

so without authentication some images cannot be downloaded.

Images with blacklisted tags don't have a download URL in their API response, but you can reconstruct it from the other info fields: https://github.com/mikf/gallery-dl/blob/e0b0e8d62a830d97670986d1b6ab8c04713a0b9f/gallery_dl/extractor/e621.py#L55-L58 Is it only the presence of a blacklisted tag or are there any other criteria where an image doesn't even appear in an API response? Do you have a concrete example of something that can only be downloaded with authentication?

the use of the API is limited per account.

Are you sure about this? https://e621.net/help/api mentions nothing about that. It only warns about a 1-request-per-second limit for API requests, not file downloads, but that is a lot more than necessary.

[biznizz]

        "e621":
        {
            "username": USERNAME,
            "password": APIKEY
        },`

Like this then. Is there any timelimit to refresh/regenerate the API key in this case, or is it a one and done deal?

EDIT: Looking at the configuration.rst page in docs, it mentions that a .netrc file. I tried looking at what that was and just got confused. Hope that keeping credentials in config will work fine without it. I have no idea how to create, set or do anything with this netrc thing.

[ButtNumz]

If you look at someone's profile there is a section that is labeled as API Limits | 60 / 60 (may not be up to date) I don't know if the account limit is actively being used atm, but I can't think of anything else it could be.

I didn't know authentication was already implemented. Also, as I said, there are workarounds for the global blacklist, and you've seem to have already addressed this with image url reconstruction. Nice!

[biznizz]

@ButtNumz, huh. Wonder if that 60 is for images downloaded or the amount of times a process that could be run on that API key...

@mikef, so we should update the e621.py code with the new one? Will that also be updated with the next gallery update and we should wait for that? I don't have any real rush to rip from there, but I'd prefer to know that I can take care of it now than forget about it later.

[mikf]

@ButtNumz The account limit (API Limits | 60 / 60) most likely only applies to API updates and not read-only operations like gallery-dl does. e621 is using a rather recent version of Danbooru's code, and there you also have the same API limit listed in your user profile. The documentation there says

• "API reads are not limited" (https://danbooru.donmai.us/wiki_pages/help:api)
• "Make up to 10 API updates at once with a regen rate of 1 update per second." (https://danbooru.donmai.us/wiki_pages/help%3Ausers)

So I don't think you need to worry about any API rate limits or the like.

@biznizz

Like this then. Is there any timelimit to refresh/regenerate the API key in this case, or is it a one and done deal?

Yes, like this, and it is a "one and done deal"

so we should update the e621.py code with the new one?

This code is already in place since commit https://github.com/mikf/gallery-dl/commit/ebc70e87ce9659df987b43603ad6164ee09e56d2

Oh, and don't worry about any .netrc stuff. It's just an alternate way to supply username and password, nothing else.