search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.15k stars 909 forks source link

Add missing docs for extractor configurations #5216

Open Gigas002 opened 6 months ago

Gigas002 commented 6 months ago

Some extractors configurations are missing docs. Searched the repo's gallery-dl.conf, gallery-dl-example.conf, configuration.rst and an html hosted version

Here's list of missing extractors in hosted html configuration:

I may be missing some things (like, I'm not sure all extractor properties and downloader/output/etc stuff listed too...) but used extractor's init for reference to list these

Personally, I don't mind searching for these in code, but it feels rather complicated, since extractor classes doesn't list serializable properties like this (sry if I'm saying stupid things, not an everyday python programmer, may be this is considered bad coding style):

class Smugmug(Extractor):
    videos: bool
    api_key: str
    api_secret: str

and you have to figure out the properties by reading full code for each extractor

Hrxn commented 6 months ago

What exactly do you think is missing here?

The options are always used in the same way (That's why "filename" is written as extractor.*.filename, basically). This is includes "cookies", "username" & "password" and so on..

Site specific options in configuration.rst are only mentioned when they are, well, site-specific.

You don't have to search any source code here.

Everything you need is in gallery-dl -K <URL>, basically, and gallery-dl -E <URL>.

This is already mentioned in the documentation, btw.

The available replacement keys depend on the extractor used. A list of keys for a specific one can be acquired by calling gallery-dl with the -K/--list-keywords command-line option. For example: [..]

You can also turn on private fields with output.private in the output here, not sure what you would need it for, though.

Gigas002 commented 6 months ago

Ah, so the extractors there listed only if these have specific options, fair enough. Does that mean, that extractors, that are not listed have only properties, which a listed in docs with wildcard? I suspect something like idolcomplex is using sankaku's properties, but how would I know that beforehand? Someone who sees the code, and/or uses the website would suggest that it is using the sankaku's properties, but that's not said explicitly, which, I personally think, would be more user-friendly.

Also, I was looking at example config and found out that you can specify custom instances, but haven't found anything in detail about it in docs, so I thought these extractors are missing or something (though I still don't see where this part is coming from).

What about sqaure-braced (in doc file) extractors? e.g. extractor.[Danbooru].threshold, what's the difference between just extractor.danbooru.%some_option%?

mikf commented 6 months ago

Everything you need is in gallery-dl -K <URL>, basically, and gallery-dl -E <URL>.

There's also --list-extractors, which lists all extractors and their category values.

I suspect something like idolcomplex is using sankaku's properties

It does not. idolcomplex has a quite different site layout than sankaku (at least than https://sankaku.app, which is what gallery-dl uses) and does not share any of its options.

It inherits from SankakuExtractor, but that's most likely a "relic" of the past when both sites shared some code.

Also, I was looking at example config and found out that you can specify custom instances, but haven't found anything in detail about it in docs, so I thought these extractors are missing or something (though I still don't see where this part is coming from).

1658

Yeah, that's not properly documented.

Some sites share the same basecategory and can share config options by using that as key instead of the regular site category and you can add additional instances for it.

What about sqaure-braced (in doc file) extractors? e.g. extractor.[Danbooru].threshold, what's the difference between just extractor.danbooru.%some_option%?

Same thing. Square-bracketed values are the basecategory and all extractors with it share this option.

For this specific case it'd be regular danbooru, atfbooru, aibooru, booruvar, and all Danbooru instances you added manually.

Gigas002 commented 6 months ago

idolcomplex has a quite different site layout than sankaku

My bad, meant chan.sankakucomplex.com to be more precise (it's still not abandoned btw). In any case, thanks for explaining all of this! If the inherited extractors have the parent's props then I guess I can find this out from code. The reason I ask this question in the first place, is because I'm creating a custom serializing/deserializing tool for gallery-dl's config (nothing really worth mentioning), and couldn't figure out all the options.