KuoriaNeko / civitai-scraper

A CivitAI scraper
GNU General Public License v3.0
36 stars 11 forks source link

How to download only nsfw models #1

Closed XimikaT closed 1 year ago

XimikaT commented 1 year ago

Hello! How to download only nsfw models with type Poses or LORA and only last version of each model in safetensor format

KuoriaNeko commented 1 year ago

@XimikaT From official docs, have you tried --param tag=nsfw ?

XimikaT commented 1 year ago

Not all models have tags nsfw. I modify your script in line 321:

        i = 0
        for item in model_list["items"]:
            if model_list["items"][i]["nsfw"] == 1:
                cai_model = CivitAIModel(dl_dir, item)
                await cai_model.new()
                await cai_model.run(verify)
            i += 1

And launch civitai.py --download --verify --dir "download" --param "sort=Newest" --param "limit=5" --param "types=LORA" --param "token=my civitai token"

The question remains with how to get only the latest version of the model with the required extension.. Mayby add primaryFileOnly = True

KuoriaNeko commented 1 year ago

The question remains with how to get only the latest version of the model with the required extension.. Mayby add primaryFileOnly = True

I just tested primaryFileOnly=True, it does not filtered out the latest model version. I think we can sort model version by modelVersions.createdAt and download the latest version.

XimikaT commented 1 year ago

I think only the first element from modelVersions is needed.

        i = 0
        for item in model_list["items"]:
            for b in reversed(range(1,len(model_list["items"][i]["modelVersions"]))):
                model_list["items"][i]["modelVersions"].pop(b)

            if model_list["items"][i]["nsfw"] == 1:
                cai_model = CivitAIModel(dl_dir, item)
                await cai_model.new()
                await cai_model.run(verify)
            i += 1

it works. Maybe a simpler solution? I'm not good at python

KuoriaNeko commented 1 year ago

I think only the first element from modelVersions is needed.

As far as the metadata I have checked, it's should work. But for prevent something unpredictable, I prefer to sort it first. Also I planned some features in next version, such as "download meta data only", will make the scraper more flexible.

KuoriaNeko commented 1 year ago

@XimikaT I already introduced arguments to download latest model only and nsfw model only, you can check the commit fd1c071c01cfc46a2435cc69ab89ed0b8c4e61a4 for more details.