OwenTruong / civitdl

A CLI python script to batch download models from CivitAI with CivitAI Api V1
Apache License 2.0
46 stars 6 forks source link

downloading wrong files #1

Closed Shandelier closed 10 months ago

Shandelier commented 10 months ago

Hi

thanks for sharing this repo

I've got a problem with batchfile download. I would like to download this list of models, but the script is downloading completly different ones.

Example ``` https://civitai.com/api/download/models/128641, https://civitai.com/api/download/models/131654, https://civitai.com/api/download/models/124628, https://civitai.com/api/download/models/127207, https://civitai.com/api/download/models/112445, ... ```
Outcome ``` models/ Alita/ Cynthia/ GiuseppeArcimboldo/ airiakizuki-lora-nochekaiser/ cozyblanket-nvwls-v1/ ```

I was expecting the outcome similar to this curl script:

curl script ```sh #!/bin/bash # Folders and their corresponding download links declare -A download_map download_map["checkpoints"]=" https://civitai.com/api/download/models/112445?type=Model&format=SafeTensor&size=pruned&fp=fp16 https://civitai.com/api/download/models/127556?type=Model&format=SafeTensor&size=pruned&fp=fp16 https://civitai.com/api/download/models/127207?type=Model&format=SafeTensor&size=pruned&fp=fp16 " download_map["vae"]=" https://civitai.com/api/download/models/128641?type=Model&format=PickleTensor https://civitai.com/api/download/models/131654?type=Model&format=SafeTensor https://civitai.com/api/download/models/124628?type=Model&format=PickleTensor https://civitai.com/api/download/models/127207?type=VAE&format=SafeTensor " # Loop through each folder and download the files for folder in "${!download_map[@]}"; do echo "Downloading files for $folder" mkdir -p "$folder" cd "$folder" while IFS= read -r link; do wget --content-disposition -nc "$link" done <<< "${download_map[$folder]}" cd .. done echo "Download completed!" ```
OwenTruong commented 10 months ago

Thanks for making the issue!

Currently, the batch script only supports non API routing (i.e civitai.com/model instead of civitai.com/api). For example, for the Copax Rustic V2 model, the url to download version 112445 of that model is the following url: https://civitai.com/models/103608?modelVersionId=112445 The script parses the following link and generates a list, [103608, 112445], and use that to search the API.

I will add this to the list of features to add since it would be pretty neat if it also supported API URLs for potential automations.

Another thing to add is support for other file formats besides safetensors.

Besides that, while testing your curl script and batch file, I realized that it would probably be better to give clearer error. For the Copax Rustic V2 model, when the specific model couldn't be found (112445 was the model's version ID so it got confused), it only gave me metadata error 404.

OwenTruong commented 10 months ago

Another issue I just realized is that the script doesn't download VAE for model 128641, 131654 and etc. It seems to download the LORA models with those id. This needs further investigation.

OwenTruong commented 10 months ago

Civitai API URL are now accepted, bug where the program didn't work for files other than safetensors is fixed, vae now is downloadable. Closing issue now.