Closed aa956 closed 7 months ago
Hi, thanks for posting the issue! For some reason, the script is not able to get the headers for the model when trying to download it. Since the only reason the script needs to get headers is to get the file name of the downloaded model, I can try to write a temporary workaround tomorrow until a better solution is figured out.
The same issue seems to be happening for this model when I run wget below, where it gave me some kind of authentication issue:
wget https://civitai.com/api/download/models/226749 --content-disposition
Name of file downloaded: login?returnUrl=%2Fmodels%2F191977&reason=download-auth
^^ This is how we are normally supposed to download a model according to civitai api
Looking at what the html wget saved with this request:
The creator of this asset requires you to be logged in to download it.
That's the difference with working url-s probably.
And looking at the api doc - some parameters have the following comment:
... authenticated user (this requires an API token or session cookie)
This is the missing part most probably.
If that is the case, the program will need an api key to work for restricted models.
I tried to look through the REST API documentation, but there doesn't seem to be any information on how to add that to a request for https://civitai.com/api/download/models/{modelId}
.
Please let me know if there is any way on how to add an api token to the request. For now, a temporary solution is to download only the metadata and images when a restricted model is requested, and have the user download the model themselves.
Related issue on civitai repo:
Wait actually never mind, it seems like all that needs to be done is to add a query string afterwards according to the issue 830 on civital: ?token=(key)
Todo:
Due to how the code is written currently and potentially making future features harder to write, it is not worth it to only download metadata if the program did not receive an api key... unless it is a feature someone wants
Hi, I have added the ability to add api key to development branch. Will merge into master branch later this month. Here is the link to the documentation for the api key for now.
If anyone would like to test it themselves, please go ahead 😄
going to leave bug report issues open until merged to master (since master or v1 currently also has this problem)
Thank you, tried with the models listed above and it works:
$ civitdl --api-key batchfile-auth.txt .
Key:
Now downloading "Belle Epoque Dresses 1880-1914"...
- Model ID: 191977
- Version ID: 226749
100%|████████████████████████████████████████████████████████████████████████████| 151M/151M [00:13<00:00, 11.0MiB/s]
Download completed for "Belle Epoque Dresses 1880-1914"
- Model ID: 191977
- Version ID: 226749
- Path: ./Belle Epoque Dresses 1880-1914/1907_Fortuny_v2-mid_191977-vid_226749.safetensors
---------------------------
Now downloading "Belle Epoque Dresses 1880-1914"...
- Model ID: 191977
- Version ID: 225207
TODO
~~ I believe requests with https encrypts the header? ~~ Actually never mind, both header and query strings are encrypted with https.
any other way to install the development build as I finally managed to build after failing with make install
Processing /workspace/civitdl
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
Building wheel for UNKNOWN (pyproject.toml) ... done
Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=961 sha256=070c8cf59a670ca8d1848edf0ff8aabc1f9b96c1933a45c074548b715c7616fa
Stored in directory: /tmp/pip-ephem-wheel-cache-xw_8vm6b/wheels/95/36/b0/3657d7cd215a46710424a38d4e53ecd2437b82f4dfdcde1524
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
Attempting uninstall: UNKNOWN
Found existing installation: UNKNOWN 0.0.0
Uninstalling UNKNOWN-0.0.0:
Successfully uninstalled UNKNOWN-0.0.0
Successfully installed UNKNOWN-0.0.0
after that I tried with setup.py
build & install which at first run didn't worked because of missing setuptools package and after installing that the build did not threw any error (at least not obviously) but something surely went wrong because after executing civitdl
and also civitconfig
it resulted to
root@C.8138066:/workspace/civitdl$ civitdl
Traceback (most recent call last):
File "/usr/local/bin/civitdl", line 33, in <module>
sys.exit(load_entry_point('civitdl==2.0.0.dev11', 'console_scripts', 'civitdl')())
File "/usr/local/bin/civitdl", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/local/lib/python3.10/dist-packages/civitdl-2.0.0.dev11-py3.10.egg/civitdl/__main__.py", line 6, in <module>
ModuleNotFoundError: No module named 'civitdl.batch'
root@C.8138066:/workspace/civitdl$ civitconfig
Traceback (most recent call last):
File "/usr/local/bin/civitconfig", line 33, in <module>
sys.exit(load_entry_point('civitdl==2.0.0.dev11', 'console_scripts', 'civitconfig')())
File "/usr/local/bin/civitconfig", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/local/lib/python3.10/dist-packages/civitdl-2.0.0.dev11-py3.10.egg/civitconfig/__main__.py", line 10, in <module>
ModuleNotFoundError: No module named 'civitconfig.args'
I'm just trying to get the civitai api working so I can download favorite models on my rented yupiter instance
I'm not sure what is the rented yupiter instance but it looks like you're trying to install pip packages as root. Please try to install in the virtual environment. Below are instructions for something Debian based, e.g. Debian, Ubuntu or Mint, for other distros look up correct package manager commands to install the python packages, else shall be very similar:
sudo apt install -y git python3-venv python3-pip
cd ~ && git clone https://github.com/OwenTruong/civitdl && cd ~/civitdl && git checkout development && python3 -m venv venv && source ./venv/bin/activate && pip install .
Or same step-by-step:
user@machine:~$ sudo apt install -y git python3-venv python3-pip
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
git is already the newest version (1:2.39.2-1.1).
python3-venv is already the newest version (3.11.2-1+b1).
python3-pip is already the newest version (23.0.1+dfsg-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
user@machine:~$ cd ~
user@machine:~$ git clone https://github.com/OwenTruong/civitdl
Cloning into 'civitdl'...
remote: Enumerating objects: 1282, done.
remote: Counting objects: 100% (344/344), done.
remote: Compressing objects: 100% (224/224), done.
remote: Total 1282 (delta 165), reused 231 (delta 105), pack-reused 938
Receiving objects: 100% (1282/1282), 385.27 KiB | 2.52 MiB/s, done.
Resolving deltas: 100% (658/658), done.
user@machine:~$ cd ~/civitdl
user@machine:~/civitdl$ git checkout development
branch 'development' set up to track 'origin/development'.
Switched to a new branch 'development'
user@machine:~$ cd ~/civitdl
user@machine:~/civitdl$ python3 -m venv venv
user@machine:~/civitdl$ source ./venv/bin/activate
(venv) user@machine:~/civitdl$
user@machine:~$ cd ~/civitdl
user@machine:~/civitdl$ source ./venv/bin/activate
(venv) user@machine:~/civitdl$ pip install .
Processing /home/user/civitdl
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting requests<3,>=2.31.0
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 2.5 MB/s eta 0:00:00
Collecting tqdm<5,>=4.66.1
Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 6.5 MB/s eta 0:00:00
Collecting appdirs<2,>=1.4.4
Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting argparse<2,>=1.4
Downloading argparse-1.4.0-py2.py3-none-any.whl (23 kB)
Collecting charset-normalizer<4,>=2
Downloading charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (140 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.3/140.3 kB 9.7 MB/s eta 0:00:00
Collecting idna<4,>=2.5
Downloading idna-3.6-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 6.1 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1
Downloading urllib3-2.1.0-py3-none-any.whl (104 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.6/104.6 kB 9.6 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
Downloading certifi-2023.11.17-py3-none-any.whl (162 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.5/162.5 kB 9.5 MB/s eta 0:00:00
Building wheels for collected packages: civitdl
Building wheel for civitdl (pyproject.toml) ... done
Created wheel for civitdl: filename=civitdl-2.0.0.dev11-py3-none-any.whl size=28382 sha256=e30ae9e559c1bd039006fd78f24f91ed40277f8d6a5ba913ef4028e6d97c91ee
Stored in directory: /tmp/pip-ephem-wheel-cache-86wxouh5/wheels/77/24/49/5daa729d9125c7743aa9f46fe4d1472cd9ef60004ec7f75ebd
Successfully built civitdl
Installing collected packages: argparse, appdirs, urllib3, tqdm, idna, charset-normalizer, certifi, requests, civitdl
Successfully installed appdirs-1.4.4 argparse-1.4.0 certifi-2023.11.17 charset-normalizer-3.3.2 civitdl-2.0.0.dev11 idna-3.6 requests-2.31.0 tqdm-4.66.1 urllib3-2.1.0
(venv) user@machine:~/civitdl$
$ $ ~/civitdl/venv/bin/civitdl --help
Usage: civitdl [-h] [-s SORTER] [-i INT] [-p | --with-prompt | --no-with-prompt] [-k] [-v | --dev | --no-dev]
srcmodel [srcmodel ...] rootdir
A CLI python script to batch download models from CivitAI with CivitAI Api V1.
positional arguments:
srcmodel
Provide one or more sources as arguments.
A source can be one of the following: model ID, CivitAI URL, string list of sources or batchfiles.
- batchfiles must be a textfile of comma separated list of sources.
- string list of sources in the following example is "sourceA, sourceB, sourceC, ...":
- Example 1: civitdl source1 "sourceA, sourceB, sourceC, ..." source2 ./path/to/root/model/directory
- Example 2: civitdl ./batchfile1.txt 123456 ~/Downloads/ComfyUI/models/loras
rootdir Root directory of where the downloaded model should go.
options:
-h, --help show this help message and exit
-s SORTER, --sorter SORTER
Specify which sorter function to use.
Default is "basic" sorter.
Provide file path to sorter function if you wish to use a custom sorter.
-i INT, --max-images INT
Specify max images to download for each model.
-p, --with-prompt, --no-with-prompt
Download images with prompt.
-k, --api-key Prompt user for api key to download models that require users to log in.
-v, --dev, --no-dev Prints out traceback and other useful information.
Processing /workspace/civitdl Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheels for collected packages: UNKNOWN Building wheel for UNKNOWN (pyproject.toml) ... done Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=961 sha256=070c8cf59a670ca8d1848edf0ff8aabc1f9b96c1933a45c074548b715c7616fa Stored in directory: /tmp/pip-ephem-wheel-cache-xw_8vm6b/wheels/95/36/b0/3657d7cd215a46710424a38d4e53ecd2437b82f4dfdcde1524 Successfully built UNKNOWN Installing collected packages: UNKNOWN Attempting uninstall: UNKNOWN Found existing installation: UNKNOWN 0.0.0 Uninstalling UNKNOWN-0.0.0: Successfully uninstalled UNKNOWN-0.0.0 Successfully installed UNKNOWN-0.0.0
I encountered the same problem a week ago when I was trying to install the development branch of civitdl globally in Ubuntu 22.04 with pip not installing the correct packages (resulting in UNKNOWN-0.0.0
). It seems like running make install
(or the equivalent pip install .
) only works locally, while for global usage, this worked for me:
python3 -m build
pip3 install -r ./requirements.txt
pip3 install --upgrade dist/*.whl
So if you prefer to install globally instead of locally using virtual environment (as mentioned in a956's comprehensive post), try the above.
( Edit: also, please create a new issue next time :) )
I have created an issue for this #57
Only for some models: