search

OwenTruong / civitdl

A CLI python script to batch download models from CivitAI with CivitAI Api V1
Apache License 2.0
46 stars 6 forks source link

bug: status 404, downloaded model from CivitAI has no content disposition header available. #27

Closed aa956 closed 7 months ago

aa956 commented 9 months ago

Only for some models:

$ cat batchfile.txt
https://civitai.com/models/191977?modelVersionId=226749,
https://civitai.com/models/191977?modelVersionId=225207,
https://civitai.com/models/191977?modelVersionId=221870,
https://civitai.com/models/191977?modelVersionId=215664,
https://civitai.com/models/191977?modelVersionId=216801,
$ git pull
Already up to date.
$ civitdl batchfile batchfile.txt .
Now downloading "Belle Epoque Dresses 1880-1914"...
            - Model ID: 191977
            - Version ID: 226749

Status 404: Downloaded model from CivitAI has no content disposition header available.
Now downloading "Belle Epoque Dresses 1880-1914"...
            - Model ID: 191977
            - Version ID: 225207

Status 404: Downloaded model from CivitAI has no content disposition header available.
Now downloading "Belle Epoque Dresses 1880-1914"...
            - Model ID: 191977
            - Version ID: 221870

Status 404: Downloaded model from CivitAI has no content disposition header available.
Now downloading "Belle Epoque Dresses 1880-1914"...
            - Model ID: 191977
            - Version ID: 215664

Status 404: Downloaded model from CivitAI has no content disposition header available.
Now downloading "Belle Epoque Dresses 1880-1914"...
            - Model ID: 191977
            - Version ID: 216801

Status 404: Downloaded model from CivitAI has no content disposition header available.
$
OwenTruong commented 9 months ago

Hi, thanks for posting the issue! For some reason, the script is not able to get the headers for the model when trying to download it. Since the only reason the script needs to get headers is to get the file name of the downloaded model, I can try to write a temporary workaround tomorrow until a better solution is figured out.

The same issue seems to be happening for this model when I run wget below, where it gave me some kind of authentication issue:

wget https://civitai.com/api/download/models/226749 --content-disposition Name of file downloaded: login?returnUrl=%2Fmodels%2F191977&reason=download-auth

^^ This is how we are normally supposed to download a model according to civitai api

aa956 commented 9 months ago

Looking at what the html wget saved with this request:

The creator of this asset requires you to be logged in to download it.

That's the difference with working url-s probably.

And looking at the api doc - some parameters have the following comment:

... authenticated user (this requires an API token or session cookie)

This is the missing part most probably.

image

OwenTruong commented 9 months ago

If that is the case, the program will need an api key to work for restricted models.

I tried to look through the REST API documentation, but there doesn't seem to be any information on how to add that to a request for https://civitai.com/api/download/models/{modelId}.

Please let me know if there is any way on how to add an api token to the request. For now, a temporary solution is to download only the metadata and images when a restricted model is requested, and have the user download the model themselves.

Related issue on civitai repo:

OwenTruong commented 9 months ago

Wait actually never mind, it seems like all that needs to be done is to add a query string afterwards according to the issue 830 on civital: ?token=(key)

Todo:

OwenTruong commented 9 months ago

Due to how the code is written currently and potentially making future features harder to write, it is not worth it to only download metadata if the program did not receive an api key... unless it is a feature someone wants

OwenTruong commented 9 months ago

Hi, I have added the ability to add api key to development branch. Will merge into master branch later this month. Here is the link to the documentation for the api key for now.

If anyone would like to test it themselves, please go ahead 😄

OwenTruong commented 9 months ago

going to leave bug report issues open until merged to master (since master or v1 currently also has this problem)

aa956 commented 9 months ago

Thank you, tried with the models listed above and it works:

$ civitdl --api-key batchfile-auth.txt .
Key:
Now downloading "Belle Epoque Dresses 1880-1914"...
            - Model ID: 191977
            - Version ID: 226749

100%|████████████████████████████████████████████████████████████████████████████| 151M/151M [00:13<00:00, 11.0MiB/s]

Download completed for "Belle Epoque Dresses 1880-1914"
            - Model ID: 191977
            - Version ID: 226749
            - Path: ./Belle Epoque Dresses 1880-1914/1907_Fortuny_v2-mid_191977-vid_226749.safetensors

---------------------------

Now downloading "Belle Epoque Dresses 1880-1914"...
            - Model ID: 191977
            - Version ID: 225207
OwenTruong commented 8 months ago

TODO

~~ I believe requests with https encrypts the header? ~~ Actually never mind, both header and query strings are encrypted with https.

dadoirie commented 8 months ago

any other way to install the development build as I finally managed to build after failing with make install

Processing /workspace/civitdl
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=961 sha256=070c8cf59a670ca8d1848edf0ff8aabc1f9b96c1933a45c074548b715c7616fa
  Stored in directory: /tmp/pip-ephem-wheel-cache-xw_8vm6b/wheels/95/36/b0/3657d7cd215a46710424a38d4e53ecd2437b82f4dfdcde1524
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
  Attempting uninstall: UNKNOWN
    Found existing installation: UNKNOWN 0.0.0
    Uninstalling UNKNOWN-0.0.0:
      Successfully uninstalled UNKNOWN-0.0.0
Successfully installed UNKNOWN-0.0.0

after that I tried with setup.py build & install which at first run didn't worked because of missing setuptools package and after installing that the build did not threw any error (at least not obviously) but something surely went wrong because after executing civitdl and also civitconfig it resulted to 

root@C.8138066:/workspace/civitdl$ civitdl
Traceback (most recent call last):
  File "/usr/local/bin/civitdl", line 33, in <module>
    sys.exit(load_entry_point('civitdl==2.0.0.dev11', 'console_scripts', 'civitdl')())
  File "/usr/local/bin/civitdl", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/civitdl-2.0.0.dev11-py3.10.egg/civitdl/__main__.py", line 6, in <module>
ModuleNotFoundError: No module named 'civitdl.batch'
root@C.8138066:/workspace/civitdl$ civitconfig
Traceback (most recent call last):
  File "/usr/local/bin/civitconfig", line 33, in <module>
    sys.exit(load_entry_point('civitdl==2.0.0.dev11', 'console_scripts', 'civitconfig')())
  File "/usr/local/bin/civitconfig", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/civitdl-2.0.0.dev11-py3.10.egg/civitconfig/__main__.py", line 10, in <module>
ModuleNotFoundError: No module named 'civitconfig.args'

I'm just trying to get the civitai api working so I can download favorite models on my rented yupiter instance

aa956 commented 8 months ago

I'm not sure what is the rented yupiter instance but it looks like you're trying to install pip packages as root. Please try to install in the virtual environment. Below are instructions for something Debian based, e.g. Debian, Ubuntu or Mint, for other distros look up correct package manager commands to install the python packages, else shall be very similar:

  1. Two one-liners, first to install git and python, second to build development version of this repository: 
    sudo apt install -y git python3-venv python3-pip
cd ~ && git clone https://github.com/OwenTruong/civitdl && cd ~/civitdl && git checkout development && python3 -m venv venv && source ./venv/bin/activate && pip install .

Or same step-by-step:

  1. Install git, python3 venv and pip 
    user@machine:~$ sudo apt install -y git python3-venv python3-pip
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
git is already the newest version (1:2.39.2-1.1).
python3-venv is already the newest version (3.11.2-1+b1).
python3-pip is already the newest version (23.0.1+dfsg-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  2. Clone this repository:
user@machine:~$ cd ~
user@machine:~$ git clone https://github.com/OwenTruong/civitdl
Cloning into 'civitdl'...
remote: Enumerating objects: 1282, done.
remote: Counting objects: 100% (344/344), done.
remote: Compressing objects: 100% (224/224), done.
remote: Total 1282 (delta 165), reused 231 (delta 105), pack-reused 938
Receiving objects: 100% (1282/1282), 385.27 KiB | 2.52 MiB/s, done.
Resolving deltas: 100% (658/658), done.
  1. Check out development branch
user@machine:~$ cd ~/civitdl
user@machine:~/civitdl$ git checkout development
branch 'development' set up to track 'origin/development'.
Switched to a new branch 'development'
  1. Create virtual environment
user@machine:~$ cd ~/civitdl
user@machine:~/civitdl$ python3 -m venv venv
user@machine:~/civitdl$ source ./venv/bin/activate
(venv) user@machine:~/civitdl$
  1. Install inside the environment
user@machine:~$ cd ~/civitdl
user@machine:~/civitdl$ source ./venv/bin/activate
(venv) user@machine:~/civitdl$ pip install .
Processing /home/user/civitdl
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting requests<3,>=2.31.0
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 2.5 MB/s eta 0:00:00
Collecting tqdm<5,>=4.66.1
  Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 6.5 MB/s eta 0:00:00
Collecting appdirs<2,>=1.4.4
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting argparse<2,>=1.4
  Downloading argparse-1.4.0-py2.py3-none-any.whl (23 kB)
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (140 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.3/140.3 kB 9.7 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.6-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 6.1 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1
  Downloading urllib3-2.1.0-py3-none-any.whl (104 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.6/104.6 kB 9.6 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2023.11.17-py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.5/162.5 kB 9.5 MB/s eta 0:00:00
Building wheels for collected packages: civitdl
  Building wheel for civitdl (pyproject.toml) ... done
  Created wheel for civitdl: filename=civitdl-2.0.0.dev11-py3-none-any.whl size=28382 sha256=e30ae9e559c1bd039006fd78f24f91ed40277f8d6a5ba913ef4028e6d97c91ee
  Stored in directory: /tmp/pip-ephem-wheel-cache-86wxouh5/wheels/77/24/49/5daa729d9125c7743aa9f46fe4d1472cd9ef60004ec7f75ebd
Successfully built civitdl
Installing collected packages: argparse, appdirs, urllib3, tqdm, idna, charset-normalizer, certifi, requests, civitdl
Successfully installed appdirs-1.4.4 argparse-1.4.0 certifi-2023.11.17 charset-normalizer-3.3.2 civitdl-2.0.0.dev11 idna-3.6 requests-2.31.0 tqdm-4.66.1 urllib3-2.1.0
(venv) user@machine:~/civitdl$
  1. Use like this
$ $ ~/civitdl/venv/bin/civitdl --help
Usage: civitdl [-h] [-s SORTER] [-i INT] [-p | --with-prompt | --no-with-prompt] [-k] [-v | --dev | --no-dev]
               srcmodel [srcmodel ...] rootdir

A CLI python script to batch download models from CivitAI with CivitAI Api V1.

positional arguments:
  srcmodel
                        Provide one or more sources as arguments.
                        A source can be one of the following: model ID, CivitAI URL, string list of sources or batchfiles.
                        - batchfiles must be a textfile of comma separated list of sources.
                        - string list of sources in the following example is "sourceA, sourceB, sourceC, ...":
                        - Example 1: civitdl source1 "sourceA, sourceB, sourceC, ..." source2 ./path/to/root/model/directory
                        - Example 2: civitdl ./batchfile1.txt 123456 ~/Downloads/ComfyUI/models/loras
  rootdir               Root directory of where the downloaded model should go.

options:
  -h, --help            show this help message and exit
  -s SORTER, --sorter SORTER
                        Specify which sorter function to use.
                        Default is "basic" sorter.
                        Provide file path to sorter function if you wish to use a custom sorter.
  -i INT, --max-images INT
                        Specify max images to download for each model.
  -p, --with-prompt, --no-with-prompt
                        Download images with prompt.
  -k, --api-key         Prompt user for api key to download models that require users to log in.
  -v, --dev, --no-dev   Prints out traceback and other useful information.
OwenTruong commented 8 months ago

Processing /workspace/civitdl Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheels for collected packages: UNKNOWN Building wheel for UNKNOWN (pyproject.toml) ... done Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=961 sha256=070c8cf59a670ca8d1848edf0ff8aabc1f9b96c1933a45c074548b715c7616fa Stored in directory: /tmp/pip-ephem-wheel-cache-xw_8vm6b/wheels/95/36/b0/3657d7cd215a46710424a38d4e53ecd2437b82f4dfdcde1524 Successfully built UNKNOWN Installing collected packages: UNKNOWN Attempting uninstall: UNKNOWN Found existing installation: UNKNOWN 0.0.0 Uninstalling UNKNOWN-0.0.0: Successfully uninstalled UNKNOWN-0.0.0 Successfully installed UNKNOWN-0.0.0

I encountered the same problem a week ago when I was trying to install the development branch of civitdl globally in Ubuntu 22.04 with pip not installing the correct packages (resulting in UNKNOWN-0.0.0). It seems like running make install (or the equivalent pip install .) only works locally, while for global usage, this worked for me:

    python3 -m build
    pip3 install -r ./requirements.txt
    pip3 install --upgrade dist/*.whl

So if you prefer to install globally instead of locally using virtual environment (as mentioned in a956's comprehensive post), try the above.

( Edit: also, please create a new issue next time :) )

OwenTruong commented 8 months ago

I have created an issue for this #57