AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
143.16k stars 26.98k forks source link

[Bug]: new download models not store in the new given path in docker #9886

Closed xwi88 closed 1 year ago

xwi88 commented 1 year ago

Is there an existing issue for this?

What happened?

When run the service with docker, set the volume mapping to inner --data-dir and host volume, and download the model, but it download into the old dir models, shall in {--data-dir}/models. But the extensions, embeddings into the right dirs, like: {--data-dir}/{extensions,embeddings}.

Steps to reproduce the problem

  1. Config the docker volumes mapping: /root/stable-diffusion-data:/stable-diffusion-webui/sd-data
  2. Run with params --data-dir sd-data
  3. Download the models by https://github.com/butaixianran/Stable-Diffusion-Webui-Civitai-Helper
  4. Console print download into /stable-diffusion-webui/models

docker-compose.yml

version: '3'
services:
  cuda-ubuntu:
    build: .
    container_name: sd-cuda12.1-ubuntu20
    image: v8fg/stable-diffusion-webui:sd-cuda12.1-ubuntu20
    stdin_open: true
    tty: true
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all # all or integer
              # device_ids: ['0', '3']
              capabilities: [gpu]
    environment:
      NVIDIA_DRIVER_CAPABILITIES: compute,utility
      NVIDIA_VISIBLE_DEVICES: all
    ports:
      - 7860:7860
    volumes:
      - /root/stable-diffusion-data:/stable-diffusion-webui/sd-data
    command: 'bash webui.sh --data-dir sd-data --allow-code --administrator --cors-allow-origins * --precision full --no-half --listen --xformers --api --enable-insecure-extension-access' # --nowebui
    # user: 1000:1000
    restart: always

What should have happened?

  1. If set the --data-dir, the models shall into new models, like: {--data-dir}/models
  2. Also can use new dir to store the files by sys generated, like: cache.json, cache.json.lock, config.json, params.txt, ui-config.json, etc, this will be well for running with docker or docker-compose.

thks.

Commit where the problem happens

a9fed7c3

What platforms do you use to access the UI ?

Linux

What browsers do you use to access the UI ?

Microsoft Edge

Command Line Arguments

`bash webui.sh --data-dir sd-data --allow-code --administrator --cors-allow-origins * --precision full --no-half --listen --xformers --api --enable-insecure-extension-access`

List of extensions

Stable-Diffusion-Webui-Civitai-Helper https://github.com/butaixianran/Stable-Diffusion-Webui-Civitai-Helper fcf5a27a (Fri Apr 14 05:39:01 2023) unknown
gif2gif https://github.com/LonicaMewinsky/gif2gif.git cc164474 (Tue Apr 25 14:30:08 2023) unknown
model-keyword https://github.com/mix1009/model-keyword.git d7a3fe80 (Wed Apr 26 12:00:36 2023) unknown
prompt_translator https://github.com/ParisNeo/prompt_translator.git 6207ea80 (Wed Apr 26 09:45:32 2023) unknown
sd_civitai_extension https://github.com/civitai/sd_civitai_extension.git d0e83e7b (Wed Apr 26 03:29:49 2023) unknown
stable-diffusion-webui-embedding-editor https://github.com/CodeExplode/stable-diffusion-webui-embedding-editor.git c8f29023 (Sun Dec 4 04:55:54 2022) unknown
stable-diffusion-webui-localization-zh_Hans https://github.com/hanamizuki-ai/stable-diffusion-webui-localization-zh_Hans.git 850a7752 (Mon Apr 24 13:00:02 2023) unknown
stable-diffusion-webui-randomize https://github.com/innightwolfsleep/stable-diffusion-webui-randomize.git c04c7611 (Tue Jan 31 14:54:33 2023) unknown
stable-diffusion-webui-tokenizer https://github.com/AUTOMATIC1111/stable-diffusion-webui-tokenizer.git ac6d541c (Sat Dec 10 12:58:31 2022) unknown

Console logs

[root@gpu-test3 cuda12.1.0-ubuntu20]# docker-compose logs -f
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  | Install script for stable-diffusion + Web UI
sd-cuda12.1.0-ubuntu20  | Tested on Debian 11 (Bullseye)
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  | Running on diffusion user
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  | Repo already cloned, using it as install directory
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  | Create and activate python venv
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  | Launching launch.py...
sd-cuda12.1.0-ubuntu20  | ################################################################
sd-cuda12.1.0-ubuntu20  | Python 3.10.6 (main, Mar 17 2023, 21:50:35) [GCC 9.4.0]
sd-cuda12.1.0-ubuntu20  | Commit hash: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2
sd-cuda12.1.0-ubuntu20  | Installing xformers
sd-cuda12.1.0-ubuntu20  | Installing requirements for Web UI
sd-cuda12.1.0-ubuntu20  | Launching Web UI with arguments: --data-dir sd-data --allow-code --administrator --cors-allow-origins * --precision full --no-half --listen --xformers --api --enable-insecure-extension-access
sd-cuda12.1.0-ubuntu20  | Downloading: "https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors" to /stable-diffusion-webui/sd-data/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors
sd-cuda12.1.0-ubuntu20  |
100% 3.97G/3.97G [05:46<00:00, 12.3MB/s]
sd-cuda12.1.0-ubuntu20  | Calculating sha256 for /stable-diffusion-webui/sd-data/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors: 6ce0161689b3853acaa03779ec93eafe75a02f4ced659bee03f50797806fa2fa
sd-cuda12.1.0-ubuntu20  | Loading weights [6ce0161689] from /stable-diffusion-webui/sd-data/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors
sd-cuda12.1.0-ubuntu20  | Creating model from config: /stable-diffusion-webui/configs/v1-inference.yaml
sd-cuda12.1.0-ubuntu20  | LatentDiffusion: Running in eps-prediction mode
sd-cuda12.1.0-ubuntu20  | DiffusionWrapper has 859.52 M params.
Downloading (…)olve/main/vocab.json: 100% 961k/961k [00:00<00:00, 1.04MB/s]
Downloading (…)olve/main/merges.txt: 100% 525k/525k [00:00<00:00, 34.7MB/s]
Downloading (…)cial_tokens_map.json: 100% 389/389 [00:00<00:00, 375kB/s]
Downloading (…)okenizer_config.json: 100% 905/905 [00:00<00:00, 863kB/s]
Downloading (…)lve/main/config.json: 100% 4.52k/4.52k [00:00<00:00, 5.00MB/s]
sd-cuda12.1.0-ubuntu20  | Applying xformers cross attention optimization.
sd-cuda12.1.0-ubuntu20  | Textual inversion embeddings loaded(0):
sd-cuda12.1.0-ubuntu20  | Model loaded in 22.9s (calculate hash: 13.2s, create model: 4.3s, apply weights to model: 1.3s, load VAE: 3.3s, move model to device: 0.9s).
sd-cuda12.1.0-ubuntu20  | Running on local URL:  http://0.0.0.0:7860
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | To create a public link, set `share=True` in `launch()`.
sd-cuda12.1.0-ubuntu20  | Startup time: 379.0s (import gradio: 3.1s, import ldm: 1.4s, other imports: 3.2s, list SD models: 347.0s, setup codeformer: 0.1s, load scripts: 0.5s, load SD checkpoint: 23.1s, create ui: 0.2s, scripts app_started_callback: 0.2s).
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | #######################################################################################################
sd-cuda12.1.0-ubuntu20  | Initializing Civitai Link
sd-cuda12.1.0-ubuntu20  | If submitting an issue on github, please provide the below text for debugging purposes:
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | Python revision: 3.10.6 (main, Mar 17 2023, 21:50:35) [GCC 9.4.0]
sd-cuda12.1.0-ubuntu20  | Civitai Link revision: d0e83e7b048f9a6472b4964fa530f8da754aba58
sd-cuda12.1.0-ubuntu20  | SD-WebUI revision:
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | Checking Civitai Link requirements...
sd-cuda12.1.0-ubuntu20  | [+] python-socketio[client] version 5.7.2 installed.
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | #######################################################################################################
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | Closing server running on port: 7860
sd-cuda12.1.0-ubuntu20  | Restarting UI...
sd-cuda12.1.0-ubuntu20  | Civitai: API loaded
sd-cuda12.1.0-ubuntu20  | Running on local URL:  http://0.0.0.0:7860
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | To create a public link, set `share=True` in `launch()`.
sd-cuda12.1.0-ubuntu20  | Civitai: Check resources for missing preview images
sd-cuda12.1.0-ubuntu20  | Startup time: 0.9s (load scripts: 0.3s, create ui: 0.2s, gradio launch: 0.3s).
sd-cuda12.1.0-ubuntu20  | Civitai: Found 1 resources missing preview images
sd-cuda12.1.0-ubuntu20  | Civitai: Failed to fetch preview images from Civitai
sd-cuda12.1.0-ubuntu20  | Civitai Link: Connected to Civitai Link Server
sd-cuda12.1.0-ubuntu20  | Civitai Link: Joined room
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 165, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 164, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 161, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 162, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 162, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 163, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 164, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 164, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 164, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 165, Clients: 0
sd-cuda12.1.0-ubuntu20  | Closing server running on port: 7860
sd-cuda12.1.0-ubuntu20  | Restarting UI...
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Get Custom Model Folder
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Load setting from: sd-data/extensions/Stable-Diffusion-Webui-Civitai-Helper/setting.json
sd-cuda12.1.0-ubuntu20  | Civitai Helper: No setting file, use default
sd-cuda12.1.0-ubuntu20  | Civitai: API loaded
sd-cuda12.1.0-ubuntu20  | Running on local URL:  http://0.0.0.0:7860
sd-cuda12.1.0-ubuntu20  |
sd-cuda12.1.0-ubuntu20  | To create a public link, set `share=True` in `launch()`.
sd-cuda12.1.0-ubuntu20  | Civitai: Check resources for missing preview images
sd-cuda12.1.0-ubuntu20  | Startup time: 0.7s (load scripts: 0.3s, create ui: 0.2s).
sd-cuda12.1.0-ubuntu20  | Civitai: Found 1 resources missing preview images
sd-cuda12.1.0-ubuntu20  | Civitai: Failed to fetch preview images from Civitai
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Start scan_model
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Scanning path: sd-data/embeddings
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Scanning path: sd-data/models/hypernetworks
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Scanning path: /stable-diffusion-webui/models/Stable-diffusion
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Scanning path: sd-data/models/Lora
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Done. Scanned 0 models, checked 0 images
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Start scan_model
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Scanning path: sd-data/embeddings
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Scanning path: sd-data/models/hypernetworks
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Scanning path: /stable-diffusion-webui/models/Stable-diffusion
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Scanning path: sd-data/models/Lora
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Done. Scanned 0 models, checked 0 images
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 165, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Run get_model_id_from_url
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Run get_model_path_by_type_and_name
sd-cuda12.1.0-ubuntu20  | Civitai Helper: failed to get model file path
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Run get_model_id_from_url
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Run get_model_path_by_type_and_name
sd-cuda12.1.0-ubuntu20  | Civitai Helper: failed to get model file path
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 162, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 163, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Getting model info by: https://civitai.com/models/1274/dreamlike-diffusion-10
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Run get_model_id_from_url
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Request model info from civitai
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Get subfolder for: /stable-diffusion-webui/models/Stable-diffusion
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Get following info for downloading:
sd-cuda12.1.0-ubuntu20  | Civitai Helper: model_name:Dreamlike Diffusion 1.0
sd-cuda12.1.0-ubuntu20  | Civitai Helper: model_type:ckp
sd-cuda12.1.0-ubuntu20  | Civitai Helper: subfolders:['/']
sd-cuda12.1.0-ubuntu20  | Civitai Helper: version_strs:['1.0_1356']
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Searching local model by version id
sd-cuda12.1.0-ubuntu20  | Civitai Helper: folder: /stable-diffusion-webui/models/Stable-diffusion/
sd-cuda12.1.0-ubuntu20  | Civitai Helper: version_id: 1356
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Start downloading from: https://civitai.com/api/download/models/1356
sd-cuda12.1.0-ubuntu20  | Civitai Link: Presence update: SD: 164, Clients: 0
sd-cuda12.1.0-ubuntu20  | Civitai Helper: File size: 2132856686
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Target file path: /stable-diffusion-webui/models/Stable-diffusion/dreamlikeDiffusion10_10.ckpt
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Downloading to temp file: /stable-diffusion-webui/models/Stable-diffusion/dreamlikeDiffusion10_10.downloading
sd-cuda12.1.0-ubuntu20  | Civitai Helper: Downloaded size: 0
[-                                                 ] 3%Civitai Link: Presence update: SD: 164, Clients: 0
[------                                            ] 12%

Additional information

No response

akx commented 1 year ago

Please either report this wherever you report issues for the v8fg/stable-diffusion-webui Docker image you're using, or in the CivitAI Helper repo.

xwi88 commented 1 year ago

Please either report this wherever you report issues for the v8fg/stable-diffusion-webui Docker image you're using, or in the CivitAI Helper repo.

Dockerfile.yml

FROM nvidia/cuda:12.1.0-base-ubuntu20.04 AS builder

WORKDIR /

ENV TZ=Asia/Shanghai \
    DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y wget git bzip2 ca-certificates curl python3 python3-venv numactl libjemalloc-dev make automake gcc g++ subversion build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev libsqlite3-dev libbz2-dev liblzma-dev ffmpeg libsm6 libxext6 && \
    git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git && cd stable-diffusion-webui && echo "export COMMANDLINE_ARGS=\"--skip-torch-cuda-test\"" >> webui-user.sh && \
    wget https://static.elias.ink/python/3/Python-3.10.6.tgz && tar -zvxf Python-3.10.6.tgz && cd Python-3.10.6/ && ./configure --enable-optimizations && make && make install && ln -s /usr/local/bin/python3.10 /usr/bin/python && \
    useradd -m -s /bin/bash diffusion && echo "diffusion:yupoo" | chpasswd && \
    chown -R diffusion /stable-diffusion-webui/

USER diffusion

WORKDIR /stable-diffusion-webui

RUN /bin/bash webui.sh --exit

CMD ["/bin/bash"]

For my use case, I mapped the docker-compose.yml directory a bit more.

version: '3'
services:
  cuda-ubuntu:
    build: .
    container_name: sd-cuda12.1-ubuntu20
    image: v8fg/stable-diffusion-webui:sd-cuda12.1-ubuntu20
    stdin_open: true
    tty: true
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all # all or integer
              # device_ids: ['0', '3']
              capabilities: [gpu]
    environment:
      TZ: Asia/Shanghai
      NVIDIA_DRIVER_CAPABILITIES: compute,utility
      NVIDIA_VISIBLE_DEVICES: all
    ports:
      - 7860:7860
    volumes:
      - /root/stable-diffusion-data:/stable-diffusion-webui/sd-data # include dirs: embeddings, extensions, odels, cache.json, config.json, ui-config.json, etc.
      - /root/stable-diffusion-data/outputs:/stable-diffusion-webui/outputs
      - /root/stable-diffusion-data/repositories:/stable-diffusion-webui/repositories
      - /root/stable-diffusion-data/models:/stable-diffusion-webui/models # new download models will still be models? issue?
    # command: ["bash", "webui.sh", "--allow-code", "--listen", "--xformers", "--api", "--enable-insecure-extension-access"]
    # command: 'bash webui.sh --allow-code --listen --xformers --api --enable-insecure-extension-access' # --nowebui
    # command: 'bash webui.sh --allow-code --administrator --cors-allow-origins * --enable-console-prompts --listen --xformers --api --enable-insecure-extension-access' # --nowebui
    command: 'bash webui.sh --data-dir sd-data --allow-code --administrator --cors-allow-origins * --precision full --no-half --listen --xformers --api --enable-insecure-extension-access' # --nowebui
    # user: 1000:1000
    restart: always