Improve loading speed by 4 minutes

[pravdomil]

Hi, I have done following:

• combined all downloads into one aria2c call
• use git clone --depth 1

And now it starts by 4 minutes faster. I can send you a PR if you want. With Colab Pro, before 11m 15s, after 7m 38s.

Here is the updated script.

!echo "Running..."

%cd /content

%env TF_CPP_MIN_LOG_LEVEL=1

!apt update -qq -y
!apt install -qq -y libunwind8-dev

!wget http://launchpadlibrarian.net/367274644/libgoogle-perftools-dev_2.5-2.2ubuntu3_amd64.deb
!wget https://launchpad.net/ubuntu/+source/google-perftools/2.5-2.2ubuntu3/+build/14795286/+files/google-perftools_2.5-2.2ubuntu3_all.deb
!wget https://launchpad.net/ubuntu/+source/google-perftools/2.5-2.2ubuntu3/+build/14795286/+files/libtcmalloc-minimal4_2.5-2.2ubuntu3_amd64.deb
!wget https://launchpad.net/ubuntu/+source/google-perftools/2.5-2.2ubuntu3/+build/14795286/+files/libgoogle-perftools4_2.5-2.2ubuntu3_amd64.deb
!dpkg -i *.deb
!rm *.deb

%env LD_PRELOAD=libtcmalloc.so

!apt install -qq -y aria2 libcairo2-dev pkg-config python3-dev
!pip install -q torch==2.0.0+cu118 torchvision==0.15.1+cu118 torchaudio==2.0.1+cu118 torchtext==0.15.1 torchdata==0.6.0 xformers==0.0.19 triton==2.0.0 --extra-index-url https://download.pytorch.org/whl/cu118 -U

!git clone --depth 1 -b v2.2 https://github.com/camenduru/stable-diffusion-webui
!git clone --depth 1 https://huggingface.co/embed/negative /content/stable-diffusion-webui/embeddings/negative
!git clone --depth 1 https://huggingface.co/embed/lora /content/stable-diffusion-webui/models/Lora/positive

# Extensions
!git clone --depth 1 https://github.com/deforum-art/deforum-for-automatic1111-webui /content/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui
!git clone --depth 1 https://github.com/camenduru/stable-diffusion-webui-images-browser /content/stable-diffusion-webui/extensions/stable-diffusion-webui-images-browser
!git clone --depth 1 https://github.com/camenduru/stable-diffusion-webui-huggingface /content/stable-diffusion-webui/extensions/stable-diffusion-webui-huggingface
!git clone --depth 1 https://github.com/camenduru/sd-civitai-browser /content/stable-diffusion-webui/extensions/sd-civitai-browser
!git clone --depth 1 https://github.com/kohya-ss/sd-webui-additional-networks /content/stable-diffusion-webui/extensions/sd-webui-additional-networks
!git clone --depth 1 https://github.com/Mikubill/sd-webui-controlnet /content/stable-diffusion-webui/extensions/sd-webui-controlnet
!git clone --depth 1 https://github.com/fkunn1326/openpose-editor /content/stable-diffusion-webui/extensions/openpose-editor
!git clone --depth 1 https://github.com/jexom/sd-webui-depth-lib /content/stable-diffusion-webui/extensions/sd-webui-depth-lib
!git clone --depth 1 https://github.com/hnmr293/posex /content/stable-diffusion-webui/extensions/posex
!git clone --depth 1 https://github.com/nonnonstop/sd-webui-3d-open-pose-editor /content/stable-diffusion-webui/extensions/sd-webui-3d-open-pose-editor
!git clone --depth 1 https://github.com/camenduru/sd-webui-tunnels /content/stable-diffusion-webui/extensions/sd-webui-tunnels
!git clone --depth 1 https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webui
!git clone --depth 1 https://github.com/camenduru/stable-diffusion-webui-catppuccin /content/stable-diffusion-webui/extensions/stable-diffusion-webui-catppuccin
!git clone --depth 1 https://github.com/camenduru/a1111-sd-webui-locon /content/stable-diffusion-webui/extensions/a1111-sd-webui-locon
!git clone --depth 1 https://github.com/AUTOMATIC1111/stable-diffusion-webui-rembg /content/stable-diffusion-webui/extensions/stable-diffusion-webui-rembg
!git clone --depth 1 https://github.com/ashen-sensored/stable-diffusion-webui-two-shot /content/stable-diffusion-webui/extensions/stable-diffusion-webui-two-shot
!git clone --depth 1 https://github.com/thomasasfk/sd-webui-aspect-ratio-helper /content/stable-diffusion-webui/extensions/sd-webui-aspect-ratio-helper
!git clone --depth 1 https://github.com/tjm35/asymmetric-tiling-sd-webui /content/stable-diffusion-webui/extensions/asymmetric-tiling-sd-webui

# Reset
%cd /content/stable-diffusion-webui
!git reset --hard
!git -C /content/stable-diffusion-webui/repositories/stable-diffusion-stability-ai reset --hard

# Files
!echo -e "\
\nhttps://huggingface.co/embed/upscale/resolve/main/4x-UltraSharp.pth \n dir=/content/stable-diffusion-webui/models/ESRGAN \n out=4x-UltraSharp.pth \
\nhttps://raw.githubusercontent.com/camenduru/stable-diffusion-webui-scripts/main/run_n_times.py \n dir=/content/stable-diffusion-webui/scripts \n out=run_n_times.py \
\n\
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11e_sd15_ip2p_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11e_sd15_ip2p_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11e_sd15_shuffle_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11e_sd15_shuffle_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_canny_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_canny_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11f1p_sd15_depth_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11f1p_sd15_depth_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_inpaint_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_inpaint_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_lineart_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_lineart_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_mlsd_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_mlsd_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_normalbae_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_normalbae_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_openpose_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_openpose_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_scribble_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_scribble_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_seg_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_seg_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_softedge_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_softedge_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15s2_lineart_anime_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15s2_lineart_anime_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11f1e_sd15_tile_fp16.safetensors \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11f1e_sd15_tile_fp16.safetensors \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11e_sd15_ip2p_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11e_sd15_ip2p_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11e_sd15_shuffle_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11e_sd15_shuffle_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_canny_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_canny_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11f1p_sd15_depth_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11f1p_sd15_depth_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_inpaint_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_inpaint_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_lineart_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_lineart_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_mlsd_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_mlsd_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_normalbae_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_normalbae_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_openpose_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_openpose_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_scribble_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_scribble_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_seg_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_seg_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15_softedge_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15_softedge_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11p_sd15s2_lineart_anime_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11p_sd15s2_lineart_anime_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/control_v11f1e_sd15_tile_fp16.yaml \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=control_v11f1e_sd15_tile_fp16.yaml \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_style_sd14v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_style_sd14v1.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_sketch_sd14v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_sketch_sd14v1.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_seg_sd14v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_seg_sd14v1.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_openpose_sd14v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_openpose_sd14v1.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_keypose_sd14v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_keypose_sd14v1.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_depth_sd14v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_depth_sd14v1.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_color_sd14v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_color_sd14v1.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_canny_sd14v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_canny_sd14v1.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_canny_sd15v2.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_canny_sd15v2.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_depth_sd15v2.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_depth_sd15v2.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_sketch_sd15v2.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_sketch_sd15v2.pth \
\nhttps://huggingface.co/ckpt/ControlNet-v1-1/resolve/main/t2iadapter_zoedepth_sd15v1.pth \n dir=/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models \n out=t2iadapter_zoedepth_sd15v1.pth \
\n\
\nhttps://huggingface.co/ckpt/sd14/resolve/main/sd-v1-4.ckpt \n dir=/content/stable-diffusion-webui/models/Stable-diffusion \n out=sd-v1-4.ckpt \
\n" | aria2c --console-log-level=error --continue --max-connection-per-server 16 --split 16 --min-split-size 10M --input-file -

# Fix
!sed -i -e '''/    prepare_environment()/a\    os.system\(f\"""sed -i -e ''\"s/dict()))/dict())).cuda()/g\"'' /content/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/util.py""")''' /content/stable-diffusion-webui/launch.py
!sed -i -e 's/\"sd_model_checkpoint\"\,/\"sd_model_checkpoint\,sd_vae\,CLIP_stop_at_last_layers\"\,/g' /content/stable-diffusion-webui/modules/shared.py

# Launch
!echo "Launching..."
!python launch.py --listen --xformers --enable-insecure-extension-access --theme dark --gradio-queue --multiple

[onelittlechildawa]

Is there any reason why nobody used --depth 1?

[pravdomil]

Rapid developement

[steinhaug]

Great tip there, I have made sure I have that one in my setup. Check it out by the way, Im a Colab user trying to make this "setup" be as fast and relioable as possible using Google Drive as offloading for files.

Lesson learned is that running the WebUI from Google Drive is not possible in any reasonable way, and right now I am having some pain getting the Lora extra folders to work especially as I have a set of some 2000+ Loras to extend with. Maby Google Drive just isnt practical when using to many files. Who knows, then again. Works pretty great at the moment - Ill add some instructions next update.

Check it out here, basically latest WebUI with all extensions and offloading large files into Drive:

[camenduru]

thanks ❤