New Update cant find extensions? or even textual embeddings

[Maki9009]

after i update to the new notebook, it cant seem to find and load my Deforum extension Nor my Dreambooth extension. and also my Textual inversion embeddings. it all worked out fine before this update

[IndexProselyte]

I am also having a error pop up when installing dependencies:

HTTPError: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/Engineer-of-Stuff/stable-diffusion-paperspace/main/lfs/save_steps.py

[Cyberes]

@Maki9009 I changed the way the WebUI was stored and it may be using the wrong path. Do you know where you had it installed? I bet it is in /notebooks/stable-diffusion-webui but after this update the notebook installed a second copy to /storage/stable-diffusion/stable-diffusion-webui. To fix this set repo_storage_dir to the correct directory.

@IndexProselyte A lot of the custom scripts are outdated and were depreciated in favor of extensions. If you'd like to continue using scripts you use this old commit: https://github.com/Engineer-of-Stuff/stable-diffusion-paperspace/blob/b6c474928c5d0a312248609fe476086efe191dad/StableDiffusionUI_Voldemort_paperspace.ipynb

Specifically this:

import shutil
import requests
!pip install moviepy==1.0.3
!apt update
!apt install -y potrace python3-tk

def download_file_dir(url, output_dir):
    # output_dir must have a trailing slash
    local_filename = url.split('/')[-1]
    with requests.get(url, stream=True) as r:
        r.raise_for_status()
        with open(f'{output_dir}{local_filename}', 'wb') as f:
            for chunk in r.iter_content(chunk_size=8192):
                f.write(chunk)
    return local_filename
def do_script_download(scripts_list, domain, path):
    for item in scripts_list:
        download_file_dir(f'https://{domain}/{item}', path)
        print(f'{item.split("/")[-1]} downloaded...')

do_script_download([
    'GRMrGecko/stable-diffusion-webui-automatic/advanced_matrix/scripts/advanced_prompt_matrix.py',
    'dfaker/stable-diffusion-webui-cv2-external-masking-script/main/external_masking.py',
    'memes-forever/Stable-diffusion-webui-video/main/videos.py',
    'yownas/seed_travel/main/scripts/seed_travel.py',
    'Animator-Anon/Animator/main/animation.py',
    'Filarius/stable-diffusion-webui/master/scripts/vid2vid.py',
    'GeorgLegato/Txt2Vectorgraphics/main/txt2vectorgfx.py',
    'yownas/shift-attention/main/scripts/shift_attention.py',
    'DiceOwl/StableDiffusionStuff/main/loopback_superimpose.py',
    'Engineer-of-Stuff/stable-diffusion-paperspace/main/other/save_steps.py',
    'Pfaeff/sd-web-ui-scripts/main/moisaic.py'
], 'raw.githubusercontent.com', f'{repo_storage_dir}/stable-diffusion-webui/scripts/')

do_script_download([
    'dfaker/f88aa62e3a14b559fe4e5f6b345db664/raw/791dabfa0ab26399aa2635bcbc1cf6267aa4ffc2/alternate_sampler_noise_schedules.py',
    'camenduru/9ec5f8141db9902e375967e93250860f/raw/c1a03eb447548adbef1858c0e69d3567a390d2f4/run_n_times.py'
], 'gist.githubusercontent.com', f'{repo_storage_dir}/stable-diffusion-webui/scripts/')

# Download and set up txt2img2img
update_repo_if_not_exists(f'{repo_storage_dir}/stable-diffusion-webui/txt2img2img_root', 'https://github.com/ThereforeGames/txt2img2img.git')
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2img2img_root/scripts" "{repo_storage_dir}/stable-diffusion-webui"
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2img2img_root/txt2img2img" "{repo_storage_dir}/stable-diffusion-webui"
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2img2img_root/venv" "{repo_storage_dir}/stable-diffusion-webui"

# Download and set up txt2mask
update_repo_if_not_exists(f'{repo_storage_dir}/stable-diffusion-webui/txt2mask', 'https://github.com/ThereforeGames/txt2mask.git')
!echo "Copying txt2mask..."
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2mask/repositories/clipseg" "{repo_storage_dir}/stable-diffusion-webui/repositories"
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2mask/scripts/" "{repo_storage_dir}/stable-diffusion-webui/"

# Install the dynamic-prompts/wildcard script
# !git clone https://github.com/adieyal/sd-dynamic-prompting/ extensions/dynamic-prompts

and install it like this: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts#installing-and-using-custom-scripts

save_steps.py is now located here: https://github.com/Engineer-of-Stuff/stable-diffusion-paperspace/blob/master/other/save_steps.py and I've updated the code above to fix the link change.

[IndexProselyte]

@Cyberes in the end i decided to install the latest jupyter notebook to maybe fix the issue. Yet that notebook cannot be read as Paperspace doesn't consider it a JSON file. I reverted to the: 417227fdf6b12e8d1dbe20b134234a0bc658f1a1 commit version which could be read.

And for some reason it fixed the "Install requirements and download repositories" HTTP404 error i posted about.

[Cyberes]

Did you save the HTML page by accident? You need to save the raw file: https://raw.githubusercontent.com/Engineer-of-Stuff/stable-diffusion-paperspace/master/StableDiffusionUI_Voldemort_paperspace.ipynb

It's odd that fixed the 404 error since that file shouldn't exist on that URL anymore.

I tested everything on my end before pushing the update out but maybe I missed something? I'll double check.

[IndexProselyte]

Did you save the HTML page by accident? You need to save the raw file: https://raw.githubusercontent.com/Engineer-of-Stuff/stable-diffusion-paperspace/master/StableDiffusionUI_Voldemort_paperspace.ipynb

It's odd that fixed the 404 error since that file shouldn't exist on that URL anymore.

I tested everything on my end before pushing the update out but maybe I missed something? I'll double check.

I have simply downloaded the code zip file from which i uploaded the jupyter file. It wouldn't open even after i did the exact same process 3 times. It always threw a "File couldn't be open as it isn't JSON". I have not tested it on a brand new notebook so it may behave differently there.

The older version is still capable of generating images so i at least have that.

[Cyberes]

I can't reproduce the error you're describing. Does it work if you run this command on the machine to download the file?

wget https://raw.githubusercontent.com/Engineer-of-Stuff/stable-diffusion-paperspace/master/StableDiffusionUI_Voldemort_paperspace.ipynb

[Maki9009]

@Maki9009 I changed the way the WebUI was stored and it may be using the wrong path. Do you know where you had it installed? I bet it is in /notebooks/stable-diffusion-webui but after this update the notebook installed a second copy to /storage/stable-diffusion/stable-diffusion-webui. To fix this set repo_storage_dir to the correct directory.

@IndexProselyte A lot of the custom scripts are outdated and were depreciated in favor of extensions. If you'd like to continue using scripts you use this old commit: https://github.com/Engineer-of-Stuff/stable-diffusion-paperspace/blob/b6c474928c5d0a312248609fe476086efe191dad/StableDiffusionUI_Voldemort_paperspace.ipynb

Specifically this:

import shutil
import requests
!pip install moviepy==1.0.3
!apt update
!apt install -y potrace python3-tk

def download_file_dir(url, output_dir):
    # output_dir must have a trailing slash
    local_filename = url.split('/')[-1]
    with requests.get(url, stream=True) as r:
        r.raise_for_status()
        with open(f'{output_dir}{local_filename}', 'wb') as f:
            for chunk in r.iter_content(chunk_size=8192):
                f.write(chunk)
    return local_filename
def do_script_download(scripts_list, domain, path):
    for item in scripts_list:
        download_file_dir(f'https://{domain}/{item}', path)
        print(f'{item.split("/")[-1]} downloaded...')

do_script_download([
    'GRMrGecko/stable-diffusion-webui-automatic/advanced_matrix/scripts/advanced_prompt_matrix.py',
    'dfaker/stable-diffusion-webui-cv2-external-masking-script/main/external_masking.py',
    'memes-forever/Stable-diffusion-webui-video/main/videos.py',
    'yownas/seed_travel/main/scripts/seed_travel.py',
    'Animator-Anon/Animator/main/animation.py',
    'Filarius/stable-diffusion-webui/master/scripts/vid2vid.py',
    'GeorgLegato/Txt2Vectorgraphics/main/txt2vectorgfx.py',
    'yownas/shift-attention/main/scripts/shift_attention.py',
    'DiceOwl/StableDiffusionStuff/main/loopback_superimpose.py',
    'Engineer-of-Stuff/stable-diffusion-paperspace/main/other/save_steps.py',
    'Pfaeff/sd-web-ui-scripts/main/moisaic.py'
], 'raw.githubusercontent.com', f'{repo_storage_dir}/stable-diffusion-webui/scripts/')

do_script_download([
    'dfaker/f88aa62e3a14b559fe4e5f6b345db664/raw/791dabfa0ab26399aa2635bcbc1cf6267aa4ffc2/alternate_sampler_noise_schedules.py',
    'camenduru/9ec5f8141db9902e375967e93250860f/raw/c1a03eb447548adbef1858c0e69d3567a390d2f4/run_n_times.py'
], 'gist.githubusercontent.com', f'{repo_storage_dir}/stable-diffusion-webui/scripts/')

# Download and set up txt2img2img
update_repo_if_not_exists(f'{repo_storage_dir}/stable-diffusion-webui/txt2img2img_root', 'https://github.com/ThereforeGames/txt2img2img.git')
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2img2img_root/scripts" "{repo_storage_dir}/stable-diffusion-webui"
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2img2img_root/txt2img2img" "{repo_storage_dir}/stable-diffusion-webui"
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2img2img_root/venv" "{repo_storage_dir}/stable-diffusion-webui"

# Download and set up txt2mask
update_repo_if_not_exists(f'{repo_storage_dir}/stable-diffusion-webui/txt2mask', 'https://github.com/ThereforeGames/txt2mask.git')
!echo "Copying txt2mask..."
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2mask/repositories/clipseg" "{repo_storage_dir}/stable-diffusion-webui/repositories"
!cp -r "{repo_storage_dir}/stable-diffusion-webui/txt2mask/scripts/" "{repo_storage_dir}/stable-diffusion-webui/"

# Install the dynamic-prompts/wildcard script
# !git clone https://github.com/adieyal/sd-dynamic-prompting/ extensions/dynamic-prompts

and install it like this: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts#installing-and-using-custom-scripts

save_steps.py is now located here: https://github.com/Engineer-of-Stuff/stable-diffusion-paperspace/blob/master/other/save_steps.py and I've updated the code above to fix the link change.

i changed it to /notebooks/stable-diffusion-webui but its still wrong... but It still didn't work because I'm pretty sure it became /notebooks/stable-diffusion-webui/stable-diffusion-webui , what should i change it to to make it work?

[Cyberes]

Then try /notebooks/

[IndexProselyte]

I can't reproduce the error you're describing. Does it work if you run this command on the machine to download the file?

wget https://raw.githubusercontent.com/Engineer-of-Stuff/stable-diffusion-paperspace/master/StableDiffusionUI_Voldemort_paperspace.ipynb

After downloading it thru the wget command it opens and actually works. I have tried to wget the whole repo and that works also. It seems i am having a PC issue when uploading from myself? Well no matter issue has been solved.

[Cyberes]

Great, glad that worked! Jupiter can be a little janky sometimes.

[Cyberes]

@IndexProselyte I've added the scripts downloader back into the notebook in the Tools section, see abca842b30a4317462ca605ae90b709af93baa33. Please update your notebook.