AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
143.55k stars 27.02k forks source link

Hanging on 100% issue #15744

Open CoddedApple opened 6 months ago

CoddedApple commented 6 months ago

What happened?

So the issue of the program getting stuck at 100% has been reported multiple times. In my case it seems to happen after changing the prompt, it'll run a hundred of the same prompt just fine but if I change it even slightly it'll have a chance to delay the final step for who knows how long (it took an hour once).

I only just started to use live preview (it existed before I did) and can see that every step before the final is done quickly, so can I just change the code on my end so that it preemptively stops 1 step before the final and then save it as it would normally, skipping the 100% delay issue?

I'm not familiar with the project's code so I have no idea what would need changing to do this.

Steps to reproduce the problem

Generate Change prompt. Generate with a chance for delay.

What should have happened?

No delay at 100%. What is usually a minute per gen can take upwards of 10 minutes at times.

SysInfo

SysInfo

```json { "Platform": "Windows-10-10.0.19045-SP0", "Python": "3.10.11", "Version": "v1.7.0", "Commit": "cf2772fab0af5573da775e7437e6acdca424f26e", "Script path": "X:\\stable-diffusion-webui", "Data path": "X:\\stable-diffusion-webui", "Extensions dir": "X:\\stable-diffusion-webui\\extensions", "Checksum": "ecd5c9bebbb2be1e878bbaf7a6c61e12ec2ff24f240964e4170521e23feaa64a", "Commandline": [ "launch.py", "--xformers" ], "Torch env info": { "torch_version": "2.0.1+cu118", "is_debug_build": "False", "cuda_compiled_version": "11.8", "gcc_version": null, "clang_version": null, "cmake_version": null, "os": "Microsoft Windows 10 Home", "libc_version": "N/A", "python_version": "3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)", "python_platform": "Windows-10-10.0.19045-SP0", "is_cuda_available": "True", "cuda_runtime_version": "11.8.89\r", "cuda_module_loading": "LAZY", "nvidia_driver_version": "546.17", "nvidia_gpu_models": "GPU 0: NVIDIA GeForce RTX 2060", "cudnn_version": null, "pip_version": "pip3", "pip_packages": [ "numpy==1.23.5", "open-clip-torch==2.20.0", "pytorch-lightning==1.9.4", "torch==2.0.1+cu118", "torchdiffeq==0.2.3", "torchmetrics==1.2.0", "torchsde==0.2.6", "torchvision==0.15.2+cu118" ], "conda_packages": null, "hip_compiled_version": "N/A", "hip_runtime_version": "N/A", "miopen_runtime_version": "N/A", "caching_allocator_config": "", "is_xnnpack_available": "True", "cpu_info": [ "Architecture=9", "CurrentClockSpeed=3000", "DeviceID=CPU0", "Family=198", "L2CacheSize=2048", "L2CacheSpeed=", "Manufacturer=GenuineIntel", "MaxClockSpeed=3000", "Name=Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz", "ProcessorType=3", "Revision=" ] }, "Exceptions": [ { "exception": "A tensor with all NaNs was produced in VAE. This could be because there's not enough precision to represent the picture. Try adding --no-half-vae commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.", "traceback": [ [ "X:\\stable-diffusion-webui\\modules\\processing.py, line 600, decode_latent_batch", "devices.test_for_nans(sample, \"vae\")" ], [ "X:\\stable-diffusion-webui\\modules\\devices.py, line 150, test_for_nans", "raise NansException(message)" ] ] } ], "CPU": { "model": "Intel64 Family 6 Model 158 Stepping 13, GenuineIntel", "count logical": 8, "count physical": 8 }, "RAM": { "total": "48GB", "used": "24GB", "free": "24GB" }, "Extensions": [ { "name": "a1111-sd-webui-tagcomplete", "path": "X:\\stable-diffusion-webui\\extensions\\a1111-sd-webui-tagcomplete", "version": "3eef536b", "branch": "main", "remote": "https://github.com/DominikDoom/a1111-sd-webui-tagcomplete.git" }, { "name": "sd-dynamic-prompts", "path": "X:\\stable-diffusion-webui\\extensions\\sd-dynamic-prompts", "version": "1567e787", "branch": "main", "remote": "https://github.com/adieyal/sd-dynamic-prompts.git" }, { "name": "sd-webui-controlnet", "path": "X:\\stable-diffusion-webui\\extensions\\sd-webui-controlnet", "version": "aa3e2055", "branch": "main", "remote": "https://github.com/Mikubill/sd-webui-controlnet.git" }, { "name": "sd-webui-regional-prompter", "path": "X:\\stable-diffusion-webui\\extensions\\sd-webui-regional-prompter", "version": "50493ec0", "branch": "main", "remote": "https://github.com/hako-mikan/sd-webui-regional-prompter.git" } ], "Inactive extensions": [ ], "Environment": { "COMMANDLINE_ARGS": "--xformers", "GRADIO_ANALYTICS_ENABLED": "False" }, "Config": { "samples_save": true, "samples_format": "png", "samples_filename_pattern": "[seed]", "save_images_add_number": true, "grid_save": true, "grid_format": "jpg", "grid_extended_filename": false, "grid_only_if_multiple": true, "grid_prevent_empty_spots": false, "grid_zip_filename_pattern": "", "n_rows": -1, "font": "", "grid_text_active_color": "#000000", "grid_text_inactive_color": "#999999", "grid_background_color": "#ffffff", "enable_pnginfo": true, "save_txt": false, "save_images_before_face_restoration": false, "save_images_before_highres_fix": false, "save_images_before_color_correction": false, "save_mask": false, "save_mask_composite": false, "jpeg_quality": 80, "webp_lossless": false, "export_for_4chan": true, "img_downscale_threshold": 4.0, "target_side_length": 4000, "img_max_size_mp": 200, "use_original_name_batch": true, "use_upscaler_name_as_suffix": false, "save_selected_only": true, "save_init_img": false, "temp_dir": "", "clean_temp_dir_at_start": false, "outdir_samples": "", "outdir_txt2img_samples": "X:\\txt2img", "outdir_img2img_samples": "X:\\img2img", "outdir_extras_samples": "outputs/extras-images", "outdir_grids": "", "outdir_txt2img_grids": "X:\\txt2img-grids", "outdir_img2img_grids": "X:\\img2img-grids", "outdir_save": "log/images", "outdir_init_images": "outputs/init-images", "save_to_dirs": false, "grid_save_to_dirs": false, "use_save_to_dirs_for_ui": false, "directories_filename_pattern": "[date]", "directories_max_prompt_words": 8, "ESRGAN_tile": 192, "ESRGAN_tile_overlap": 8, "realesrgan_enabled_models": [ "R-ESRGAN 4x+", "R-ESRGAN 4x+ Anime6B" ], "upscaler_for_img2img": null, "face_restoration_model": "CodeFormer", "code_former_weight": 0.5, "face_restoration_unload": false, "show_warnings": true, "memmon_poll_rate": 8, "samples_log_stdout": false, "multiple_tqdm": true, "print_hypernet_extra": true, "list_hidden_files": true, "disable_mmap_load_safetensors": false, "unload_models_when_training": false, "pin_memory": false, "save_optimizer_state": false, "save_training_settings_to_txt": true, "dataset_filename_word_regex": "", "dataset_filename_join_string": " ", "training_image_repeats_per_epoch": 1, "training_write_csv_every": 500, "training_xattention_optimizations": false, "training_enable_tensorboard": false, "training_tensorboard_save_images": false, "training_tensorboard_flush_every": 120, "sd_model_checkpoint": "", "sd_checkpoint_cache": 0, "sd_vae_checkpoint_cache": 0, "sd_vae": "nai.vae.pt", "sd_vae_as_default": true, "sd_unet": "Automatic", "inpainting_mask_weight": 1.0, "initial_noise_multiplier": 1.0, "img2img_color_correction": false, "img2img_fix_steps": false, "img2img_background_color": "#ffffff", "enable_quantization": false, "enable_emphasis": true, "enable_batch_seeds": true, "comma_padding_backtrack": 20, "CLIP_stop_at_last_layers": 2, "upcast_attn": false, "auto_vae_precision": true, "randn_source": "GPU", "sdxl_crop_top": 0, "sdxl_crop_left": 0, "sdxl_refiner_low_aesthetic_score": 2.5, "sdxl_refiner_high_aesthetic_score": 6.0, "cross_attention_optimization": "Automatic", "s_min_uncond": 0, "token_merging_ratio": 0, "token_merging_ratio_img2img": 0, "token_merging_ratio_hr": 0, "pad_cond_uncond": false, "experimental_persistent_cond_cache": false, "use_old_emphasis_implementation": false, "use_old_karras_scheduler_sigmas": false, "no_dpmpp_sde_batch_determinism": false, "use_old_hires_fix_width_height": false, "dont_fix_second_order_samplers_schedule": false, "hires_fix_use_firstpass_conds": false, "interrogate_keep_models_in_memory": false, "interrogate_return_ranks": false, "interrogate_clip_num_beams": 1, "interrogate_clip_min_length": 24, "interrogate_clip_max_length": 48, "interrogate_clip_dict_limit": 1500, "interrogate_clip_skip_categories": [], "interrogate_deepbooru_score_threshold": 0.5, "deepbooru_sort_alpha": true, "deepbooru_use_spaces": true, "deepbooru_escape": true, "deepbooru_filter_tags": "", "extra_networks_show_hidden_directories": true, "extra_networks_hidden_models": "When searched", "extra_networks_default_multiplier": 1.0, "extra_networks_card_width": 0, "extra_networks_card_height": 0, "extra_networks_card_text_scale": 1.0, "extra_networks_card_show_desc": true, "extra_networks_add_text_separator": " ", "ui_extra_networks_tab_reorder": "", "textual_inversion_print_at_load": false, "textual_inversion_add_hashes_to_infotext": true, "sd_hypernetwork": "None", "localization": "None", "gradio_theme": "Default", "img2img_editor_height": 720, "return_grid": true, "return_mask": false, "return_mask_composite": false, "do_not_show_images": false, "send_seed": true, "send_size": true, "js_modal_lightbox": true, "js_modal_lightbox_initially_zoomed": true, "js_modal_lightbox_gamepad": false, "js_modal_lightbox_gamepad_repeat": 250, "show_progress_in_title": true, "samplers_in_dropdown": true, "dimensions_and_batch_together": true, "keyedit_precision_attention": 0.1, "keyedit_precision_extra": 0.05, "keyedit_delimiters": ".,\\/!?%^*;:{}=`~()", "keyedit_move": true, "quicksettings_list": [ "sd_model_checkpoint" ], "ui_tab_order": [], "hidden_tabs": [], "ui_reorder_list": [], "hires_fix_show_sampler": true, "hires_fix_show_prompts": true, "disable_token_counters": false, "add_model_hash_to_info": true, "add_model_name_to_info": true, "add_user_name_to_info": false, "add_version_to_infotext": true, "disable_weights_auto_swap": true, "infotext_styles": "Apply if any", "show_progressbar": true, "live_previews_enable": true, "live_previews_image_format": "jpeg", "show_progress_grid": false, "show_progress_every_n_steps": 2, "show_progress_type": "Approx NN", "live_preview_content": "Prompt", "live_preview_refresh_period": 1000, "hide_samplers": [], "eta_ddim": 0.0, "eta_ancestral": 1.0, "ddim_discretize": "uniform", "s_churn": 0.0, "s_tmin": 0.0, "s_noise": 1.0, "k_sched_type": "Automatic", "sigma_min": 0.0, "sigma_max": 0.0, "rho": 0.0, "eta_noise_seed_delta": 0, "always_discard_next_to_last_sigma": false, "uni_pc_variant": "bh1", "uni_pc_skip_type": "time_uniform", "uni_pc_order": 3, "uni_pc_lower_order_final": true, "postprocessing_enable_in_main_ui": [], "postprocessing_operation_order": [], "upscaling_max_images_in_cache": 5, "disabled_extensions": [], "disable_all_extensions": "none", "restore_config_state_file": "", "sd_checkpoint_hash": "", "ldsr_steps": 100, "ldsr_cached": false, "SCUNET_tile": 256, "SCUNET_tile_overlap": 8, "SWIN_tile": 192, "SWIN_tile_overlap": 8, "lora_functional": false, "sd_lora": "None", "lora_preferred_name": "Alias from file", "lora_add_hashes_to_infotext": false, "lora_show_all": false, "lora_hide_unknown_for_versions": [], "extra_options": [], "extra_options_accordion": false, "canvas_hotkey_zoom": "Alt", "canvas_hotkey_adjust": "Ctrl", "canvas_hotkey_move": "F", "canvas_hotkey_fullscreen": "S", "canvas_hotkey_reset": "R", "canvas_hotkey_overlap": "O", "canvas_show_tooltip": true, "canvas_blur_prompt": false, "canvas_disabled_functions": [ "Overlap" ], "sd_vae_overrides_per_model_preferences": false, "hypertile_enable_unet": false, "hypertile_enable_unet_secondpass": false, "hypertile_max_depth_unet": 3, "hypertile_max_tile_unet": 256, "hypertile_swap_size_unet": 3, "hypertile_enable_vae": false, "hypertile_max_depth_vae": 3, "hypertile_max_tile_vae": 128, "hypertile_swap_size_vae": 3, "tac_tagFile": "danbooru.csv", "tac_active": true, "tac_activeIn.txt2img": true, "tac_activeIn.img2img": true, "tac_activeIn.negativePrompts": true, "tac_activeIn.thirdParty": true, "tac_activeIn.modelList": "", "tac_activeIn.modelListMode": "Blacklist", "tac_slidingPopup": true, "tac_maxResults": 5, "tac_showAllResults": false, "tac_resultStepLength": 100, "tac_delayTime": 100, "tac_useWildcards": true, "tac_sortWildcardResults": true, "tac_useEmbeddings": true, "tac_includeEmbeddingsInNormalResults": false, "tac_useHypernetworks": true, "tac_useLoras": true, "tac_useLycos": true, "tac_showWikiLinks": false, "tac_showExtraNetworkPreviews": true, "tac_replaceUnderscores": true, "tac_escapeParentheses": true, "tac_appendComma": true, "tac_appendSpace": true, "tac_alwaysSpaceAtEnd": true, "tac_modelKeywordCompletion": "Never", "tac_modelKeywordLocation": "Start of prompt", "tac_wildcardCompletionMode": "To next folder level", "tac_alias.searchByAlias": true, "tac_alias.onlyShowAlias": false, "tac_translation.translationFile": "None", "tac_translation.oldFormat": false, "tac_translation.searchByTranslation": true, "tac_translation.liveTranslation": false, "tac_extra.extraFile": "extra-quality-tags.csv", "tac_extra.addMode": "Insert before", "tac_chantFile": "demo-chants.json", "tac_keymap": "{\n \"MoveUp\": \"ArrowUp\",\n \"MoveDown\": \"ArrowDown\",\n \"JumpUp\": \"PageUp\",\n \"JumpDown\": \"PageDown\",\n \"JumpToStart\": \"Home\",\n \"JumpToEnd\": \"End\",\n \"ChooseSelected\": \"Enter\",\n \"ChooseFirstOrSelected\": \"Tab\",\n \"Close\": \"Escape\"\n}", "tac_colormap": "{\n \"danbooru\": {\n \"-1\": [\"red\", \"maroon\"],\n \"0\": [\"lightblue\", \"dodgerblue\"],\n \"1\": [\"indianred\", \"firebrick\"],\n \"3\": [\"violet\", \"darkorchid\"],\n \"4\": [\"lightgreen\", \"darkgreen\"],\n \"5\": [\"orange\", \"darkorange\"]\n },\n \"e621\": {\n \"-1\": [\"red\", \"maroon\"],\n \"0\": [\"lightblue\", \"dodgerblue\"],\n \"1\": [\"gold\", \"goldenrod\"],\n \"3\": [\"violet\", \"darkorchid\"],\n \"4\": [\"lightgreen\", \"darkgreen\"],\n \"5\": [\"tomato\", \"darksalmon\"],\n \"6\": [\"red\", \"maroon\"],\n \"7\": [\"whitesmoke\", \"black\"],\n \"8\": [\"seagreen\", \"darkseagreen\"]\n }\n}", "tac_refreshTempFiles": "Refresh TAC temp files", "save_images_replace_action": "Replace", "save_incomplete_images": false, "notification_audio": true, "notification_volume": 100, "use_old_scheduling": false, "extra_networks_dir_button_function": false, "extra_networks_card_order_field": "Path", "extra_networks_card_order": "Ascending", "lora_in_memory_limit": 0, "persistent_cond_cache": true, "batch_cond_uncond": true, "s_tmax": 0, "sgm_noise_multiplier": false, "sd_checkpoints_limit": 1, "sd_checkpoints_keep_in_cpu": true, "tiling": false, "hires_fix_refiner_pass": "second pass", "sd_vae_encode_method": "Full", "sd_vae_decode_method": "Full", "img2img_extra_noise": 0, "img2img_sketch_default_brush_color": "#ffffff", "img2img_inpaint_mask_brush_color": "#ffffff", "img2img_inpaint_sketch_default_brush_color": "#ffffff", "img2img_batch_show_results_limit": 32, "gallery_height": "", "add_vae_name_to_info": true, "add_vae_hash_to_info": true, "infotext_skip_pasting": [], "live_preview_allow_lowvram_full": false, "live_preview_fast_interrupt": false, "js_live_preview_in_modal_lightbox": false, "keyedit_delimiters_whitespace": [ "Tab", "Carriage Return", "Line Feed" ], "extra_options_txt2img": [], "extra_options_img2img": [], "extra_options_cols": 1, "compact_prompt_box": false, "sd_checkpoint_dropdown_use_short": false, "txt2img_settings_accordion": false, "img2img_settings_accordion": false, "gradio_themes_cache": true, "api_enable_requests": true, "api_forbid_local_requests": true, "api_useragent": "", "auto_launch_browser": "Disable", "enable_console_prompts": false, "show_gradio_deprecation_warnings": true, "hide_ldm_prints": true, "dump_stacks_on_signal": false, "face_restoration": false, "postprocessing_existing_caption_action": "Ignore", "canvas_auto_expand": true, "tac_wildcardExclusionList": "", "tac_skipWildcardRefresh": false, "tac_useLoraPrefixForLycos": true, "tac_modelSortOrder": "Name", "tac_useStyleVars": false, "tac_frequencySort": true, "tac_frequencyFunction": "Logarithmic (weak)", "tac_frequencyMinCount": 3, "tac_frequencyMaxAge": 30, "tac_frequencyRecommendCap": 10, "tac_frequencyIncludeAlias": false, "dp_ignore_whitespace": false, "dp_write_raw_template": false, "dp_write_prompts_to_file": false, "dp_parser_variant_start": "{", "dp_parser_variant_end": "}", "dp_parser_wildcard_wrap": "__", "dp_limit_jinja_prompts": false, "dp_auto_purge_cache": false, "dp_wildcard_manager_no_dedupe": false, "dp_wildcard_manager_no_sort": false, "dp_wildcard_manager_shuffle": false, "dp_magicprompt_default_model": "Gustavosta/MagicPrompt-Stable-Diffusion", "dp_magicprompt_batch_size": 1, "control_net_detectedmap_dir": "detected_maps", "control_net_models_path": "", "control_net_modules_path": "", "control_net_unit_count": 3, "control_net_model_cache_size": 2, "control_net_inpaint_blur_sigma": 7, "control_net_no_detectmap": false, "control_net_detectmap_autosaving": false, "control_net_allow_script_control": false, "control_net_sync_field_args": true, "controlnet_show_batch_images_in_ui": false, "controlnet_increment_seed_during_batch": false, "controlnet_disable_openpose_edit": false, "controlnet_disable_photopea_edit": false, "controlnet_photopea_warning": true, "controlnet_ignore_noninpaint_mask": false, "controlnet_clip_detector_on_cpu": false, "controlnet_control_type_dropdown": false, "regprp_debug": false, "regprp_hidepmask": false }, "Startup": { "total": 20.960586547851562, "records": { "initial startup": 0.0553586483001709, "prepare environment/checks": 0.010970830917358398, "prepare environment/git version info": 0.06526303291320801, "prepare environment/torch GPU test": 2.168682098388672, "prepare environment/clone repositores": 0.23125052452087402, "prepare environment/run extensions installers/a1111-sd-webui-tagcomplete": 0.0, "prepare environment/run extensions installers/sd-dynamic-prompts": 0.3091740608215332, "prepare environment/run extensions installers/sd-webui-controlnet": 0.2828691005706787, "prepare environment/run extensions installers/sd-webui-regional-prompter": 0.0, "prepare environment/run extensions installers": 0.5920431613922119, "prepare environment": 3.1100974082946777, "launcher": 0.0019943714141845703, "import torch": 7.1912078857421875, "import gradio": 1.2739925384521484, "setup paths": 1.5650112628936768, "import ldm": 0.006981372833251953, "import sgm": 0.0, "initialize shared": 0.32887840270996094, "other imports": 0.919750452041626, "opts onchange": 0.0, "setup SD model": 0.0029926300048828125, "setup codeformer": 0.17453384399414062, "setup gfpgan": 0.02892160415649414, "set samplers": 0.0, "list extensions": 0.004987001419067383, "restore config state file": 0.0, "list SD models": 0.7826824188232422, "list localizations": 0.0009980201721191406, "load scripts/custom_code.py": 0.008975982666015625, "load scripts/img2imgalt.py": 0.000997304916381836, "load scripts/loopback.py": 0.0, "load scripts/outpainting_mk_2.py": 0.0, "load scripts/poor_mans_outpainting.py": 0.000997304916381836, "load scripts/postprocessing_caption.py": 0.0, "load scripts/postprocessing_codeformer.py": 0.0, "load scripts/postprocessing_create_flipped_copies.py": 0.0, "load scripts/postprocessing_focal_crop.py": 0.0019953250885009766, "load scripts/postprocessing_gfpgan.py": 0.0009970664978027344, "load scripts/postprocessing_split_oversized.py": 0.0, "load scripts/postprocessing_upscale.py": 0.0, "load scripts/processing_autosized_crop.py": 0.0009975433349609375, "load scripts/prompt_matrix.py": 0.0, "load scripts/prompts_from_file.py": 0.0, "load scripts/sd_upscale.py": 0.0009970664978027344, "load scripts/xyz_grid.py": 0.0029909610748291016, "load scripts/ldsr_model.py": 2.0528385639190674, "load scripts/lora_script.py": 0.6170594692230225, "load scripts/scunet_model.py": 0.04787158966064453, "load scripts/swinir_model.py": 0.04606056213378906, "load scripts/hotkey_config.py": 0.0009968280792236328, "load scripts/extra_options_section.py": 0.0, "load scripts/hypertile_script.py": 0.0937502384185791, "load scripts/hypertile_xyz.py": 0.0, "load scripts/model_keyword_support.py": 0.004293203353881836, "load scripts/shared_paths.py": 0.0009911060333251953, "load scripts/tag_autocomplete_helper.py": 0.45704221725463867, "load scripts/tag_frequency_db.py": 0.0010280609130859375, "load scripts/dynamic_prompting.py": 0.19117522239685059, "load scripts/adapter.py": 0.0009984970092773438, "load scripts/api.py": 0.27554821968078613, "load scripts/batch_hijack.py": 0.0009677410125732422, "load scripts/cldm.py": 0.0, "load scripts/controlnet.py": 0.2533755302429199, "load scripts/controlnet_diffusers.py": 0.0009980201721191406, "load scripts/controlnet_lllite.py": 0.0, "load scripts/controlnet_lora.py": 0.0009961128234863281, "load scripts/controlnet_model_guess.py": 0.0, "load scripts/controlnet_sparsectrl.py": 0.0, "load scripts/controlnet_version.py": 0.0009982585906982422, "load scripts/enums.py": 0.0010223388671875, "load scripts/external_code.py": 0.0, "load scripts/global_state.py": 0.0010027885437011719, "load scripts/hook.py": 0.0, "load scripts/infotext.py": 0.0, "load scripts/logging.py": 0.0009908676147460938, "load scripts/lvminthin.py": 0.0, "load scripts/movie2movie.py": 0.0010008811950683594, "load scripts/supported_preprocessor.py": 0.0010008811950683594, "load scripts/utils.py": 0.0, "load scripts/xyz_grid_support.py": 0.0009942054748535156, "load scripts/attention.py": 0.0, "load scripts/latent.py": 0.004989147186279297, "load scripts/regions.py": 0.0009663105010986328, "load scripts/rp.py": 0.050436973571777344, "load scripts/rps.py": 0.010969877243041992, "load scripts/refiner.py": 0.0, "load scripts/seed.py": 0.0, "load scripts": 4.139312267303467, "load upscalers": 0.0029921531677246094, "refresh VAE": 0.006013631820678711, "refresh textual inversion templates": 0.0, "scripts list_optimizers": 0.0009970664978027344, "scripts list_unets": 0.0, "reload hypernetworks": 0.0019659996032714844, "initialize extra networks": 0.029920339584350586, "scripts before_ui_callback": 0.0, "create ui": 1.082993507385254, "gradio launch": 0.27393460273742676, "add APIs": 0.005983829498291016, "app_started_callback/lora_script.py": 0.0009968280792236328, "app_started_callback/tag_autocomplete_helper.py": 0.005984067916870117, "app_started_callback/api.py": 0.0029921531677246094, "app_started_callback": 0.00997304916381836 } }, "Packages": [ "-rotobuf==4.25.3", "absl-py==2.0.0", "accelerate==0.21.0", "addict==2.4.0", "aenum==3.1.15", "aiofiles==23.2.1", "aiohttp==3.9.0", "aiosignal==1.3.1", "albumentations==1.4.3", "altair==5.1.2", "antlr4-python3-runtime==4.9.3", "anyio==3.7.1", "astunparse==1.6.3", "async-timeout==4.0.3", "attrs==23.1.0", "basicsr==1.4.2", "beautifulsoup4==4.12.2", "blendmodes==2022", "blis==0.7.11", "boltons==23.1.1", "cachetools==5.3.2", "catalogue==2.0.10", "certifi==2023.11.17", "cffi==1.16.0", "chardet==5.2.0", "charset-normalizer==3.3.2", "clean-fid==0.1.35", "click==8.1.7", "clip==1.0", "cloudpathlib==0.16.0", "colorama==0.4.6", "coloredlogs==15.0.1", "colorlog==6.8.2", "confection==0.1.3", "contourpy==1.2.0", "cssselect2==0.7.0", "cycler==0.12.1", "cymem==2.0.8", "cython==3.0.9", "deepdanbooru==1.0.2", "deprecation==2.1.0", "depth-anything==2024.1.22.0", "dsine==2024.3.23", "dynamicprompts==0.31.0", "easydict==1.13", "einops==0.4.1", "embreex==2.17.7.post4", "exceptiongroup==1.2.0", "facexlib==0.3.0", "fake-useragent==1.4.0", "fastapi==0.94.0", "ffmpy==0.3.1", "filelock==3.13.1", "filterpy==1.4.5", "flatbuffers==24.3.7", "fonttools==4.45.0", "frozenlist==1.4.0", "fsspec==2023.10.0", "ftfy==6.1.1", "future==0.18.3", "fvcore==0.1.5.post20221221", "gast==0.5.4", "gdown==4.7.1", "geffnet==1.0.2", "gfpgan==1.3.8", "gitdb==4.0.11", "gitpython==3.1.32", "glob2==0.5", "google-auth-oauthlib==1.1.0", "google-auth==2.23.4", "google-pasta==0.2.0", "gradio-client==0.5.0", "gradio==3.41.2", "grpcio==1.59.3", "h11==0.12.0", "h5py==3.10.0", "handrefinerportable==2024.2.12.0", "httpcore==0.15.0", "httpx==0.24.1", "huggingface-hub==0.19.4", "humanfriendly==10.0", "idna==3.4", "imageio==2.33.0", "importlib-metadata==6.8.0", "importlib-resources==6.1.1", "inflection==0.5.1", "insightface==0.7.3", "iopath==0.1.9", "jax==0.4.25", "jinja2==3.1.2", "joblib==1.3.2", "jsonmerge==1.8.0", "jsonschema-specifications==2023.11.1", "jsonschema==4.20.0", "keras==3.1.1", "kiwisolver==1.4.5", "kornia==0.6.7", "langcodes==3.3.0", "lark==1.1.2", "lazy-loader==0.3", "libclang==18.1.1", "lightning-utilities==0.10.0", "llvmlite==0.41.1", "lmdb==1.4.1", "lpips==0.1.4", "lxml==5.1.0", "mapbox-earcut==1.0.1", "markdown-it-py==3.0.0", "markdown==3.5.1", "markupsafe==2.1.3", "matplotlib==3.8.2", "mdurl==0.1.2", "mediapipe==0.10.11", "ml-dtypes==0.3.2", "mpmath==1.3.0", "multidict==6.0.4", "murmurhash==1.0.10", "namex==0.0.7", "networkx==3.2.1", "numba==0.58.1", "numpy==1.23.5", "oauthlib==3.2.2", "omegaconf==2.2.3", "onnx==1.16.0", "onnxruntime-gpu==1.17.1", "open-clip-torch==2.20.0", "opencv-contrib-python==4.9.0.80", "opencv-python-headless==4.9.0.80", "opencv-python==4.8.1.78", "opt-einsum==3.3.0", "optree==0.10.0", "orjson==3.9.10", "packaging==23.2", "pandas==2.1.3", "piexif==1.1.3", "pillow==9.5.0", "pip==23.0.1", "platformdirs==4.0.0", "portalocker==2.8.2", "preshed==3.0.9", "prettytable==3.10.0", "protobuf==3.20.3", "psutil==5.9.5", "pyasn1-modules==0.3.0", "pyasn1==0.5.1", "pycollada==0.8", "pycparser==2.21", "pydantic==1.10.13", "pydub==0.25.1", "pygments==2.17.2", "pyparsing==3.1.1", "pyreadline3==3.4.1", "pysocks==1.7.1", "python-dateutil==2.8.2", "python-multipart==0.0.6", "pytorch-lightning==1.9.4", "pytz==2023.3.post1", "pywavelets==1.5.0", "pywin32==306", "pyyaml==6.0.1", "realesrgan==0.3.0", "referencing==0.31.0", "regex==2023.10.3", "reportlab==4.1.0", "requests-oauthlib==1.3.1", "requests==2.31.0", "resize-right==0.0.2", "rich==13.7.1", "rpds-py==0.13.1", "rsa==4.9", "rtree==1.2.0", "safetensors==0.3.1", "scikit-image==0.21.0", "scikit-learn==1.4.1.post1", "scipy==1.11.4", "semantic-version==2.10.0", "send2trash==1.8.0", "sentencepiece==0.1.99", "setuptools==65.5.0", "shapely==2.0.3", "six==1.16.0", "smart-open==6.4.0", "smmap==5.0.1", "sniffio==1.3.0", "sounddevice==0.4.6", "soupsieve==2.5", "spacy-legacy==3.0.12", "spacy-loggers==1.0.5", "spacy==3.7.2", "srsly==2.4.8", "starlette==0.26.1", "svg.path==6.3", "svglib==1.5.1", "sympy==1.12", "tabulate==0.9.0", "tb-nightly==2.16.0a20231121", "tensorboard-data-server==0.7.2", "tensorboard==2.16.2", "tensorflow-intel==2.16.1", "tensorflow-io-gcs-filesystem==0.31.0", "tensorflow==2.16.1", "termcolor==2.4.0", "tf-keras-nightly==2.16.0.dev2023112110", "thinc==8.2.1", "threadpoolctl==3.4.0", "tifffile==2023.9.26", "timm==0.9.2", "tinycss2==1.2.1", "tokenizers==0.13.3", "tomesd==0.1.3", "tomli==2.0.1", "toolz==0.12.0", "torch==2.0.1+cu118", "torchdiffeq==0.2.3", "torchmetrics==1.2.0", "torchsde==0.2.6", "torchvision==0.15.2+cu118", "tqdm==4.66.1", "trampoline==0.1.2", "transformers==4.30.2", "trimesh==4.2.2", "typer==0.9.0", "typing-extensions==4.10.0", "tzdata==2023.3", "urllib3==2.1.0", "uvicorn==0.24.0.post1", "vhacdx==0.0.6", "wasabi==1.1.2", "wcwidth==0.2.11", "weasel==0.3.4", "webencodings==0.5.1", "websockets==11.0.3", "werkzeug==3.0.1", "wheel==0.43.0", "wrapt==1.16.0", "xatlas==0.0.9", "xformers==0.0.20", "xxhash==3.4.1", "yacs==0.1.8", "yapf==0.40.2", "yarl==1.9.3", "zipp==3.17.0", "zipunicode==1.1.1" ] } ```

w-e-w commented 6 months ago

I don't know what's causing issue but you got to realize 100% is not 100%

the progress while we showing is only the sampling steps but there's also other things that happen after such as the VAE or potentially other post processing so the true progress is 100%+vae+other post processing

so if you "preemptively stops 1 step before the final" all that dose is make 100%+vae+other post processing in to 95%+vae+other post processing

if it gets stuck afterwards like in VAE, it's still stuck

I'm guessing the issue is during VAE your system doesn't have enough vram and start using system ram which causes massive slow down see https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/11063#issue-1744192919

live preview works differently from normal image, (it essentially decodes the latent space using a different method by default using a cheaper method so it's quicker)

2024-05-09 15_57_18_786 chrome

if the issue is with VRAM usage during VAE pahse there's a couple of things you can try you can try using TAESD as the final VAE, using TAESD will result in image quality drop but should be much faster and use less VRAM 2024-05-09 15_58_10_921 chrome or you can try https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111 this can reduce the vram usage during VAE pahse


if the issue is it getting stuck during other post processing then more investigation needs to be done


you should really fill in the issue report with the proper information we requested like the Sysinfo, if you uploaded that then I can see your settings and what system you're on what graphics card you're using which may give us clue on what is causing the issue

CoddedApple commented 6 months ago

@w-e-w Updated Sysinfo Just checked a few generations, vram is around 4.6~/6.0 during hires then spikes to 5.7~/6.0 on final processing for a 576x640 image upscaled by 2x.

w-e-w commented 6 months ago

as expected RTX 2060 with 6GB vram if it spikes up to 6 then then it's likely that it's actually going into System RAM which can cause slow down

so I think it's highly likely it's caused by not enough VRAM if you want to test i if this is the case you can you fellow https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/11063#issue-1744192919 and set CUDA - Sysmem Fallback Policy.to Prefer No Sysmem Fallback this ment the driver will not try to use system RAM but just ERROR when not enouth VRAM (image gen will fail)

it is possible in your situation --medvram or --lowvram may help enableing FP8 weight in setting may also help basically all the settings that reduces vram use may help

CoddedApple commented 6 months ago

@w-e-w I don't know if it's actually a vram issue. For a normal 50-90 second gen the vram will spike to 5.8~ on final and normal ram will bump up a bit. I just got a generation that took 8+ minutes but it also only spiked to 5.8 and bumped up normal ram a little, so there's no difference there besides the time it takes to finish. Between the 8 minute gen and the gen before it which took 54 seconds, the only difference was the seed and a slight change made to the prompt, the size and everything else was kept the same.

I followed the instructions and turned CUDA - Sysmem Fallback Policy but it didn't do anything different when generating. I even tried making a couple images twice as large as I normally do, capped at vram 5.8 and uses a bunch of ram but no delay and no failure.

theonlyblbl commented 4 months ago

it is possible in your situation --medvram or --lowvram may help enableing FP8 weight in setting may also help basically all the settings that reduces vram use may help

Didn't have the same issue as the author of this post but enabling FP8 weight helped me when I had this error : RuntimeError: CUDA error: out of memory (I have 4GB VRAM while using SDXL models)

theonlyblbl commented 4 months ago

it is possible in your situation --medvram or --lowvram may help enableing FP8 weight in setting may also help basically all the settings that reduces vram use may help

Didn't have the same issue as the author of this post but enabling FP8 weight helped me when I had this error : RuntimeError: CUDA error: out of memory (I have 4GB VRAM while using SDXL models)

I still have this issue but it doesn't show up as often as before