AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
140.6k stars 26.61k forks source link

[Bug]: Unwanted artifacts introduced at last diffusion step on some schedulers #13928

Open flux627 opened 10 months ago

flux627 commented 10 months ago

Is there an existing issue for this?

What happened?

I was showing/explaining to some friends how the diffusion process works recently, which led me to configure webui to always show full quality preview images at a very fast interval (essentially showing me all steps as they are processed). I ended up kind of enjoying this (even if generations take longer) and left it on.

I started noticing that for some scheduler types, the result would converge with mostly minor adjustments for the last 30% of the steps, however when the final image was revealed, it looked terrible suddenly. Immediately, I thought this must have been a VAE-switch issue (where the previews were using a different VAE than the final conversion) but I verified this not the case- regardless of VAE being used, this happens.

After going through all schedulers, I have isolated the issue to the following schedulers (roughly ordered from most severe to least). All images are generated with this prompt + configuration:

cgi artstyle, astral projection, the soul leaving the body, artist's interpretation
Steps: 20, Sampler: [SAMPLER], CFG scale: 6, Seed: 2942166222, Size: 768x768, Model hash: 6ce0161689, Model: v1-5-pruned-emaonly, RNG: CPU, Version: v1.6.0-2-g4afaaf8a
DPM++ 2M SDE Heun: Penultimate Final
DPM++-2M-SDE-Heun-penultimate DPM++-2M-SDE-Heun-final
DPM++ 2M SDE: Penultimate Final
DPM++-2M-SDE-penultimate DPM++-2M-SDE-final
DPM++ 2M: Penultimate Final
DPM++-2M-penultimate DPM++-2M-final
DPM++ 3M SDE (arguably an improvement, but still a large jump in progression between them): Penultimate Final
DPM++-3M-SDE-penultimate DPM++-3M-SDE-final

Another note is that I thought maybe the option named "Always discard next-to-last sigma" was related to this issue, and while it does produce images without this problem, the images are entirely different.

Steps to reproduce the problem

I'm running on an M1 Macbook Pro. See image generation configuration above.

What should have happened?

It seems there might be an extra process that may or not be intentional that is changing the image in an unfavorable way at the very end with these schedulers. If it is intentional, there should be an option to either toggle its effects, or change its intensity.

Sysinfo

{
    "Platform": "macOS-13.3.1-arm64-arm-64bit",
    "Python": "3.10.11",
    "Version": "v1.6.0-2-g4afaaf8a",
    "Commit": "4afaaf8a020c1df457bcf7250cb1c7f609699fa7",
    "Script path": "/Users/julien/projects/stable-diffusion-webui",
    "Data path": "/Users/julien/projects/stable-diffusion-webui",
    "Extensions dir": "/Users/julien/projects/stable-diffusion-webui/extensions",
    "Checksum": "2393428f9a2834a52d5405bab6e1da1325ba8a7b500788325591eb2214743238",
    "Commandline": [
        "launch.py",
        "--api",
        "--skip-torch-cuda-test",
        "--upcast-sampling",
        "--no-half-vae",
        "--use-cpu",
        "interrogate"
    ],
    "Torch env info": {
        "torch_version": "2.0.1",
        "is_debug_build": "False",
        "cuda_compiled_version": null,
        "gcc_version": null,
        "clang_version": "14.0.3 (clang-1403.0.22.14.1)",
        "cmake_version": "version 3.26.4",
        "os": "macOS 13.3.1 (arm64)",
        "libc_version": "N/A",
        "python_version": "3.10.11 (main, Apr  7 2023, 07:24:53) [Clang 14.0.0 (clang-1400.0.29.202)] (64-bit runtime)",
        "python_platform": "macOS-13.3.1-arm64-arm-64bit",
        "is_cuda_available": "False",
        "cuda_runtime_version": null,
        "cuda_module_loading": "N/A",
        "nvidia_driver_version": null,
        "nvidia_gpu_models": null,
        "cudnn_version": null,
        "pip_version": "pip3",
        "pip_packages": [
            "numpy==1.23.5",
            "open-clip-torch==2.20.0",
            "pytorch-lightning==1.9.4",
            "torch==2.0.1",
            "torchdiffeq==0.2.3",
            "torchmetrics==0.11.4",
            "torchsde==0.2.5",
            "torchvision==0.15.2"
        ],
        "conda_packages": null,
        "hip_compiled_version": "N/A",
        "hip_runtime_version": "N/A",
        "miopen_runtime_version": "N/A",
        "caching_allocator_config": "",
        "is_xnnpack_available": "True",
        "cpu_info": "Apple M1 Max"
    },
    "Exceptions": [],
    "CPU": {
        "model": "arm",
        "count logical": 10,
        "count physical": 10
    },
    "RAM": {
        "total": "64GB",
        "used": "21GB",
        "free": "31MB",
        "active": "11GB",
        "inactive": "11GB"
    },
    "Extensions": [
        {
            "name": "a1111-sd-webui-lycoris",
            "path": "/Users/julien/projects/stable-diffusion-webui/extensions/a1111-sd-webui-lycoris",
            "version": "025dea96",
            "branch": "main",
            "remote": "https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris.git"
        },
        {
            "name": "adetailer",
            "path": "/Users/julien/projects/stable-diffusion-webui/extensions/adetailer",
            "version": "0f3b6b0e",
            "branch": "main",
            "remote": "https://github.com/Bing-su/adetailer.git"
        },
        {
            "name": "multidiffusion-upscaler-for-automatic1111",
            "path": "/Users/julien/projects/stable-diffusion-webui/extensions/multidiffusion-upscaler-for-automatic1111",
            "version": "de488810",
            "branch": "main",
            "remote": "https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111.git"
        },
        {
            "name": "sd-dynamic-prompts",
            "path": "/Users/julien/projects/stable-diffusion-webui/extensions/sd-dynamic-prompts",
            "version": "e53332eb",
            "branch": "main",
            "remote": "https://github.com/adieyal/sd-dynamic-prompts.git"
        },
        {
            "name": "sd-webui-controlnet",
            "path": "/Users/julien/projects/stable-diffusion-webui/extensions/sd-webui-controlnet",
            "version": "0b7f4310",
            "branch": "main",
            "remote": "https://github.com/Mikubill/sd-webui-controlnet.git"
        },
        {
            "name": "sd-webui-model-converter",
            "path": "/Users/julien/projects/stable-diffusion-webui/extensions/sd-webui-model-converter",
            "version": "b1f10148",
            "branch": "main",
            "remote": "https://github.com/Akegarasu/sd-webui-model-converter.git"
        },
        {
            "name": "stable-diffusion-webui-rembg",
            "path": "/Users/julien/projects/stable-diffusion-webui/extensions/stable-diffusion-webui-rembg",
            "version": "3d9eedbb",
            "branch": "master",
            "remote": "https://github.com/AUTOMATIC1111/stable-diffusion-webui-rembg.git"
        },
        {
            "name": "ultimate-upscale-for-automatic1111",
            "path": "/Users/julien/projects/stable-diffusion-webui/extensions/ultimate-upscale-for-automatic1111",
            "version": "c99f382b",
            "branch": "master",
            "remote": "https://github.com/Coyote-A/ultimate-upscale-for-automatic1111.git"
        }
    ],
    "Inactive extensions": [],
    "Environment": {
        "COMMANDLINE_ARGS": "--skip-torch-cuda-test --upcast-sampling --no-half-vae --use-cpu interrogate",
        "GIT": "git",
        "GRADIO_ANALYTICS_ENABLED": "False",
        "TORCH_COMMAND": "pip install torch==2.0.1 torchvision==0.15.2"
    },
    "Config": {
        "samples_save": true,
        "samples_format": "png",
        "samples_filename_pattern": "",
        "save_images_add_number": true,
        "grid_save": true,
        "grid_format": "png",
        "grid_extended_filename": false,
        "grid_only_if_multiple": true,
        "grid_prevent_empty_spots": false,
        "n_rows": -1,
        "enable_pnginfo": true,
        "save_txt": false,
        "save_images_before_face_restoration": false,
        "save_images_before_highres_fix": false,
        "save_images_before_color_correction": false,
        "save_mask": false,
        "save_mask_composite": false,
        "jpeg_quality": 80,
        "webp_lossless": false,
        "export_for_4chan": true,
        "img_downscale_threshold": 4.0,
        "target_side_length": 4000,
        "img_max_size_mp": 200,
        "use_original_name_batch": true,
        "use_upscaler_name_as_suffix": false,
        "save_selected_only": true,
        "save_init_img": false,
        "temp_dir": "",
        "clean_temp_dir_at_start": false,
        "outdir_samples": "",
        "outdir_txt2img_samples": "outputs/txt2img-images",
        "outdir_img2img_samples": "outputs/img2img-images",
        "outdir_extras_samples": "outputs/extras-images",
        "outdir_grids": "",
        "outdir_txt2img_grids": "outputs/txt2img-grids",
        "outdir_img2img_grids": "outputs/img2img-grids",
        "outdir_save": "log/images",
        "outdir_init_images": "outputs/init-images",
        "save_to_dirs": true,
        "grid_save_to_dirs": true,
        "use_save_to_dirs_for_ui": false,
        "directories_filename_pattern": "[date]",
        "directories_max_prompt_words": 8,
        "ESRGAN_tile": 192,
        "ESRGAN_tile_overlap": 8,
        "realesrgan_enabled_models": [
            "R-ESRGAN 4x+",
            "R-ESRGAN 4x+ Anime6B"
        ],
        "upscaler_for_img2img": "4x_fatal_Anime_500000_G",
        "face_restoration_model": "CodeFormer",
        "code_former_weight": 0.5,
        "face_restoration_unload": false,
        "show_warnings": false,
        "memmon_poll_rate": 8,
        "samples_log_stdout": false,
        "multiple_tqdm": true,
        "print_hypernet_extra": false,
        "list_hidden_files": true,
        "unload_models_when_training": false,
        "pin_memory": false,
        "save_optimizer_state": false,
        "save_training_settings_to_txt": true,
        "dataset_filename_word_regex": "",
        "dataset_filename_join_string": " ",
        "training_image_repeats_per_epoch": 1,
        "training_write_csv_every": 500,
        "training_xattention_optimizations": false,
        "training_enable_tensorboard": false,
        "training_tensorboard_save_images": false,
        "training_tensorboard_flush_every": 120,
        "sd_model_checkpoint": "v1-5-pruned-emaonly.safetensors [6ce0161689]",
        "sd_checkpoint_cache": 2,
        "sd_vae_checkpoint_cache": 0,
        "sd_vae": "Automatic",
        "sd_vae_as_default": true,
        "inpainting_mask_weight": 1.0,
        "initial_noise_multiplier": 1,
        "img2img_color_correction": false,
        "img2img_fix_steps": false,
        "img2img_background_color": "#ffffff",
        "enable_quantization": false,
        "enable_emphasis": true,
        "enable_batch_seeds": true,
        "comma_padding_backtrack": 20,
        "CLIP_stop_at_last_layers": 1,
        "upcast_attn": false,
        "randn_source": "CPU",
        "cross_attention_optimization": "Automatic",
        "s_min_uncond": 0,
        "token_merging_ratio": 0.0,
        "token_merging_ratio_img2img": 0.0,
        "token_merging_ratio_hr": 0.0,
        "use_old_emphasis_implementation": false,
        "use_old_karras_scheduler_sigmas": false,
        "no_dpmpp_sde_batch_determinism": false,
        "use_old_hires_fix_width_height": false,
        "dont_fix_second_order_samplers_schedule": false,
        "interrogate_keep_models_in_memory": true,
        "interrogate_return_ranks": true,
        "interrogate_clip_num_beams": 1,
        "interrogate_clip_min_length": 24,
        "interrogate_clip_max_length": 48,
        "interrogate_clip_dict_limit": 1500,
        "interrogate_clip_skip_categories": [],
        "interrogate_deepbooru_score_threshold": 0.5,
        "deepbooru_sort_alpha": false,
        "deepbooru_use_spaces": true,
        "deepbooru_escape": true,
        "deepbooru_filter_tags": "",
        "extra_networks_show_hidden_directories": true,
        "extra_networks_hidden_models": "When searched",
        "extra_networks_default_view": "cards",
        "extra_networks_default_multiplier": 1.0,
        "extra_networks_card_width": 0,
        "extra_networks_card_height": 0,
        "extra_networks_add_text_separator": " ",
        "ui_extra_networks_tab_reorder": "",
        "sd_hypernetwork": "None",
        "localization": "None",
        "gradio_theme": "Default",
        "img2img_editor_height": 720,
        "return_grid": true,
        "return_mask": false,
        "return_mask_composite": false,
        "do_not_show_images": false,
        "send_seed": true,
        "send_size": false,
        "font": "",
        "js_modal_lightbox": true,
        "js_modal_lightbox_initially_zoomed": true,
        "js_modal_lightbox_gamepad": false,
        "js_modal_lightbox_gamepad_repeat": 250,
        "show_progress_in_title": true,
        "samplers_in_dropdown": true,
        "dimensions_and_batch_together": true,
        "keyedit_precision_attention": 0.1,
        "keyedit_precision_extra": 0.05,
        "keyedit_delimiters": ".,\\/!?%^*;:{}=`~()",
        "quicksettings_list": [
            "sd_model_checkpoint",
            "CLIP_stop_at_last_layers",
            "sd_vae",
            "upscaler_for_img2img",
            "img2img_extra_noise",
            "always_discard_next_to_last_sigma"
        ],
        "ui_tab_order": [],
        "hidden_tabs": [],
        "ui_reorder": "inpaint, sampler, checkboxes, hires_fix, dimensions, cfg, seed, batch, override_settings, scripts",
        "hires_fix_show_sampler": true,
        "hires_fix_show_prompts": true,
        "add_model_hash_to_info": true,
        "add_model_name_to_info": true,
        "add_version_to_infotext": true,
        "disable_weights_auto_swap": true,
        "show_progressbar": true,
        "live_previews_enable": true,
        "live_previews_image_format": "png",
        "show_progress_grid": true,
        "show_progress_every_n_steps": 1,
        "show_progress_type": "Full",
        "live_preview_content": "Combined",
        "live_preview_refresh_period": 250.0,
        "hide_samplers": [],
        "eta_ddim": 0.0,
        "eta_ancestral": 1.0,
        "ddim_discretize": "uniform",
        "s_churn": 0.0,
        "s_tmin": 0.0,
        "s_noise": 1.0,
        "eta_noise_seed_delta": 0,
        "always_discard_next_to_last_sigma": false,
        "uni_pc_variant": "bh1",
        "uni_pc_skip_type": "time_uniform",
        "uni_pc_order": 3,
        "uni_pc_lower_order_final": true,
        "postprocessing_enable_in_main_ui": [],
        "postprocessing_operation_order": [],
        "upscaling_max_images_in_cache": 5,
        "disabled_extensions": [],
        "disable_all_extensions": "none",
        "restore_config_state_file": "",
        "sd_checkpoint_hash": "6ce0161689b3853acaa03779ec93eafe75a02f4ced659bee03f50797806fa2fa",
        "ldsr_steps": 100,
        "ldsr_cached": false,
        "SCUNET_tile": 256,
        "SCUNET_tile_overlap": 8,
        "SWIN_tile": 192,
        "SWIN_tile_overlap": 8,
        "lora_functional": false,
        "sd_lora": "None",
        "lora_preferred_name": "Alias from file",
        "lora_add_hashes_to_infotext": true,
        "control_net_model_config": "models/cldm_v15.yaml",
        "control_net_model_adapter_config": "models/t2iadapter_sketch_sd14v1.yaml",
        "control_net_detectedmap_dir": "detected_maps",
        "control_net_models_path": "",
        "control_net_modules_path": "",
        "control_net_max_models_num": 3,
        "control_net_model_cache_size": 3,
        "control_net_no_detectmap": false,
        "control_net_detectmap_autosaving": false,
        "control_net_allow_script_control": false,
        "control_net_sync_field_args": false,
        "controlnet_show_batch_images_in_ui": false,
        "controlnet_increment_seed_during_batch": false,
        "controlnet_disable_control_type": false,
        "controlnet_disable_openpose_edit": false,
        "sd_lyco": "None",
        "ad_max_models": 2,
        "ad_save_previews": false,
        "ad_save_images_before": false,
        "ad_only_seleted_scripts": true,
        "ad_script_names": "dynamic_prompting,dynamic_thresholding,wildcard_recursive,wildcards",
        "ad_bbox_sortby": "None",
        "ui_reorder_list": [
            "inpaint",
            "sampler",
            "checkboxes",
            "hires_fix",
            "dimensions",
            "cfg",
            "seed",
            "batch",
            "override_settings",
            "scripts"
        ],
        "grid_zip_filename_pattern": "",
        "sd_unet": "Automatic",
        "pad_cond_uncond": false,
        "experimental_persistent_cond_cache": false,
        "hires_fix_use_firstpass_conds": false,
        "disable_token_counters": false,
        "extra_options": [],
        "extra_options_accordion": false,
        "infotext_styles": "Apply if any",
        "k_sched_type": "Automatic",
        "sigma_min": 0.0,
        "sigma_max": 0.0,
        "rho": 0.0,
        "canvas_hotkey_zoom": "Alt",
        "canvas_hotkey_adjust": "Ctrl",
        "canvas_hotkey_move": "F",
        "canvas_hotkey_fullscreen": "S",
        "canvas_hotkey_reset": "R",
        "canvas_hotkey_overlap": "O",
        "canvas_show_tooltip": true,
        "canvas_disabled_functions": [
            "Overlap"
        ],
        "dp_ignore_whitespace": false,
        "dp_write_raw_template": false,
        "dp_write_prompts_to_file": false,
        "dp_parser_variant_start": "{",
        "dp_parser_variant_end": "}",
        "dp_parser_wildcard_wrap": "__",
        "dp_limit_jinja_prompts": false,
        "dp_auto_purge_cache": false,
        "dp_wildcard_manager_no_dedupe": false,
        "dp_wildcard_manager_no_sort": false,
        "dp_wildcard_manager_shuffle": false,
        "dp_magicprompt_default_model": "Gustavosta/MagicPrompt-Stable-Diffusion",
        "dp_magicprompt_batch_size": 1,
        "sd_vae_overrides_per_model_preferences": false,
        "grid_text_active_color": "#000000",
        "grid_text_inactive_color": "#999999",
        "grid_background_color": "#ffffff",
        "save_incomplete_images": false,
        "SWIN_torch_compile": false,
        "face_restoration": false,
        "auto_launch_browser": "Local",
        "show_gradio_deprecation_warnings": true,
        "disable_mmap_load_safetensors": false,
        "hide_ldm_prints": true,
        "api_enable_requests": true,
        "api_forbid_local_requests": true,
        "api_useragent": "",
        "sd_checkpoints_limit": 1,
        "sd_checkpoints_keep_in_cpu": true,
        "tiling": false,
        "hires_fix_refiner_pass": "second pass",
        "sdxl_crop_top": 0.0,
        "sdxl_crop_left": 0.0,
        "sdxl_refiner_low_aesthetic_score": 2.5,
        "sdxl_refiner_high_aesthetic_score": 6.0,
        "auto_vae_precision": true,
        "sd_vae_encode_method": "Full",
        "sd_vae_decode_method": "Full",
        "img2img_extra_noise": 0,
        "img2img_sketch_default_brush_color": "#ffffff",
        "img2img_inpaint_mask_brush_color": "#ffffff",
        "img2img_inpaint_sketch_default_brush_color": "#ffffff",
        "persistent_cond_cache": true,
        "batch_cond_uncond": true,
        "use_old_scheduling": false,
        "extra_networks_card_text_scale": 1,
        "extra_networks_card_show_desc": true,
        "textual_inversion_print_at_load": false,
        "textual_inversion_add_hashes_to_infotext": true,
        "lora_show_all": false,
        "lora_hide_unknown_for_versions": [],
        "lora_in_memory_limit": 0,
        "gradio_themes_cache": true,
        "gallery_height": "",
        "keyedit_move": true,
        "extra_options_txt2img": [],
        "extra_options_img2img": [],
        "extra_options_cols": 1,
        "add_user_name_to_info": false,
        "live_preview_allow_lowvram_full": false,
        "live_preview_fast_interrupt": false,
        "s_tmax": 0,
        "sgm_noise_multiplier": false,
        "canvas_auto_expand": true,
        "canvas_blur_prompt": false
    },
    "Startup": {
        "total": 8.304733037948608,
        "records": {
            "initial startup": 0.0009179115295410156,
            "prepare environment/checks": 4.601478576660156e-05,
            "prepare environment/git version info": 0.028856992721557617,
            "prepare environment/torch GPU test": 0.0016930103302001953,
            "prepare environment/clone repositores": 0.04094505310058594,
            "prepare environment/run extensions installers/ultimate-upscale-for-automatic1111": 0.0013949871063232422,
            "prepare environment/run extensions installers/adetailer": 0.0750889778137207,
            "prepare environment/run extensions installers/sd-webui-controlnet": 0.1654829978942871,
            "prepare environment/run extensions installers/stable-diffusion-webui-rembg": 0.05023598670959473,
            "prepare environment/run extensions installers/sd-webui-model-converter": 0.0008530616760253906,
            "prepare environment/run extensions installers/sd-dynamic-prompts": 0.09299492835998535,
            "prepare environment/run extensions installers/multidiffusion-upscaler-for-automatic1111": 0.0006568431854248047,
            "prepare environment/run extensions installers/a1111-sd-webui-lycoris": 0.0009169578552246094,
            "prepare environment/run extensions installers": 0.38765525817871094,
            "prepare environment": 0.5166823863983154,
            "launcher": 0.0017957687377929688,
            "import torch": 2.4986770153045654,
            "import gradio": 1.104233980178833,
            "setup paths": 0.49476099014282227,
            "import ldm": 0.002538919448852539,
            "import sgm": 3.0994415283203125e-06,
            "initialize shared": 0.12614083290100098,
            "other imports": 0.7064361572265625,
            "opts onchange": 0.00026798248291015625,
            "setup SD model": 0.002928018569946289,
            "setup codeformer": 0.07014107704162598,
            "setup gfpgan": 0.00947880744934082,
            "set samplers": 2.8133392333984375e-05,
            "list extensions": 0.00020003318786621094,
            "restore config state file": 3.814697265625e-06,
            "list SD models": 0.049662113189697266,
            "list localizations": 0.00023794174194335938,
            "load scripts/custom_code.py": 0.0031380653381347656,
            "load scripts/img2imgalt.py": 0.0004439353942871094,
            "load scripts/loopback.py": 0.0002789497375488281,
            "load scripts/outpainting_mk_2.py": 0.0003180503845214844,
            "load scripts/poor_mans_outpainting.py": 0.000125885009765625,
            "load scripts/postprocessing_codeformer.py": 0.00010895729064941406,
            "load scripts/postprocessing_gfpgan.py": 9.608268737792969e-05,
            "load scripts/postprocessing_upscale.py": 0.00012993812561035156,
            "load scripts/prompt_matrix.py": 0.00010800361633300781,
            "load scripts/prompts_from_file.py": 0.0002262592315673828,
            "load scripts/refiner.py": 0.00020575523376464844,
            "load scripts/sd_upscale.py": 0.0002181529998779297,
            "load scripts/seed.py": 0.0001857280731201172,
            "load scripts/xyz_grid.py": 0.001148223876953125,
            "load scripts/lycoris_script.py": 0.26354312896728516,
            "load scripts/!adetailer.py": 0.4468038082122803,
            "load scripts/tilediffusion.py": 0.003242969512939453,
            "load scripts/vae_optimize.py": 0.0006380081176757812,
            "load scripts/dynamic_prompting.py": 0.021508216857910156,
            "load scripts/adapter.py": 0.0006718635559082031,
            "load scripts/api.py": 0.7024557590484619,
            "load scripts/batch_hijack.py": 0.0005800724029541016,
            "load scripts/cldm.py": 0.0004119873046875,
            "load scripts/controlnet.py": 0.07448172569274902,
            "load scripts/controlnet_version.py": 0.00019311904907226562,
            "load scripts/external_code.py": 0.0004100799560546875,
            "load scripts/global_state.py": 0.00022792816162109375,
            "load scripts/hook.py": 0.00040912628173828125,
            "load scripts/logging.py": 0.00019884109497070312,
            "load scripts/lvminthin.py": 0.0001583099365234375,
            "load scripts/movie2movie.py": 0.0006709098815917969,
            "load scripts/processor.py": 0.00029087066650390625,
            "load scripts/utils.py": 0.0001361370086669922,
            "load scripts/xyz_grid_support.py": 0.001077890396118164,
            "load scripts/convert.py": 0.01726508140563965,
            "load scripts/postprocessing_rembg.py": 0.0004150867462158203,
            "load scripts/ultimate-upscale.py": 0.0005202293395996094,
            "load scripts/ldsr_model.py": 0.019198894500732422,
            "load scripts/lora_script.py": 0.11435890197753906,
            "load scripts/scunet_model.py": 0.020205020904541016,
            "load scripts/swinir_model.py": 0.020918846130371094,
            "load scripts/hotkey_config.py": 0.0003383159637451172,
            "load scripts/extra_options_section.py": 0.0002448558807373047,
            "load scripts": 1.7183337211608887,
            "load upscalers": 0.004153013229370117,
            "refresh VAE": 0.0010361671447753906,
            "refresh textual inversion templates": 4.076957702636719e-05,
            "scripts list_optimizers": 0.0001881122589111328,
            "scripts list_unets": 2.86102294921875e-06,
            "reload hypernetworks": 0.005914926528930664,
            "initialize extra networks": 0.0049211978912353516,
            "scripts before_ui_callback": 0.02525496482849121,
            "create ui": 0.559603214263916,
            "gradio launch": 0.3989229202270508,
            "add APIs": 0.05714106559753418,
            "app_started_callback/api.py": 0.0013997554779052734,
            "app_started_callback/lora_script.py": 0.00015425682067871094,
            "app_started_callback": 0.001550912857055664
        }
    },
    "Packages": [
        "absl-py==1.4.0",
        "accelerate==0.21.0",
        "addict==2.4.0",
        "aenum==3.1.14",
        "aiofiles==23.1.0",
        "aiohttp==3.8.4",
        "aiosignal==1.3.1",
        "altair==5.0.1",
        "antlr4-python3-runtime==4.9.3",
        "anyio==3.7.0",
        "async-timeout==4.0.2",
        "attrs==23.1.0",
        "basicsr==1.4.2",
        "beautifulsoup4==4.12.2",
        "blendmodes==2022",
        "boltons==23.0.0",
        "cachetools==5.3.1",
        "certifi==2023.5.7",
        "cffi==1.15.1",
        "charset-normalizer==3.1.0",
        "clean-fid==0.1.35",
        "click==8.1.3",
        "clip==1.0",
        "coloredlogs==15.0.1",
        "contourpy==1.1.0",
        "cssselect2==0.7.0",
        "cycler==0.11.0",
        "deprecation==2.1.0",
        "dynamicprompts==0.29.0",
        "einops==0.4.1",
        "exceptiongroup==1.1.1",
        "facexlib==0.3.0",
        "fastapi==0.94.0",
        "ffmpy==0.3.0",
        "filelock==3.12.2",
        "filterpy==1.4.5",
        "flatbuffers==23.5.26",
        "fonttools==4.40.0",
        "frozenlist==1.3.3",
        "fsspec==2023.6.0",
        "ftfy==6.1.1",
        "future==0.18.3",
        "fvcore==0.1.5.post20221221",
        "gdown==4.7.1",
        "gfpgan==1.3.8",
        "gitdb==4.0.10",
        "gitpython==3.1.32",
        "google-auth-oauthlib==1.0.0",
        "google-auth==2.20.0",
        "gradio-client==0.5.0",
        "gradio==3.41.2",
        "grpcio==1.56.0",
        "h11==0.12.0",
        "httpcore==0.15.0",
        "httpx==0.24.1",
        "huggingface-hub==0.15.1",
        "humanfriendly==10.0",
        "idna==3.4",
        "imageio==2.31.1",
        "importlib-metadata==6.7.0",
        "importlib-resources==6.1.1",
        "inflection==0.5.1",
        "iopath==0.1.9",
        "jinja2==3.1.2",
        "jsonmerge==1.8.0",
        "jsonschema==4.17.3",
        "kiwisolver==1.4.4",
        "kornia==0.6.7",
        "lark==1.1.2",
        "lazy-loader==0.2",
        "lightning-utilities==0.8.0",
        "linkify-it-py==2.0.2",
        "llvmlite==0.40.1",
        "lmdb==1.4.1",
        "lpips==0.1.4",
        "lxml==4.9.2",
        "markdown-it-py==2.2.0",
        "markdown==3.4.3",
        "markupsafe==2.1.3",
        "matplotlib==3.7.1",
        "mdit-py-plugins==0.3.3",
        "mdurl==0.1.2",
        "mediapipe==0.10.1",
        "mpmath==1.3.0",
        "multidict==6.0.4",
        "networkx==3.1",
        "numba==0.57.1",
        "numpy==1.23.5",
        "oauthlib==3.2.2",
        "omegaconf==2.2.3",
        "onnxruntime==1.15.1",
        "open-clip-torch==2.20.0",
        "opencv-contrib-python==4.7.0.72",
        "opencv-python==4.7.0.72",
        "orjson==3.9.1",
        "packaging==23.1",
        "pandas==2.0.2",
        "piexif==1.1.3",
        "pillow==9.5.0",
        "pip==23.0.1",
        "platformdirs==3.8.0",
        "pooch==1.7.0",
        "portalocker==2.7.0",
        "protobuf==3.20.0",
        "psutil==5.9.5",
        "pyasn1-modules==0.3.0",
        "pyasn1==0.5.0",
        "pycparser==2.21",
        "pydantic==1.10.9",
        "pydub==0.25.1",
        "pygments==2.15.1",
        "pymatting==1.1.8",
        "pyparsing==3.1.0",
        "pyrsistent==0.19.3",
        "pysocks==1.7.1",
        "python-dateutil==2.8.2",
        "python-multipart==0.0.6",
        "pytorch-lightning==1.9.4",
        "pytz==2023.3",
        "pywavelets==1.4.1",
        "pyyaml==6.0",
        "realesrgan==0.3.0",
        "regex==2023.6.3",
        "rembg==2.0.38",
        "reportlab==4.0.4",
        "requests-oauthlib==1.3.1",
        "requests==2.31.0",
        "resize-right==0.0.2",
        "rsa==4.9",
        "safetensors==0.3.1",
        "scikit-image==0.21.0",
        "scipy==1.10.1",
        "seaborn==0.12.2",
        "semantic-version==2.10.0",
        "send2trash==1.8.2",
        "sentencepiece==0.1.99",
        "setuptools==67.6.1",
        "six==1.16.0",
        "smmap==5.0.0",
        "sniffio==1.3.0",
        "sounddevice==0.4.6",
        "soupsieve==2.4.1",
        "starlette==0.26.1",
        "svglib==1.5.1",
        "sympy==1.12",
        "tabulate==0.9.0",
        "tb-nightly==2.14.0a20230623",
        "tensorboard-data-server==0.7.1",
        "termcolor==2.3.0",
        "tifffile==2023.4.12",
        "timm==0.9.2",
        "tinycss2==1.2.1",
        "tokenizers==0.13.3",
        "tomesd==0.1.3",
        "tomli==2.0.1",
        "toolz==0.12.0",
        "torch==2.0.1",
        "torchdiffeq==0.2.3",
        "torchmetrics==0.11.4",
        "torchsde==0.2.5",
        "torchvision==0.15.2",
        "tqdm==4.65.0",
        "trampoline==0.1.2",
        "transformers==4.30.2",
        "typing-extensions==4.6.3",
        "tzdata==2023.3",
        "uc-micro-py==1.0.2",
        "ultralytics==8.0.124",
        "urllib3==1.26.16",
        "uvicorn==0.22.0",
        "wcwidth==0.2.6",
        "webencodings==0.5.1",
        "websockets==11.0.3",
        "werkzeug==2.3.6",
        "wheel==0.40.0",
        "yacs==0.1.8",
        "yapf==0.40.1",
        "yarl==1.9.2",
        "zipp==3.15.0"
    ]
}

What browsers do you use to access the UI ?

Google Chrome

Console logs

100%|███████████████████████████████████████████| 20/20 [00:48<00:00,  2.42s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:48<00:00,  2.41s/it]
100%|███████████████████████████████████████████| 20/20 [01:17<00:00,  3.86s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:17<00:00,  3.88s/it]
100%|███████████████████████████████████████████| 20/20 [00:44<00:00,  2.20s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:43<00:00,  2.15s/it]
100%|███████████████████████████████████████████| 20/20 [00:54<00:00,  2.74s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:52<00:00,  2.64s/it]
100%|███████████████████████████████████████████| 20/20 [00:51<00:00,  2.57s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:49<00:00,  2.46s/it]
100%|███████████████████████████████████████████| 20/20 [00:51<00:00,  2.58s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:49<00:00,  2.47s/it]
100%|███████████████████████████████████████████| 20/20 [00:48<00:00,  2.43s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:46<00:00,  2.35s/it]
100%|███████████████████████████████████████████| 20/20 [00:55<00:00,  2.78s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:53<00:00,  2.67s/it]
100%|███████████████████████████████████████████| 20/20 [00:46<00:00,  2.34s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:45<00:00,  2.28s/it]
100%|███████████████████████████████████████████| 20/20 [00:43<00:00,  2.18s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:41<00:00,  2.08s/it]
100%|███████████████████████████████████████████| 20/20 [01:18<00:00,  3.90s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:17<00:00,  3.90s/it]
100%|███████████████████████████████████████████| 20/20 [00:44<00:00,  2.22s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:43<00:00,  2.19s/it]
100%|███████████████████████████████████████████| 20/20 [00:52<00:00,  2.61s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:49<00:00,  2.50s/it]
100%|███████████████████████████████████████████| 20/20 [00:51<00:00,  2.58s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:49<00:00,  2.47s/it]
100%|███████████████████████████████████████████| 20/20 [00:43<00:00,  2.16s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:41<00:00,  2.06s/it]
100%|███████████████████████████████████████████| 20/20 [00:44<00:00,  2.21s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:42<00:00,  2.10s/it]
100%|███████████████████████████████████████████| 20/20 [01:10<00:00,  3.52s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:10<00:00,  3.52s/it]
100%|███████████████████████████████████████████| 20/20 [01:10<00:00,  3.52s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:08<00:00,  3.40s/it]
100%|███████████████████████████████████████████| 20/20 [01:08<00:00,  3.44s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:06<00:00,  3.33s/it]
100%|███████████████████████████████████████████| 20/20 [01:10<00:00,  3.53s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:09<00:00,  3.47s/it]
100%|███████████████████████████████████████████| 20/20 [00:32<00:00,  1.62s/it]
Total progress:  35%|█████████▊                  | 7/20 [00:31<00:58,  4.53s/it]
51it [01:24,  1.66s/it]███████▊                  | 7/20 [00:31<00:59,  4.57s/it]
Total progress:  85%|██████████████████████▉    | 17/20 [01:20<00:14,  4.72s/it]
100%|███████████████████████████████████████████| 20/20 [00:41<00:00,  2.08s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:41<00:00,  2.07s/it]
100%|███████████████████████████████████████████| 20/20 [01:11<00:00,  3.57s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:11<00:00,  3.55s/it]
100%|███████████████████████████████████████████| 20/20 [01:08<00:00,  3.41s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:07<00:00,  3.39s/it]
100%|███████████████████████████████████████████| 20/20 [01:07<00:00,  3.39s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:07<00:00,  3.38s/it]
100%|███████████████████████████████████████████| 20/20 [01:10<00:00,  3.51s/it]
Total progress: 100%|███████████████████████████| 20/20 [01:08<00:00,  3.41s/it]
100%|███████████████████████████████████████████| 19/19 [00:40<00:00,  2.12s/it]
Total progress:  95%|█████████████████████████▋ | 19/20 [00:39<00:02,  2.07s/it]
100%|███████████████████████████████████████████| 19/19 [00:39<00:00,  2.07s/it]
Total progress:  95%|█████████████████████████▋ | 19/20 [00:39<00:02,  2.06s/it]
100%|███████████████████████████████████████████| 19/19 [00:42<00:00,  2.26s/it]
Total progress:  95%|█████████████████████████▋ | 19/20 [00:39<00:02,  2.08s/it]
100%|███████████████████████████████████████████| 20/20 [00:43<00:00,  2.17s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:44<00:00,  2.21s/it]
Total progress: 100%|███████████████████████████| 20/20 [00:44<00:00,  2.20s/it]

Additional information

No response

silveroxides commented 10 months ago

I decided to test this on my machine. i5-7600k, Win11, GPU RTX 4060 ti 16 GB and for me the images that appear are in accordance to the uploaded ones on penultimate side when I activate this option. Are you certain you managed to get it activated when testing?

"Another note is that I thought maybe the option named "Always discard next-to-last sigma" was related to this issue, and while it does produce images without this problem, the images are entirely different."

In addition the issue was only present on this checkpoint which from what I know is only fine tuned on 512x512 images yet you sample at 768x768.

flux627 commented 10 months ago

I'm not sure what happened before, but now this seems to work.

While this is great news, I wonder if this is an acceptable solution. Why would this option need to be turned on for them to function properly? Is there a reason why these schedulers don't already do what this option does to fix them? Would they at least benefit from some indication that they only work with this special, hidden, non-obvious option?

flux627 commented 10 months ago

In addition the issue was only present on this checkpoint which from what I know is only fine tuned on 512x512 images yet you sample at 768x768.

I simplified for reproduction, but this was happening on all other checkpoints I was trying it on as well. I was using Hires fix, which indeed runs an img2img diffusion process at 768x768. If this bug only happens when generating on a mismatched resolution, it is still a bug. Generating at arbitrary resolutions should be supported. (I think this is upheld implicitly by virtue of how hires fix works.)

For what it's worth- I just tried it, and I'm still getting the same problem when generating 512x512 images:

DPM++ 2M SDE: Penultimate Final
1 00006-2942166222