[Bug]: Fails to unload last used model when switching models

Checklist

[X] The issue exists after disabling all extensions
[ ] The issue exists on a clean installation of webui
[ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
[X] The issue exists in the current version of the webui
[X] The issue has not been reported before recently
[ ] The issue has been reported before but has not been fixed yet

What happened?

Previous checkpoint does not unload from memory when switching to a new checkpoint. Lora and Lycoris implementation is unaffected, specifically only happens when switching checkpoints. Very likely related to commit https://github.com/Panchovix/stable-diffusion-webui-reForge/commit/d77ace3cb2497bf319f8eab1e4a316cee7259d8e, reverting to commit https://github.com/Panchovix/stable-diffusion-webui-reForge/commit/9c70926801e089f6efcb9c2a759a7b81cc4e44de resolves this issue.

Steps to reproduce the problem

Load a checkpoint by generating an image
Switch checkpoints and generate another image
Switch checkpoints and generate another image

What should have happened?

WebUI should properly unload the checkpoint after it is no longer in use/the wrapped callback should properly fire off.

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

Sysinfo

{ "Platform": "Windows-10-10.0.22631-SP0", "Python": "3.10.11", "Version": "f0.0.18v1.10.0RC-latest-497-g03c3c9e3", "Commit": "082d72a977f884be03f3230bab8e009150237c03 ", "Commandline": [ "launch.py", "--disable-xformers", "--disable-nan-check", "--pin-shared-memory", "--cuda-malloc", "--cuda-stream" ], "Torch env info": { "torch_version": "2.2.2+cu121", "is_debug_build": "False", "cuda_compiled_version": "12.1", "gcc_version": null, "clang_version": null, "cmake_version": "version 3.28.3", "os": "Microsoft Windows 11 Pro", "libc_version": "N/A", "python_version": "3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)", "python_platform": "Windows-10-10.0.22631-SP0", "is_cuda_available": "True", "cuda_runtime_version": "11.8.89\r", "cuda_module_loading": "LAZY", "nvidia_driver_version": "552.44", "nvidia_gpu_models": "GPU 0: NVIDIA GeForce RTX 3090", "cudnn_version": null, "pip_version": "pip3", "pip_packages": [ "clip-anytorch==2.5.0", "lion-pytorch==0.0.6", "numpy==1.26.2", "onnx==1.14.1", "onnx-graphsurgeon==0.3.12", "onnxruntime==1.13.1", "open-clip-torch==2.20.0", "pytorch-lightning==1.9.4", "torch==2.2.2+cu121", "torchaudio==2.2.2+cu121", "torchdiffeq==0.2.3", "torchmetrics==0.11.0", "torchsde==0.2.6", "torchvision==0.17.2+cu121" ], "conda_packages": null, "hip_compiled_version": "N/A", "hip_runtime_version": "N/A", "miopen_runtime_version": "N/A", "caching_allocator_config": "backend:cudaMallocAsync", "is_xnnpack_available": "True", "cpu_info": [ "Architecture=9", "CurrentClockSpeed=3700", "DeviceID=CPU0", "Family=205", "L2CacheSize=9728", "L2CacheSpeed=", "Manufacturer=GenuineIntel", "MaxClockSpeed=3700", "Name=12th Gen Intel(R) Core(TM) i5-12600K", "ProcessorType=3", "Revision=" ] }, "Exceptions": [], "CPU": { "model": "Intel64 Family 6 Model 151 Stepping 2, GenuineIntel", "count logical": 16, "count physical": 10 }, "RAM": { "total": "32GB", "used": "15GB", "free": "17GB" }, "Startup": { "total": 45.30704092979431, "records": { "initial startup": 0.018036842346191406, "prepare environment/checks": 0.00851130485534668, "prepare environment/git version info": 0.09314441680908203, "prepare environment/torch GPU test": 1.7084028720855713, "prepare environment/clone repositores": 0.15220975875854492, "prepare environment/run extensions_builtin installers/canvas-zoom-and-pan": 0.0, "prepare environment/run extensions_builtin installers/extra-options-section": 0.0, "prepare environment/run extensions_builtin installers/forge_legacy_preprocessors": 0.301959753036499, "prepare environment/run extensions_builtin installers/forge_preprocessor_inpaint": 0.0, "prepare environment/run extensions_builtin installers/forge_preprocessor_marigold": 0.0, "prepare environment/run extensions_builtin installers/forge_preprocessor_normalbae": 0.0, "prepare environment/run extensions_builtin installers/forge_preprocessor_recolor": 0.0, "prepare environment/run extensions_builtin installers/forge_preprocessor_reference": 0.0, "prepare environment/run extensions_builtin installers/forge_preprocessor_revision": 0.0, "prepare environment/run extensions_builtin installers/forge_preprocessor_tile": 0.0, "prepare environment/run extensions_builtin installers/LDSR": 0.0010004043579101562, "prepare environment/run extensions_builtin installers/Lora": 0.0, "prepare environment/run extensions_builtin installers/mobile": 0.0, "prepare environment/run extensions_builtin installers/prompt-bracket-checker": 0.0, "prepare environment/run extensions_builtin installers/ScuNET": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_controlllite": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_controlnet": 0.2969057559967041, "prepare environment/run extensions_builtin installers/sd_forge_controlnet_example": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_dynamic_thresholding": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_fooocus_inpaint": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_freeu": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_hypertile": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_ipadapter": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_kohya_hrfix": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_latent_modifier": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_multidiffusion": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_neveroom": 0.0010075569152832031, "prepare environment/run extensions_builtin installers/sd_forge_photomaker": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_sag": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_stylealign": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_svd": 0.0, "prepare environment/run extensions_builtin installers/sd_forge_z123": 0.0, "prepare environment/run extensions_builtin installers/soft-inpainting": 0.0, "prepare environment/run extensions_builtin installers/SwinIR": 0.0, "prepare environment/run extensions_builtin installers": 0.6008734703063965, "prepare environment": 5.971784830093384, "launcher": 0.00725245475769043, "import torch": 7.289178848266602, "import gradio": 0.6588046550750732, "setup paths": 0.5453004837036133, "import ldm": 0.005999326705932617, "import sgm": 0.0, "initialize shared": 0.047280311584472656, "other imports": 0.6421997547149658, "opts onchange": 0.0, "setup SD model": 0.0, "setup codeformer": 0.002001047134399414, "setup gfpgan": 0.013757705688476562, "set samplers": 0.0, "list extensions": 0.0025076866149902344, "restore config state file": 0.0, "list SD models": 0.03587150573730469, "list localizations": 0.0011713504791259766, "load scripts/custom_code.py": 0.007056236267089844, "load scripts/img2imgalt.py": 0.0009415149688720703, "load scripts/loopback.py": 0.0, "load scripts/outpainting_mk_2.py": 0.0, "load scripts/poor_mans_outpainting.py": 0.0013077259063720703, "load scripts/postprocessing_caption.py": 0.0, "load scripts/postprocessing_codeformer.py": 0.0010027885437011719, "load scripts/postprocessing_create_flipped_copies.py": 0.0, "load scripts/postprocessing_focal_crop.py": 0.005006551742553711, "load scripts/postprocessing_gfpgan.py": 0.0, "load scripts/postprocessing_split_oversized.py": 0.0009951591491699219, "load scripts/postprocessing_upscale.py": 0.0, "load scripts/processing_autosized_crop.py": 0.0010559558868408203, "load scripts/prompt_matrix.py": 0.0, "load scripts/prompts_from_file.py": 0.0009431838989257812, "load scripts/sd_upscale.py": 0.0, "load scripts/xyz_grid.py": 0.004775524139404297, "load scripts/ldsr_model.py": 0.6216516494750977, "load scripts/lora_script.py": 1.6568269729614258, "load scripts/scunet_model.py": 0.08301591873168945, "load scripts/swinir_model.py": 0.07204961776733398, "load scripts/hotkey_config.py": 0.0, "load scripts/extra_options_section.py": 0.0012493133544921875, "load scripts/legacy_preprocessors.py": 0.008629560470581055, "load scripts/preprocessor_inpaint.py": 0.011592864990234375, "load scripts/preprocessor_marigold.py": 0.1029508113861084, "load scripts/preprocessor_normalbae.py": 0.008066415786743164, "load scripts/preprocessor_recolor.py": 0.0, "load scripts/forge_reference.py": 0.0025224685668945312, "load scripts/preprocessor_revision.py": 0.0009965896606445312, "load scripts/preprocessor_tile.py": 0.0009992122650146484, "load scripts/forge_controllllite.py": 0.014441967010498047, "load scripts/controlnet.py": 0.7233881950378418, "load scripts/xyz_grid_support.py": 0.002000093460083008, "load scripts/sd_forge_controlnet_example.py": 0.0010004043579101562, "load scripts/forge_dynamic_thresholding.py": 0.0019986629486083984, "load scripts/forge_fooocus_inpaint.py": 0.0010824203491210938, "load scripts/forge_freeu.py": 0.12163543701171875, "load scripts/forge_hypertile.py": 0.0028247833251953125, "load scripts/forge_ipadapter.py": 0.012799501419067383, "load scripts/kohya_hrfix.py": 0.0020034313201904297, "load scripts/forge_latent_modifier.py": 0.003366231918334961, "load scripts/forge_multidiffusion.py": 0.009516716003417969, "load scripts/forge_never_oom.py": 0.0009996891021728516, "load scripts/forge_photomaker.py": 0.002997875213623047, "load scripts/forge_sag.py": 0.0019991397857666016, "load scripts/forge_stylealign.py": 0.0, "load scripts/forge_svd.py": 0.10950779914855957, "load scripts/forge_z123.py": 0.08874249458312988, "load scripts/soft_inpainting.py": 0.0010004043579101562, "load scripts/smea.py": 0.0020508766174316406, "load scripts/api.py": 0.35265541076660156, "load scripts/gen_hashing.py": 0.0826115608215332, "load scripts/info.py": 0.08820843696594238, "load scripts/link.py": 0.08614540100097656, "load scripts/pasted.py": 0.0918588638305664, "load scripts/previews.py": 0.0838167667388916, "load scripts/settings.py": 0.07657694816589355, "load scripts/ui_funcs.py": 0.0, "load scripts/dtg.py": 0.23250722885131836, "load scripts/image_browser.py": 0.21125054359436035, "load scripts/model_keyword.py": 0.10178279876708984, "load scripts/toolkit_gui.py": 0.20426249504089355, "load scripts/model_keyword_support.py": 0.006497859954833984, "load scripts/shared_paths.py": 0.0009996891021728516, "load scripts/tag_frequency_db.py": 0.0009996891021728516, "load scripts/cc.py": 0.007063388824462891, "load scripts/cc_callback.py": 0.08338785171508789, "load scripts/cc_colorpicker.py": 0.0009992122650146484, "load scripts/cc_const.py": 0.0, "load scripts/cc_hdr.py": 0.0, "load scripts/cc_scaling.py": 0.0010001659393310547, "load scripts/cc_settings.py": 0.08312034606933594, "load scripts/cc_style.py": 0.0010654926300048828, "load scripts/cc_xyz.py": 0.0, "load scripts/comments.py": 0.08637547492980957, "load scripts/refiner.py": 0.001222372055053711, "load scripts/sampler.py": 0.0007152557373046875, "load scripts/seed.py": 0.0, "load scripts": 7.495496988296509, "load upscalers": 0.008581161499023438, "refresh VAE": 0.003508329391479492, "refresh textual inversion templates": 0.0, "scripts list_optimizers": 0.06181168556213379, "scripts list_unets": 0.0, "reload hypernetworks": 0.0, "initialize extra networks": 0.004005908966064453, "scripts before_ui_callback": 0.0010046958923339844, "create ui": 17.566843271255493, "gradio launch": 4.597225666046143, "add APIs": 0.3663618564605713, "app_started_callback/lora_script.py": 0.0, "app_started_callback/controlnet.py": 0.0009999275207519531, "app_started_callback/!adetailer.py": 0.0, "app_started_callback/link.py": 0.0, "app_started_callback/api.py": 0.0, "app_started_callback/info.py": 0.0009999275207519531, "app_started_callback/previews.py": 0.0010006427764892578, "app_started_callback/model_keyword.py": 0.0, "app_started_callback/tag_autocomplete_helper.py": 0.0025055408477783203, "app_started_callback": 0.005506038665771484 } }, "Packages": [ "-ensorflow-intel==2.11.0", "absl-py==1.4.0", "accelerate==0.21.0", "addict==2.4.0", "aenum==3.1.11", "aiofiles==23.2.1", "aiohttp==3.8.3", "aiosignal==1.3.1", "albumentations==1.4.3", "aliyun-python-sdk-core==2.14.0", "aliyun-python-sdk-kms==2.16.2", "altair==4.2.2", "analytics-python==1.4.0", "ansicon==1.89.0", "antlr4-python3-runtime==4.9.3", "anyio==3.6.2", "appdirs==1.4.4", "argon2-cffi-bindings==21.2.0", "argon2-cffi==23.1.0", "arrow==1.2.3", "astunparse==1.6.3", "async-timeout==4.0.2", "attrs==22.2.0", "backoff==2.2.1", "basicsr==1.4.2", "baukit==0.0.1", "bayesian-optimization==1.4.2", "bcrypt==4.0.1", "beautifulsoup4==4.12.2", "bidict==0.22.1", "bitsandbytes-windows==0.37.5", "bitsandbytes==0.41.2.post2", "blendmodes==2022", "blessed==1.20.0", "boltons==21.0.0", "cachetools==5.2.1", "certifi==2022.12.7", "cffi==1.15.1", "chardet==3.0.4", "charset-normalizer==2.1.1", "clean-fid==0.1.35", "click==8.1.3", "clip-anytorch==2.5.0", "clip-interrogator==0.6.0", "cloudpickle==2.2.1", "cmake==3.28.3", "colorama==0.4.6", "coloredlogs==15.0.1", "colorlog==6.8.2", "compel==2.0.2", "contourpy==1.0.7", "crcmod==1.7", "croniter==1.4.1", "cryptography==39.0.0", "cssselect2==0.7.0", "cstr==0.1.0", "cycler==0.11.0", "cython==3.0.8", "dadaptation==3.1", "datasets==2.11.0", "dateutils==0.6.12", "deepdiff==6.3.1", "deepl==1.13.0", "deprecation==2.1.0", "depth-anything==2024.1.22.0", "diffuser==0.0.1", "diffusers==0.25.0", "dill==0.3.6", "diskcache==5.6.3", "distro==1.9.0", "docker-pycreds==0.4.0", "dsine==2024.3.23", "easydict==1.11", "easygui==0.98.3", "einops==0.4.1", "embreex==2.17.7.post4", "entrypoints==0.4", "exceptiongroup==1.1.1", "facexlib==0.3.0", "fairscale==0.4.13", "fastapi==0.94.0", "ffmpy==0.3.0", "filelock==3.9.0", "filterpy==1.4.5", "flatbuffers==23.5.26", "font-roboto==0.0.1", "fonts==0.0.3", "fonttools==4.38.0", "frozenlist==1.3.3", "fsspec==2024.3.1", "ftfy==6.1.1", "future==0.18.3", "fvcore==0.1.5.post20221221", "gast==0.4.0", "gdown==5.1.0", "geffnet==1.0.2", "gfpgan==1.3.8", "gitdb==4.0.10", "gitpython==3.1.32", "glob2==0.5", "google-auth-oauthlib==0.4.6", "google-auth==2.16.0", "google-pasta==0.2.0", "googletrans==4.0.0rc1", "gradio-client==0.5.0", "gradio==3.41.2", "graphsurgeon==0.4.6", "grpcio==1.51.1", "h11==0.12.0", "h2==3.2.0", "h5py==3.8.0", "handrefinerportable==2024.2.12.0", "hpack==3.0.0", "hstspreload==2023.1.1", "httpcore==0.15.0", "httpx==0.24.1", "huggingface-hub==0.19.4", "humanfriendly==10.0", "hydra-core==1.3.2", "hyperframe==5.2.0", "hyperopt==0.2.7", "idna==2.10", "image-reward==1.3", "imageio-ffmpeg==0.4.9", "imageio==2.34.0", "img2texture==1.0.6", "imhist==0.0.4", "importlib-metadata==6.0.0", "importlib-resources==6.1.1", "imwatermark==0.0.2", "inflection==0.5.1", "iniconfig==2.0.0", "inquirer==3.1.3", "insightface==0.7.3", "invisible-watermark==0.2.0", "iopath==0.1.9", "itsdangerous==2.1.2", "jax==0.4.28", "jaxlib==0.4.28", "jinja2==3.1.2", "jinxed==1.2.0", "jmespath==0.10.0", "joblib==1.2.0", "jsonmerge==1.8.0", "jsonschema==4.17.3", "k-diffusion==0.0.12", "keras==2.14.0", "keyboard==0.13.5", "kiwisolver==1.4.4", "kornia==0.6.7", "lark==1.1.2", "lazy-loader==0.2", "libclang==15.0.6.1", "library==0.0.0", "lightning-cloud==0.5.37", "lightning-lite==1.8.6", "lightning-utilities==0.8.0", "lightning==2.0.6", "linkify-it-py==1.0.3", "lion-pytorch==0.0.6", "llama-cpp-python==0.2.26+cu121", "llvmlite==0.42.0", "lmdb==1.4.0", "lpips==0.1.4", "lxml==5.1.0", "lycoris-lora==2.0.2", "mako==1.3.2", "mapbox-earcut==1.0.1", "markdown-it-py==2.2.0", "markdown==3.4.1", "markupsafe==2.1.3", "matplotlib==3.6.3", "matrix-client==0.4.0", "mdit-py-plugins==0.3.3", "mdurl==0.1.2", "mediapipe==0.10.14", "ml-dtypes==0.2.0", "model-index==0.1.11", "monotonic==1.6", "mpmath==1.3.0", "multidict==6.0.4", "multiprocess==0.70.14", "networkx==3.2.1", "ngrok==1.0.0", "numba==0.59.0", "numpy==1.26.2", "oauthlib==3.2.2", "omegaconf==2.2.3", "onnx-graphsurgeon==0.3.12", "onnx==1.14.1", "onnxruntime==1.13.1", "open-clip-torch==2.20.0", "openai==1.14.3", "opencv-contrib-python==4.7.0.72", "opencv-python-headless==4.10.0.82", "opencv-python==4.9.0.80", "opendatalab==0.0.10", "openmim==0.3.9", "openxlab==0.0.34", "opt-einsum==3.3.0", "ordered-set==4.1.0", "orjson==3.8.5", "oss2==2.17.0", "packaging==23.0", "pandas==1.5.3", "paramiko==2.12.0", "pathtools==0.1.2", "piexif==1.1.3", "pilgram==1.2.1", "pillow-avif-plugin==1.4.3", "pillow==9.5.0", "pip==24.0", "platformdirs==4.2.0", "pluggy==1.0.0", "pooch==1.8.1", "portalocker==2.8.2", "prettytable==3.9.0", "prodigyopt==1.0", "protobuf==3.20.0", "psutil==5.9.5", "py-cpuinfo==9.0.0", "py4j==0.10.9.7", "pyarrow==12.0.1", "pyasn1-modules==0.2.8", "pyasn1==0.4.8", "pyaudio==0.2.13", "pycocotools==2.0.7", "pycollada==0.8", "pycparser==2.21", "pycryptodome==3.16.0", "pycuda==2024.1", "pydantic==1.10.15", "pydeprecate==0.3.2", "pydub==0.25.1", "pygments==2.14.0", "pyjwt==2.8.0", "pymatting==1.1.12", "pynacl==1.5.0", "pyparsing==3.0.9", "pyreadline3==3.4.1", "pyrsistent==0.19.3", "pysocks==1.7.1", "pytest==7.2.2", "python-dateutil==2.8.2", "python-dotenv==0.21.1", "python-editor==1.0.4", "python-engineio==4.9.0", "python-multipart==0.0.6", "python-socketio==5.11.1", "pytools==2023.1.1", "pytorch-lightning==1.9.4", "pytz==2023.4", "pywavelets==1.4.1", "pywin32==305", "pyyaml==6.0", "qudida==0.0.4", "readchar==4.0.5", "realesrgan==0.3.0", "regex==2022.10.31", "rembg==2.0.56", "reportlab==4.1.0", "requests-oauthlib==1.3.1", "requests==2.28.2", "resize-right==0.0.2", "responses==0.18.0", "rfc3986==1.5.0", "rich==13.4.2", "rsa==4.9", "rtree==1.2.0", "safetensors==0.4.2", "scikit-image==0.21.0", "scikit-learn==1.5.0", "scipy==1.11.4", "seaborn==0.12.2", "segment-anything==1.0", "semantic-version==2.10.0", "send2trash==1.8.2", "sentencepiece==0.1.97", "sentry-sdk==1.13.0", "setproctitle==1.3.2", "setuptools==69.5.1", "shapely==2.0.2", "simple-websocket==1.0.0", "simpleeval==0.9.13", "six==1.16.0", "smmap==5.0.0", "sniffio==1.3.0", "sounddevice==0.4.6", "soundfile==0.12.1", "soupsieve==2.4.1", "spandrel-extra-arches==0.1.1", "spandrel==0.3.4", "speechrecognition==3.9.0", "starlette==0.26.1", "starsessions==1.3.0", "svg.path==6.3", "svglib==1.5.1", "sympy==1.12", "tabulate==0.9.0", "tb-nightly==2.12.0a20230120", "tensorboard-data-server==0.6.1", "tensorboard-plugin-wit==1.8.1", "tensorboard==2.14.1", "tensorflow-estimator==2.14.0", "tensorflow-intel==2.14.0", "tensorflow-io-gcs-filesystem==0.31.0", "tensorflow-io==0.31.0", "tensorflow==2.14.0", "tensorrt==8.5.3.1", "termcolor==2.2.0", "thop==0.1.1.post2209072238", "threadpoolctl==3.1.0", "tifffile==2022.10.10", "timm==0.9.2", "tinycss2==1.2.1", "tipo-kgen==0.0.9", "tk==0.1.0", "tokenizers==0.12.1", "tomesd==0.1.3", "toml==0.10.2", "tomli==2.0.1", "toolz==0.12.0", "torch==2.2.2+cu121", "torchaudio==2.2.2+cu121", "torchdiffeq==0.2.3", "torchmetrics==0.11.0", "torchsde==0.2.6", "torchvision==0.17.2+cu121", "tqdm==4.65.0", "traitlets==5.9.0", "trampoline==0.1.2", "transformers==4.30.2", "trimesh==4.1.3", "typing-extensions==4.12.0", "uc-micro-py==1.0.1", "uff==0.6.9", "ultralytics==8.2.24", "urllib3==1.26.15", "uvicorn==0.20.0", "vhacdx==0.0.5", "voluptuous==0.13.1", "wandb==0.15.11", "wcwidth==0.2.6", "webencodings==0.5.1", "websocket-client==1.6.1", "websockets==10.4", "werkzeug==2.2.2", "wheel==0.38.4", "wrapt==1.14.1", "wsproto==1.2.0", "xformers==0.0.24", "xxhash==3.2.0", "yacs==0.1.8", "yapf==0.32.0", "yarl==1.8.2", "zipp==3.11.0" ] }

Console logs

[LORA] Loaded for SDXL-UNet with 788 keys at weight 0.8 (skipped 0 keys)
[LORA] Loaded for SDXL-UNet with 788 keys at weight 0.8 (skipped 0 keys)
To load target model SDXL
Begin to load 1 model
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  16553.020650863647
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  15529.020650863647
Moving model(s) has taken 7.29 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:13<00:00,  2.02it/s]
To load target model AutoencoderKL████████████████████████████████████▉                | 28/37 [00:22<00:04,  2.25it/s]
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  16528.538228988647
[Memory Management] Model Memory (MB) =  159.55708122253418
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  15344.981147766113
Moving model(s) has taken 0.04 seconds
Cleanup minimal inference memory.
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 35/35 [00:03<00:00, 11.27it/s]
To load target model SDXL
Begin to load 1 model
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  21214.427976608276
[Memory Management] Model Memory (MB) =  4880.078125
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  15310.349851608276
Moving model(s) has taken 1.53 seconds
100%|████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:10<00:00,  1.13s/it]
Loading CLiP model ViT-L/14 ███████████████████████████████████████████████████████████| 37/37 [00:39<00:00,  1.20s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 37/37 [00:47<00:00,  1.27s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 37/37 [00:47<00:00,  1.20s/it]To load target model SDXL                                                                         | 0/9 [00:00<?, ?it/s]
Begin to load 1 model
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  21240.19974708557
[Memory Management] Model Memory (MB) =  4880.078125
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  15336.121622085571
Moving model(s) has taken 1.59 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:12<00:00,  2.17it/s]
Total progress:  76%|█████████████████████████████████████████████████▉                | 28/37 [00:14<00:04,  1.93it/s]
 ------------ Unloading least recently used model: .safetensors -------------
Loading model (1 out of 1)
Loading weights .safetensors
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.text_projection'}
loaded straight to GPU
To load target model SDXL
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  11464.422870635986
[Memory Management] Model Memory (MB) =  0.03814697265625
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  10440.38472366333
Moving model(s) has taken 0.05 seconds
Loading VAE weights specified in settings: sdxl_vae-fp16fix-blessed.safetensors
To load target model SDXLClipModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  11464.371143341064
[Memory Management] Model Memory (MB) =  2144.3546981811523
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  8296.016445159912
Moving model(s) has taken 0.61 seconds
Model loaded in 14.9s (calculate hash: 0.4s, load weights from disk: 0.4s, forge load real models: 11.5s, load VAE: 0.6s, load textual inversion embeddings: 1.2s, calculate empty prompt: 0.7s).
Event handler triggered: sd_model_checkpoint
                                                                                                                       [LORA] Loaded safetensors for SDXL-UNet with 788 keys at weight 0.8 (skipped 0 keys)
[LORA] Loaded safetensors for SDXL-UNet with 788 keys at weight 0.8 (skipped 0 keys)
To load target model SDXL
Begin to load 1 model
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  9705.407234191895
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  8681.407234191895
Moving model(s) has taken 3.35 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:13<00:00,  2.14it/s]
To load target model AutoencoderKL████████████████████████████████████▉                | 28/37 [00:17<00:04,  2.20it/s]
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  9153.155281066895
[Memory Management] Model Memory (MB) =  159.55708122253418
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  7969.59819984436
Moving model(s) has taken 0.05 seconds
Cleanup minimal inference memory.
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 35/35 [00:03<00:00, 11.65it/s]
To load target model SDXL
Begin to load 1 model
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  14398.814559936523
[Memory Management] Model Memory (MB) =  4880.078125
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  8494.736434936523
Moving model(s) has taken 1.29 seconds
100%|████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:09<00:00,  1.10s/it]
Memory cleanup has taken 1.16 seconds██████████████████████████████████████████████████| 37/37 [00:33<00:00,  1.16s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 37/37 [00:36<00:00,  1.00it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 37/37 [00:36<00:00,  1.16s/it]

Additional information

I took a look at the commit and while the model is removed from the array of loaded models, at no point is free_memory() in model_management.py called. So the model is being removed from the loaded models list without deallocating the memory. In fact, no method to unload the memory is actually called despite there being a print message. Unlike Auto1111, Forge does not automatically deallocate the model when it's removed from the loaded_sd_model array, it calls unload_all_models() whenever the model name changes from the checkpoint name in the dropdown.

I recommend immediately reverting the commit because the primary advantage of Forge is its superior memory handling and this is a pretty big issue but if you want to maintain this functionality in reForge, you would have to rewrite the callback so that it actually calls model_management.py to unload the model and free up memory.

Panchovix / stable-diffusion-webui-reForge