[Bug]: subsequent vae decodes use more VRAM than the first generation of an instance

Our0b0r05 commented 2 months ago

Checklist

[X] The issue exists after disabling all extensions
[X] The issue exists on a clean installation of webui
[ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
[X] The issue exists in the current version of the webui
[ ] The issue has not been reported before recently
[ ] The issue has been reported before but has not been fixed yet

What happened?

The first generation of an instance uses a lower amount of VRAM than subsequent generations.

Steps to reproduce the problem

change the resolution to a decent size (1664x2432 in my case)
generate two separate images.

What should have happened?

Both generations should have used the same amount of VRAM for the vae decode but every generation after the first uses substantially more (up to 6gb in my testing)

What browsers do you use to access the UI ?

Mozilla Firefox

Sysinfo

sysinfo-2024-08-09-05-52.json

Console logs

Gen1:
To load target model AutoencoderKL█████████████████████████████████████████████████████| 30/30 [00:50<00:00,  3.16s/it]
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  11163.0780544281
[Memory Management] Model Memory (MB) =  159.55708122253418
[Memory Management] Minimal Inference Memory (MB) =  16811.4375
[Memory Management] Estimated Remaining GPU Memory (MB) =  -5807.916526794434

Gen2:
Unloading models for lowram load.██████████████████████████████████████████████████████| 30/30 [00:48<00:00,  3.13s/it]
1 models unloaded.
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  10993.583637237549
[Memory Management] Model Memory (MB) =  4897.086494445801
[Memory Management] Minimal Inference Memory (MB) =  16811.4375
[Memory Management] Estimated Remaining GPU Memory (MB) =  -10714.940357208252
[Memory Management] Requested ASYNC Preserved Memory (MB) =  4397.43345489502
[Memory Management] Parameters Loaded to ASYNC Stream (MB) =  499.791259765625
[Memory Management] Parameters Loaded to GPU (MB) =  4397.2570877075195

Additional information

No response

Zotikus1001 commented 2 months ago

From my experience, those memory values are all out of whack, and not to be believed, especially when using any memory affecting launch flags.

Our0b0r05 commented 2 months ago

From my experience, those memory values are all out of whack, and not to be believed, especially when using any memory affecting launch flags.

I've tried using no startup flags and the issue persists. Mind you I'm not basing this completely off of console output but also resource management software. I can confirm that the Vae decode actually does use more VRAM after the first generation.

Panchovix commented 2 months ago

Hi there, I did some reverts yesterday related to model management, can you see if it works as it should on dev_upstream?

Also, it could be maybe for the re-do of the OOM extension. You can also try git revert -m 1 4bc6747 on your local setup and that should basically revert the extension as how it is on main branch.

Also, does this happens on main branch?

Panchovix commented 2 months ago

I have also created a new experimental branch with some changes https://github.com/Panchovix/stable-diffusion-webui-reForge/commits/dev_upstream_experimental, can you see if the issue persists there?

Our0b0r05 commented 2 months ago

I have also created a new experimental branch with some changes https://github.com/Panchovix/stable-diffusion-webui-reForge/commits/dev_upstream_experimental, can you see if the issue persists there?

It looks like the bug is fixed on this branch thanks.

Panchovix / stable-diffusion-webui-reForge