OOM when using the same workflow I was able to use before

Hello. I don't know if this has to do with this issue, for a few days now I have been having memory issues when using Comfy that I didn't use to have before. I thought it had to do with LCM as I started incorporating it in my workflow but turns out, now even normal workflows won't work. I don't use Comfy's UI but instead my Krita plugin. I have always been able to work with batches of three square images, up to three loras and two controlnets with my 4GB vram card. However, today, attempting to generate a single 512x512 image with two loras and a controlnet will output the following:

got prompt
model_type EPS
adm 0
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
missing {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
loaded straight to GPU
Requested to load BaseModel
Loading 1 new model
loading in lowvram mode 256.0
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "C:\Users\jason\Stable-Diffusion\web-ui\ComfyUI\execution.py", line 153, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "C:\Users\jason\Stable-Diffusion\web-ui\ComfyUI\execution.py", line 83, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "C:\Users\jason\Stable-Diffusion\web-ui\ComfyUI\execution.py", line 76, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "C:\Users\jason\Stable-Diffusion\web-ui\ComfyUI\nodes.py", line 304, in encode
    t = vae.encode(pixels[:,:,:,:3])
  File "C:\Users\jason\Stable-Diffusion\web-ui\ComfyUI\comfy\sd.py", line 237, in encode
    self.first_stage_model = self.first_stage_model.to(self.device)
  File "C:\Users\jason\Stable-Diffusion\web-ui\automatic\venv\lib\site-packages\torch\nn\modules\module.py", line 1145, in to
    return self._apply(convert)
  File "C:\Users\jason\Stable-Diffusion\web-ui\automatic\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  File "C:\Users\jason\Stable-Diffusion\web-ui\automatic\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  File "C:\Users\jason\Stable-Diffusion\web-ui\automatic\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  [Previous line repeated 3 more times]
  File "C:\Users\jason\Stable-Diffusion\web-ui\automatic\venv\lib\site-packages\torch\nn\modules\module.py", line 820, in _apply
    param_applied = fn(param)
  File "C:\Users\jason\Stable-Diffusion\web-ui\automatic\venv\lib\site-packages\torch\nn\modules\module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
torch.cuda.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated     : 3.43 GiB
Requested               : 9.00 MiB
Device limit            : 4.00 GiB
Free (according to CUDA): 0 bytes
PyTorch limit (set by user-supplied memory fraction)
                        : 17179869184.00 GiB

This typically also causes Krita to crash afterwards.

comfyanonymous / ComfyUI

OOM when using the same workflow I was able to use before #1978