Cannot use only disliked images

When I try to use a single disliked image, I get an error of mismatched matrices devices.
Below is the full console output of running 1 liked and 1 disliked image then interrupting it after seeing that it works, but then clearing the liked image and running it again.
I found that the self.time_embed Sequential is on device('cpu') with dtype=float32 but the t_emb is on device('cuda:0') with dtype=torch.cuda.HalfTensor.
Is this a problem on my end?

D:\stable-diffusion-webui>set COMMANDLINE_ARGS=--medvram --no-half-vae --gradio-img2img-tool editor --api --cors-allow-origins http://localhost:5173
venv "D:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]
Version: v1.5.2
Commit hash: c9c8485bc1e8720aba70f029d25cba1c4abf2b5c

Launching Web UI with arguments: --medvram --no-half-vae --gradio-img2img-tool editor --api --cors-allow-origins http://localhost:5173
No SDP backend available, likely because you are running in pytorch versions < 2.0. In fact, you are using PyTorch 1.13.1+cu117. You might want to consider upgrading.
No module 'xformers'. Proceeding without it.
==============================================================================
You are running torch 1.13.1+cu117.
The program is tested to work with torch 2.0.0.
To reinstall the desired version, run with commandline flag --reinstall-torch.
Beware that this will cause a lot of large files to be downloaded, as well as
there are reports of issues with training tab on the latest version.

Use --skip-version-check commandline argument to disable this check.
==============================================================================
ControlNet v1.1.150
ControlNet v1.1.150
Loading FABRIC v0.5.2
Loading weights [2fbe5ba6e2] from D:\stable-diffusion-webui\models\Stable-diffusion\openjourney2.ckpt
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 145.6s (launcher: 37.7s, import torch: 57.8s, import gradio: 7.3s, setup paths: 10.1s, import ldm: 0.2s, other imports: 11.0s, opts onchange: 0.8s, setup codeformer: 1.0s, list extensions: 0.1s, list SD models: 2.4s, load scripts: 14.1s, create ui: 1.7s, gradio launch: 1.3s, add APIs: 0.2s).
Creating model from config: D:\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying attention optimization: Doggettx... done.
Model loaded in 102.9s (load weights from disk: 41.7s, load config: 0.1s, create model: 4.3s, apply weights to model: 19.8s, apply half(): 19.7s, apply dtype to VAE: 0.2s, load VAE: 0.7s, move model to device: 0.2s, hijack: 0.3s, load textual inversion embeddings: 4.5s, calculate empty prompt: 11.5s).
[FABRIC] Patching U-Net forward pass... (1 likes, 1 dislikes)
 10%|████████▏                                                                         | 2/20 [03:38<32:42, 109.00s/it]
[FABRIC] Restoring original U-Net forward pass                                          | 2/20 [00:58<08:43, 29.08s/it]
Total progress:  10%|██████▋                                                            | 2/20 [01:56<17:26, 58.12s/it]
[FABRIC] Patching U-Net forward pass... (0 likes, 1 dislikes)                           | 2/20 [01:56<08:43, 29.08s/it]
  0%|                                                                                           | 0/20 [00:10<?, ?it/s]
*** Error completing request
*** Arguments: ('task(cq5zqtyy586myyf)', 'beautiful scenery nature glass bottle landscape, , purple galaxy bottle,', 'text, watermark', [], 20, 16, False, False, 1, 1, 8, 0.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, '', '', [], <gradio.routes.Request object at 0x000001F2E3E553F0>, 0, <controlnet.py.UiControlNetUnit object at 0x000001F131A43D60>, [], ['f3db6f2acb7a763c.png'], True, 4, 0, 1, 0, 0.01, 1, False, False, None, None, '', '', '', '', 'Auto rename', {'label': 'Upload avatars config'}, 'Open outputs directory', 'Export to WebUI style', True, {'label': 'Presets'}, {'label': 'QC preview'}, '', [], 'Select', 'QC scan', 'Show pics', None, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, None, False, 50) {}
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\modules\call_queue.py", line 58, in f
        res = list(func(*args, **kwargs))
      File "D:\stable-diffusion-webui\modules\call_queue.py", line 37, in f
        res = func(*args, **kwargs)
      File "D:\stable-diffusion-webui\modules\txt2img.py", line 62, in txt2img
        processed = processing.process_images(p)
      File "D:\stable-diffusion-webui\modules\processing.py", line 677, in process_images
        res = process_images_inner(p)
      File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "D:\stable-diffusion-webui\modules\processing.py", line 794, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "D:\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\marking.py", line 28, in process_sample
        return process.sample_before_CN_hack(*args, **kwargs)
      File "D:\stable-diffusion-webui\modules\processing.py", line 1054, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "D:\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 464, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
      File "D:\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 303, in launch_sampling
        return func()
      File "D:\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 464, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "D:\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 189, in forward
        x_out[a:b] = self.inner_model(x_in[a:b], sigma_in[a:b], cond=make_condition_dict(subscript_cond(cond_in, a, b), image_cond_in[a:b]))
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "D:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
      File "D:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
      File "D:\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "D:\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
        return self.__orig_func(*args, **kwargs)
      File "D:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
        x_recon = self.model(x_noisy, t, **cond)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1212, in _call_impl
        result = forward_call(*input, **kwargs)
      File "D:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
        out = self.diffusion_model(x, t, context=cc)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "D:\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\patching.py", line 110, in new_forward
        return self._fabric_old_forward(x, timesteps, context, **kwargs)
      File "D:\stable-diffusion-webui\modules\sd_unet.py", line 91, in UNetModel_forward
        return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, **kwargs)
      File "D:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 789, in forward
        emb = self.time_embed(t_emb)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 204, in forward
        input = module(input)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
        return forward_call(*input, **kwargs)
      File "D:\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 361, in network_Linear_forward
        return torch.nn.Linear_forward_before_network(self, input)
      File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
    RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)

---
dvruette / sd-webui-fabric

Cannot use only disliked images #27
