[Bug]: RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same

[Agnyy]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

I am training Textual Inversion and sometimes an error occurs. I have not been able to trace the patterns for what reason this is happening. This problem occurs in 80% of cases.

Steps to reproduce the problem

1. I start the program.
2. Set up "Create embedding"
3. Move on to training
4. After setting up, start training

What should have happened?

Training should have started.

Commit where the problem happens

Commit hash: a9eab236d7e8afa4d6205127904a385b2c43bb24

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

@echo off

for /d %%i in (tmp\tmp*,tmp\pip*) do rd /s /q "%%i" 2>nul || ("%%i" && exit /b 1) & del /q tmp\tmp* > nul 2>&1 & rd /s /q pip\cache 2>nul

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS= --deepdanboor
set PYTORCH_CUDA_ALLOC_CONF= garbage_collection_threshold:0.6,max_split_size_mb:128

call webui.bat

List of extensions

Console logs

venv "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Commit hash: a9eab236d7e8afa4d6205127904a385b2c43bb24
Installing requirements for Web UI

Launching Web UI with arguments: --deepdanboor
No module 'xformers'. Proceeding without it.
Civitai Helper: Get Custom Model Folder
Civitai Helper: Load setting from: C:\Neural networks\Stable Diffusion\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\setting.json
Civitai Helper: No setting file, use default
Loading weights [c6bbc15e32] from C:\Neural networks\Stable Diffusion\stable-diffusion-webui\models\Stable-diffusion\Paint\sd-v1-5-inpainting.ckpt
Creating model from config: C:\Neural networks\Stable Diffusion\stable-diffusion-webui\configs\v1-inpainting-inference.yaml
LatentInpaintDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
Applying cross attention optimization (Doggettx).
Textual inversion embeddings loaded(14): ChaClo, comdre_1, NNPG, NPGDWLF-1000, NPGDWLF-50, NPGDWLF_02, NPGDWLF_03, NPWLF, WNPG, style-bridal, style-princess, Style-Renaissance, style-widow, YOGA_PANTS_LEGGINGSV2
Textual inversion embeddings skipped(4): style-bridal_sd2, style-hamunaptra_sd2, style-princess_sd2, style-widow_sd2
Model loaded in 11.9s (load weights from disk: 4.0s, create model: 0.7s, apply weights to model: 2.9s, apply half(): 1.1s, move model to device: 1.0s, load textual inversion embeddings: 2.2s).
[tag-editor] Settings has been read from config.json
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 36.2s (import gradio: 2.9s, import ldm: 3.7s, other imports: 2.4s, list extensions: 1.9s, setup codeformer: 0.1s, load scripts: 4.2s, load SD checkpoint: 12.1s, create ui: 8.4s, gradio launch: 0.4s).
Training at rate of 0.005 until step 3500
Preparing dataset...
100%|████████████████████████████████████████████████████████████████████████████████| 148/148 [00:11<00:00, 12.94it/s]
  0%|                                                                                         | 0/3500 [00:00<?, ?it/s]Traceback (most recent call last):
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\modules\textual_inversion\textual_inversion.py", line 489, in train_embedding
    img_c = processing.txt2img_image_conditioning(shared.sd_model, c, training_width, training_height)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\modules\processing.py", line 89, in txt2img_image_conditioning
    image_conditioning = sd_model.get_first_stage_encoding(sd_model.encode_first_stage(image_conditioning))
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 830, in encode_first_stage
    return self.first_stage_model.encode(x)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py", line 83, in encode
    h = self.encoder(x)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 523, in forward
    hs = [self.conv_in(x)]
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 202, in lora_Conv2d_forward
    return lora_forward(self, input, torch.nn.Conv2d_forward_before_lora(self, input))
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Neural networks\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same

Additional information

No response

[torridgristle]

Appears to work if I disable "Move VAE and CLIP to RAM when training if possible. Saves VRAM." after getting this message.

Previously I had used the "Unload SD checkpoint to free VRAM" action and loaded a different model before trying to train textual inversion and getting this error.

[Agnyy]

After disabling this feature. The error is starting to change.

[staff0rd]

I get the error

RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same

When the feature Move VAE and CLIP to RAM when training if possible. Saves VRAM. is already off.