RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

[davidlapetina]

Hi,

With the latest version (14.0.3) when doing video generation I got the following error on my Linux server: "RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half"

Any idea what could cause the issue? Thanks.

Here is a full stack. Generating a 14 frame video from outputs/generated/ghost.png. Device:cuda seed:941192 Loaded weights from /home/david/.cache/huggingface/hub/models--imaginairy--stable-video-diffusion/snapshots/f9dce2757a0713da6262f35438050357c2be7ee6/svd.fp16.safetensors with 0 missing and 0 unexpected keys EulerEDMSampler: 0%| | 0/26 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/david/.local/lib/python3.10/site-packages/imaginairy/cli/clickshellmod.py", line 30, in invoke command.main( File "/home/david/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/david/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/david/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, kwargs) File "/home/david/.local/lib/python3.10/site-packages/imaginairy/cli/videogen.py", line 80, in videogen_cmd generate_video( File "/home/david/.local/lib/python3.10/site-packages/imaginairy/api/video_sample.py", line 241, in generate_video samples_z = model.sampler(denoiser, randn, cond=c, uc=uc) File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/sampling.py", line 126, in call x = self.sampler_step( File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/sampling.py", line 105, in sampler_step denoised = self.denoise(x, denoiser, sigma_hat, cond, uc) File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/sampling.py", line 61, in denoise denoised = denoiser(self.guider.prepare_inputs(x, sigma, cond, uc)) File "/home/david/.local/lib/python3.10/site-packages/imaginairy/api/video_sample.py", line 235, in denoiser return model.denoiser( File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/denoiser.py", line 42, in forward network(input_tensor * c_in, c_noise, cond, additional_model_inputs) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/wrappers.py", line 30, in forward return self.diffusion_model( File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/david/.local/lib/python3.10/site-packages/imaginairy/modules/sgm/diffusionmodules/video_model.py", line 476, in forward emb = self.time_embed(t_emb) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward input = module(input) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/david/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

[brycedrennan]

Fixed in just released version 14.0.4

[davidlapetina]

Thank you so much!

[davidlapetina]

Hi,

The issue is confirmed fixed on Linux with GPU (CUDA) but when using my Mac with M1 proc (hence using the CPU) "Apple Silicon MPS (M1, M2, etc) is not currently supported for video generation. Switching to cpu generation."

I still have the same issue "RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half" return forward_call(*args, **kwargs) File "/opt/homebrew/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

I tried on my Mac because I just have 8Gb of Memory in my GPU and it seems it is not enough for video generation.

[brycedrennan]

Thanks for the update, I should probably just disable cpu generation as well then.

The issue is that some parameters are in float32 and some in float16. CUDA has an option to paper over issues when numbers are in different formats. The fix was turning that option back on. For CPU and MPS (eventually) it will be necessary to hunt down why some parameters are in a different type and convert them.

I dont think I'm going to have time in the near future to do that.