RuntimeError when inferencing with the gradio ui

[chnshx]

Hi, thank you for your work and it was very inspiring.

However, I encountered several Runtime errors when running the radio UI, such as

File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/functional.py", line 377, in einsum return _VF.einsum(equation, operands) # type: ignore[attr-defined] RuntimeError: expected scalar type Half but found Float

when there's no prompt.

and, File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/functional.py", line 5318, in multi_head_attention_forward raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.") RuntimeError: The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (4, 4).

when there's a generated final prompt.

Could you please give me some suggestions on how to solve these problems or give more detailed guidance on the workflow of UI? Thank you!

[tellurion-kanata]

Hi chnshx, could you provide the full log?

[chnshx]

Hi chnshx, could you provide the full log?

Sure. The full log is Traceback (most recent call last): File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/queueing.py", line 536, in process_events response = await route_utils.call_process_api( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/route_utils.py", line 322, in call_process_api output = await app.get_blocks().process_api( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/blocks.py", line 1935, in process_api result = await self.call_function( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/blocks.py", line 1520, in call_function prediction = await anyio.to_thread.run_sync( # type: ignore File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread return await future File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 943, in run result = context.run(func, args) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/utils.py", line 826, in wrapper response = f(args, kwargs) File "/home/user/Projects/colorizeDiffusion/gradio_ui.py", line 178, in inference results = model.generate( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/models/basemodel.py", line 504, in generate c, uc = self.prepare_conditions( File "/home/user/Projects/colorizeDiffusion/refnet/models/colorizer.py", line 45, in prepare_conditions emb = local_manipulate( File "/home/user/Projects/colorizeDiffusion/refnet/sampling/manipulation.py", line 67, in local_manipulate controls, targets = clip.encode_text(controls + targets).chunk(2) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/modules/embedder.py", line 57, in encode_text return self.transformer(text, normalize) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/modules/embedder.py", line 283, in forward z = self.encode_with_transformer(tokens.to(self.device), normalize) File "/home/user/Projects/colorizeDiffusion/refnet/modules/embedder.py", line 290, in encode_with_transformer x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask) File "/home/user/Projects/colorizeDiffusion/refnet/modules/embedder.py", line 306, in text_transformer_forward x = r(x, attn_mask=attn_mask) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/open_clip/transformer.py", line 263, in forward x = q_x + self.ls_1(self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask)) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/open_clip/transformer.py", line 250, in attention return self.attn( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1241, in forward attn_output, attn_output_weights = F.multi_head_attention_forward( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/functional.py", line 5318, in multi_head_attention_forward raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.") RuntimeError: The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (4, 4).

[tellurion-kanata]

The opensource code is pretty old and far away from my current implementation. I suppose this issue might be related to the version of "open-clip-torch." I suggest you install the version given in the environment.yml and take another try.

[chnshx]

The opensource code is pretty old and far away from my current implementation. I suppose this issue might be related to the version of "open-clip-torch." I suggest you install the version given in the environment.yml and take another try.

Thank you for your prompt response! This works, but another problem still exists. Traceback (most recent call last): File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/queueing.py", line 536, in process_events response = await route_utils.call_process_api( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/route_utils.py", line 322, in call_process_api output = await app.get_blocks().process_api( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/blocks.py", line 1935, in process_api result = await self.call_function( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/blocks.py", line 1520, in call_function prediction = await anyio.to_thread.run_sync( # type: ignore File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread return await future File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 943, in run result = context.run(func, args) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/gradio/utils.py", line 826, in wrapper response = f(args, kwargs) File "/home/user/Projects/colorizeDiffusion/gradio_ui.py", line 178, in inference results = model.generate( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/models/basemodel.py", line 550, in generate z = self.sample( File "/home/user/Projects/colorizeDiffusion/refnet/models/basemodel.py", line 120, in sample samples = sampler(x, sigmas, extra_args, seed, deterministic, step) File "/home/user/Projects/colorizeDiffusion/refnet/sampling/sampler.py", line 183, in call return self.func(self.sd, x, extra_args=sampler_extra_args, disable=False, extra_params_kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/home/user/Projects/colorizeDiffusion/k_diffusion/sampling.py", line 594, in sample_dpmpp_2m denoised = model(x, sigmas[i] * s_in, extra_args) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/sampling/denoiser.py", line 45, in forward x_out = self.inner_model(x_in, sigma_in, cond=cond).chunk(repeats) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/user/Projects/colorizeDiffusion/k_diffusion/external.py", line 112, in forward eps = self.get_eps(input c_in, self.sigma_to_t(sigma), kwargs) File "/home/user/Projects/colorizeDiffusion/k_diffusion/external.py", line 140, in get_eps return self.inner_model.apply_model(*args, kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/models/basemodel.py", line 438, in apply_model return self.model(x_noisy, t, c_concat=concat, c_crossattn=[crossattn], c_adm=cond.get("c_adm", None)) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/ldm/ddpm.py", line 690, in forward out = self.diffusion_model(x, t, context=cc, concat=c_concat, injects=injects) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/modules/unet.py", line 346, in InferenceForward h = self._forward(x, emb, args, kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/modules/unet.py", line 385, in _forward h = module(h, emb, ccontext, injects) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/modules/unet.py", line 42, in forward x = layer(x, context, inject) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/home/user/Projects/colorizeDiffusion/refnet/modules/attention.py", line 145, in forward x = block(x, context=context[i], inject=inject) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/user/Projects/colorizeDiffusion/ldm/modules/attention.py", line 278, in forward return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint) File "/home/user/Projects/colorizeDiffusion/ldm/modules/diffusionmodules/util.py", line 125, in checkpoint return CheckpointFunction.apply(func, len(inputs), args) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/autograd/function.py", line 539, in apply return super().apply(args, kwargs) # type: ignore[misc] File "/home/user/Projects/colorizeDiffusion/ldm/modules/diffusionmodules/util.py", line 140, in forward output_tensors = ctx.run_function(ctx.input_tensors) File "/home/user/Projects/colorizeDiffusion/ldm/modules/attention.py", line 281, in _forward x = self.attn1( File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/home/user/Projects/colorizeDiffusion/ldm/modules/attention.py", line 196, in forward out = einsum('b i j, b j d -> b i d', sim, v) File "/home/user/anaconda3/envs/hf/lib/python3.9/site-packages/torch/functional.py", line 377, in einsum return _VF.einsum(equation, operands) # type: ignore[attr-defined] RuntimeError: expected scalar type Half but found Float

[tellurion-kanata]

Please install xformers or change the line 196 in ldm/modules/attention.py from out = einsum('b i j, b j d -> b i d', sim, v) to out = einsum('b i j, b j d -> b i d', sim.to(x.dtype), v.to(x.dtype))

[chnshx]

Updating the version of xformer helps! Thank you so much!