Question for the encoder_hidden_states

[WayneML]

When I try to run the script, I found the encoder_hidden_states to be zero.

[WayneML]

if args.conditioning_dropout_prob is not None: random_p = torch.rand( bsz, device=latents.device, generator=generator)

Sample masks for the edit prompts.

                prompt_mask = random_p < 2 * args.conditioning_dropout_prob
                prompt_mask = prompt_mask.reshape(bsz, 1, 1)
                # Final text conditioning.
                null_conditioning = torch.zeros_like(encoder_hidden_states)
                encoder_hidden_states = torch.where(
                    prompt_mask, null_conditioning.unsqueeze(1), encoder_hidden_states.unsqueeze(1))

I found something strange in this code block，it seems that “random_p = torch.ran(bsz, device=latents.device, generator=generator)” always make random_p is one dimension and the value is 1.when you chose batch size is 1. make prompt_mask one ture but not a list of Boolean type. prompt_mask = random_p < 2 * args.conditioning_dropout_prob prompt_mask = prompt_mask.reshape(bsz, 1, 1)

Final text conditioning.

                null_conditioning = torch.zeros_like(encoder_hidden_states)
                encoder_hidden_states = torch.where(
                    prompt_mask, null_conditioning.unsqueeze(1), encoder_hidden_states.unsqueeze(1))

[WayneML]

And is this still for image2video task? It seems that it is used for the text to image.

[pixeli99]

Hi, I didn't quite understand what you meant. Are you asking why the encoder_hidden_states need to be replaced with zeros?

[mmxbc1223]

Can the encoder_hidden_states be replaced with a text embedding for text-to-video tasks？