sdxl_controlnet_agent

[cuhkfz]

Thanks for publishing the source, it is an interesting work.

I notice that in 'controller/agent', both sd_controlnet_agent and sdxl_controlnet_agent are provided. I tried to use sdxl_controlnet_agent by modifying the eval_genima.yaml file, but meet the following error:

File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py", line 1488, in call down_block_res_samples, mid_block_res_sample = self.controlnet( File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/diffusers/models/controlnet.py", line 807, in forward sample, res_samples = downsample_block( File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/diffusers/models/unets/unet_2d_blocks.py", line 1288, in forward hidden_states = attn( File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(args, kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/diffusers/models/transformers/transformer_2d.py", line 440, in forward hidden_states = block( File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/diffusers/models/attention.py", line 490, in forward attn_output = self.attn2( File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/diffusers/models/attention_processor.py", line 559, in forward return self.processor( File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/diffusers/models/attention_processor.py", line 1379, in call key = attn.to_k(encoder_hidden_states) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(args, kwargs) File "/home/user/.conda/envs/genima_env/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 116, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x2048 and 1024x320) QObject::~QObject: Timers cannot be stopped from another thread QMutex: destroying locked mutex

Could you please give me any suggestions about this error?

Thanks a lot.

[MohitShridhar]

@cuhkfz, the pre-trained checkpoint is for sd-turbo, so it won't work with sdxl-turbo.

The scripts inside genima/diffusion are for fine-tuning sdxl-turbo yourself.

[cuhkfz]

I see, thanks