Unexpected keyword argument 'scale'

[tin2tin]

Running the example, I get this error:

File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py", line 702, in __call__
    noise_pred = self.unet(
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\diffusers\models\unet_2d_condition.py", line 1029, in forward
    sample = upsample_block(
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: register_free_upblock2d.<locals>.up_forward.<locals>.forward() got an unexpected keyword argument 'scale'

I'm on the latest alpha of Diffusers and Win 11.

[lyn-rgb]

In the latest version, the upblock2d has an argument scale, https://github.com/huggingface/diffusers/blob/v0.21.2-patch/src/diffusers/models/unet_2d_blocks.py line 2292. I use the version of 0.19.3. You can modify the free_lunch_utils.py like below

def register_free_upblock2d(model, b1=1.2, b2=1.4, s1=0.9, s2=0.2):
    def up_forward(self):
        def forward(hidden_states, res_hidden_states_tuple, temb=None, upsample_size=None, scale: float = 1.0):
            for resnet in self.resnets:
                # pop res hidden states
                #print(f"in free upblock2d, hidden states shape: {hidden_states.shape}")
                res_hidden_states = res_hidden_states_tuple[-1]
                res_hidden_states_tuple = res_hidden_states_tuple[:-1]

                # --------------- FreeU code -----------------------
                # Only operate on the first two stages
                if hidden_states.shape[1] == 1280:
                    hidden_states[:,:640] = hidden_states[:,:640] * self.b1
                    res_hidden_states = Fourier_filter(res_hidden_states, threshold=1, scale=self.s1)
                if hidden_states.shape[1] == 640:
                    hidden_states[:,:320] = hidden_states[:,:320] * self.b2
                    res_hidden_states = Fourier_filter(res_hidden_states, threshold=1, scale=self.s2)
                # ---------------------------------------------------------

                hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)

                if self.training and self.gradient_checkpointing:

                    def create_custom_forward(module):
                        def custom_forward(*inputs):
                            return module(*inputs)

                        return custom_forward

                    if is_torch_version(">=", "1.11.0"):
                        hidden_states = torch.utils.checkpoint.checkpoint(
                            create_custom_forward(resnet), hidden_states, temb, use_reentrant=False
                        )
                    else:
                        hidden_states = torch.utils.checkpoint.checkpoint(
                            create_custom_forward(resnet), hidden_states, temb
                        )
                else:
                    hidden_states = resnet(hidden_states, temb, scale=scale)

            if self.upsamplers is not None:
                for upsampler in self.upsamplers:
                    hidden_states = upsampler(hidden_states, upsample_size, scale=scale)

            return hidden_states

        return forward

    for i, upsample_block in enumerate(model.unet.up_blocks):
        if isinstance_str(upsample_block, "UpBlock2D"):
            upsample_block.forward = up_forward(upsample_block)
            setattr(upsample_block, 'b1', b1)
            setattr(upsample_block, 'b2', b2)
            setattr(upsample_block, 's1', s1)
            setattr(upsample_block, 's2', s2)

[lyn-rgb]

I just created a new branch for diffusers-v0.21.2 support that you can try. I haven't tested it though.

[putuoka]

I just created a new branch for diffusers-v0.21.2 support that you can try. I haven't tested it though.

confirm working but the image result still bad https://colab.research.google.com/drive/1A4zo-D-3U-c5lLdk-dc3TMc_4wJp5RET?usp=sharing

how to use with sdxl?

[tin2tin]

Your suggested change here, made it work. Thank you! I've implemented your Free Lunch solution in Pallaidium, so soon the users will be able to test it there. Thank you!

[tin2tin]

Thank you. I posted some results here: https://github.com/ChenyangSi/FreeU/issues/14

[lyn-rgb]

I just created a new branch for diffusers-v0.21.2 support that you can try. I haven't tested it though.

confirm working but the image result still bad https://colab.research.google.com/drive/1A4zo-D-3U-c5lLdk-dc3TMc_4wJp5RET?usp=sharing

how to use with sdxl?

You can use this code in sd xl directly in the same way as sd 1.5 and add some negative words to improve the generation quality. @putuoka

[tin2tin]

Tweaking the values a bit, looking at this: https://wandb.ai/nasirk24/UNET-FreeU-SDXL/reports/FreeU-SDXL-Optimal-Parameters--Vmlldzo1NDg4NTUw?accessToken=6745kr9rjd6e9yjevkr9bpd2lm6dpn6j00428gz5l60jrhl3gj4gubrz4aepupda

This was the original values:

Using: register_free_upblock2d(pipe, b1=0.6, b2=0.4, s1=1.1, s2=1.2)

[lyn-rgb]

e values a bit, looking at t

It looks like you got the values ​​of b and s reversed. should be s1 = 0.6 s2 = 0.4 b1 = 1.1 b2 = 1.2

[tin2tin]

Ah, that made a difference. Now using a refiner is not necessary at all:

After spending the evening on trying to reproduce this, I'm affraid I may have confused the processing with img2img SDXL processing instead of FreeU. Sorry.

[putuoka]

guys i still didn't understand why the result getting worse with freeu?

generator = torch.Generator(device="cuda").manual_seed(46579129)

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt,generator=generator,).images[0]

original image without freeu block registration

image with freeu block registration enable

[xinli2008]

@lyn-rgb thank you for your contribution! It also works on 0.18.2 Diffusers version