How change the denoising strength ?

[al3dv2]

Hi how I can change the denoising strength with vid2vid ? I thinks it's to high by default, thanks

[al3dv2]

Please helps ! tks

[WyattAutomation]

Also interested in how to do this. Where is it initially even set?

[WyattAutomation]

As an update, if you set 'use_lcm_lora=True' when StreamDiffusionWrapper() is instantiated, it uses "scheduling_lcm.py" from the diffusers library and you can change the denoising strength (via the lcm_scheduler at least) without any fancy tricks right here: https://github.com/huggingface/diffusers/blob/087daee2f05c1d346719880b5bcb7282c96dac77/src/diffusers/schedulers/scheduling_lcm.py#L357

This seems to behave like denoising strength set on any model is expected to behave, when using a normal diffusers AutoPipeline...()

..just set the strength inside of the "prepare" method in pipeline.py from the src folder of Stream Diffusion, by setting "use_lcm_lora = True" on the StreamDiffusionWrapper() instance and then changing this line here to pass in a "strength" param.

so change this:

self.scheduler.set_timesteps(num_inference_steps, self.device)

to this:

self.scheduler.set_timesteps(num_inference_steps, self.device, strength=strength)

right here: https://github.com/cumulo-autumn/StreamDiffusion/blob/8ff959a3ae6536f33d6f0fbf809e9b620e346978/src/streamdiffusion/pipeline.py#L182

..with "strength" needing to just be a float value (between ~1.3 - 2.0 seems to work).

To configure the strength when instantiating a StreamDiffusionWrapper from wrapper.py instead of just hacking directly it into pipeline.py, you need to add an arg like "strength: float = 1.5," to the "prepare" method in wrapper.py like this:

    def prepare(
        self,
        prompt: str,
        negative_prompt: str = "",
        num_inference_steps: int = 50,
        guidance_scale: float = 1.2,
        delta: float = 1.0,
        strength: float = 1.5,
    ) -> None:

(this method is found here): https://github.com/cumulo-autumn/StreamDiffusion/blob/8ff959a3ae6536f33d6f0fbf809e9b620e346978/utils/wrapper.py#L176

also, do the same thing for the "prepare" method in pipeline.py here too (add strength arg): https://github.com/cumulo-autumn/StreamDiffusion/blob/8ff959a3ae6536f33d6f0fbf809e9b620e346978/src/streamdiffusion/pipeline.py#L124

Then you should have it able to be used when making an instance of StreamDiffusionWrapper() like the rest of the settings when the class is instantiated.

Main drawback is that this doesn't let you change it while it's running. Have to set it before inference, idk if there is a way to change it while it's running, another method that uses it downstream doesn't expose the option to change it once inference is going.

[WyattAutomation]

This is a sloppy example class that can be called and instantiated from another app, using "generate()" to pass a prompt and image to it for generation, and using the hack above to get at least some control of denoising strength. Note that "streamdiff_utils" is a custom folder I created in the same directory of my app, with a copy of wrapper.py from the repo, with streamdiffusion installed in my environment's site-packages:

import os
import sys

from typing import Literal, Dict, Optional

sys.path.append(os.path.join(os.path.dirname(__file__), "..", ".."))

from streamdiff_utils.wrapper import StreamDiffusionWrapper

CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))

from diffusers import LCMScheduler

class DiffusionGenerator:
    def __init__(self, init_prompt="", guidance_scale=1.75, strength=1.6):
        self.model_id_or_path: str = "SimianLuo/LCM_Dreamshaper_v7"
        self.taesd_model: str = "madebyollin/taesd"
        self.lora_dict: Optional[Dict[str, float]] = None
        self.prompt: str = init_prompt
        self.negative_prompt: str = "low quality, bad quality, blurry, low resolution"
        self.width: int = 512
        self.height: int = 512
        self.acceleration: Literal["none", "xformers", "tensorrt"] = "tensorrt"
        self.use_denoising_batch: bool = True
        self.guidance_scale = float(guidance_scale)
        self.strength = float(strength)
        self.cfg_type: Literal["none", "full", "self", "initialize"] = "none"
        self.seed: int = 123456
        self.delta: float = 0.5

        if self.guidance_scale <= 1.0:
            cfg_type = "none"

        self.stream = StreamDiffusionWrapper(
            model_id_or_path=self.model_id_or_path,
            use_tiny_vae=self.taesd_model,
            t_index_list=[35, 45],
            frame_buffer_size=1,
            width=self.width,
            height=self.height,
            use_lcm_lora=True,
            output_type="pil",
            warmup=10,
            acceleration=self.acceleration,
            do_add_noise=False,
            mode="img2img",
            use_denoising_batch=self.use_denoising_batch,
            cfg_type=self.cfg_type,
            seed=self.seed,
        )

        self.stream.prepare(
            prompt=self.prompt,
            negative_prompt=self.negative_prompt,
            num_inference_steps=50,
            guidance_scale=self.guidance_scale,
            delta=self.delta,
            strength=self.strength,
        )

    def generate(self, image, prompt=None):
        output_image = self.stream(image=image, prompt=prompt)
        return output_image        

[zjysteven]

Shouldn't the strength be within [0.0, 1.0]? https://github.com/huggingface/diffusers/blob/2523390c269a37209fbb96359d494369aff89361/src/diffusers/pipelines/controlnet/pipeline_controlnet_img2img.py#L946-L951

[happyFish]

When I pass any value other than 1.0 as strength into self.scheduler.set_timesteps I get an error:

in prepare
    alpha_prod_t_sqrt = self.scheduler.alphas_cumprod[timestep].sqrt()
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
IndexError: index 1499 is out of bounds for dimension 0 with size 1000

Any one else tried and experience this error?