KSampler Variation nodes have issues with Cascade and batches

[clayne]

Noticed when using the variation based ksamplers with cascade stage b and batches that generations get progressively more corrupted. I was able to isolate it to the use of multiplying variation_weight by batch index as being the source of the problem. The following change "fixes" the issue, at the expense of changing behavior:

index 6dba799..0b93303 100644
--- a/noise_context.py
+++ b/noise_context.py
@@ -9,28 +9,28 @@ class NoiseContext():
         self.variation_weight = variation_weight    

     def get_mixed_noise_function(cls, variation_seed, variation_weight):
         def prepare_mixed_noise(latent_image:torch.Tensor, seed, batch_inds):
             single_image_latent = latent_image[0].unsqueeze_(0)
             different_noise = NoiseContext.original_noise_function(single_image_latent, variation_seed, batch_inds)
             original_noise = NoiseContext.original_noise_function(single_image_latent, seed, batch_inds)
             if latent_image.shape[0]==1:
                 mixed_noise = original_noise * (1.0-variation_weight) + different_noise * (variation_weight)
             else:
                 mixed_noise = torch.empty_like(latent_image)
                 for i in range(latent_image.shape[0]):
-                    mixed_noise[i] = original_noise * (1.0-variation_weight*i) + different_noise * (variation_weight*i)
+                    mixed_noise[i] = original_noise * (1.0-variation_weight) + different_noise * (variation_weight)
             return mixed_noise
         return prepare_mixed_noise

     def hijack(self):
         comfy.sample.prepare_noise = self.get_mixed_noise_function(self.variation_seed, self.variation_weight)

     def unhijack(self):
         comfy.sample.prepare_noise = NoiseContext.original_noise_function

     def __enter__(self):
         self.hijack()

     def __exit__(self, exc_type, exc_value, exc_traceback):
         self.unhijack()

Without the change:

With the change:

I was on the fence about reporting this because it somewhat feels like a "well just don't do that" situation, but figured I'd throw it out there. The way the cascade stage B sampler works is different than SD15 and SDXL and perhaps perturbing noise like this just simply isn't compatible with stage B.

In stage B, the four channels of tokens from Stage A are passed to a latent diffusion model where the tokens are turned into floating point values and then de-noised as in Stable Diffusion. At the same time, a neural network called a Semantic Compressor transforms the image data into a 24x24 “semantic representation” having 1280 channels. This information is compressed even further using convolution to have just 16 channels. The semantic information about the image is then combined with the conditioning from the text prompt and this is used to control the diffusion model. In other words, Stage B is a bit like combining an IP Adapter with a diffusion model, using the source image to guide diffusion to produce a similar image at the output. The training objective for Stage B is to have the diffusion model generate denoised latents that match the latents at the input. No conversion back to image space is done in Stage B, which greatly reduces the cost of training this stage.

Decent overview of SC: https://www.reddit.com/r/comfyui/comments/1auwepl/understanding_stable_cascade/

Note: cubiq's variation related nodes from the essentials pack also appear to be affected by this, namely KSampler Stochastic Variations, whereas KSampler Variations with Noise Injection is immune.

[clayne]

Sorry, looks like I spoke too soon about Stage B, it appears to affect Stage C as well. Basically any usage of a variation-based KSampler seems to cause problems with Cascade:

It's a bit surprising to me because I figured keeping it to Stage C only would be fine but apparently not.

[chrisgoringe]

I've retired this node pack. Use https://github.com/chrisgoringe/cg-noisetools