Haoming02 / sd-webui-vectorscope-cc

An Extension for Automatic1111 Webui that performs Offset Noise* natively
MIT License
165 stars 7 forks source link
stable-diffusion-webui stable-diffusion-webui-plugin

# SD Webui Vectorscope CC This is an Extension for the Automatic1111 Webui, which performs a kind of Offset Noise natively during inference, allowing you to adjust the brightness, contrast, and color of the generations.

Also supports both old & new Forge

Example Images


Base Image w/o Extension

Infotext - **Checkpoint:** [realisticVisionV51](https://civitai.com/models/4201?modelVersionId=130072) - **Positive Prompt:** `(high quality, best quality), a 4k cinematic photo of a gentleman in suit, street in a city at night, (depth of field, bokeh)` - **Negative Prompt:** `(low quality, worst quality:1.2), [EasyNegative, EasyNegativeV2]` ```cpp Steps: 32, Sampler: DPM++ 2M Karras, CFG scale: 7.5, Seed: 3709157017, Size: 512x512, Denoising strength: 0.5 Clip skip: 2, Token merging ratio: 0.2, Token merging ratio hr: 0.2, RNG: CPU, NGMS: 4 Hires upscale: 2, Hires steps: 16, Hires upscaler: 2xNomosUni_esrgan_multijpg ```
Vibrant Cold "Movie when Mexico"
  • Alt: True
  • Saturation: 1.75
  • Noise: Ones
  • Scaling: 1 - Cos
  • Brightness: -5.0
  • Contrast: 2.5
  • Saturation: 0.75
  • R: -3.0
  • B: 3.0
  • Noise: Ones
  • Scaling: 1 - Sin
  • Brightness: 2.5
  • Contrast: -2.5
  • Saturation: 1.25
  • R: 1.5
  • G: 3.0
  • B: -4.0
  • Noise: Ones
  • Scaling: 1 - Sin

How to Use

Note: Since this Extension modifies the underlying latent tensor, the composition may change drastically depending on the parameters

Basic Parameters

Color Channels

Channel Lower Higher
R Cyan Red
G Magenta Green
B Yellow Blue

The color picker isn't 100% accurate due to multiple layers of conversions...

Style Presets

You can also find pre-made Styles by the community available online
  • eg. The Photomatix Styles (right click on the link, click Save link as, then save the .json file into the sd-webui-vectorscope-cc extension folder)

Advanced Parameters

Noise Settings

let x denote the latent Tensor ; let y denote the operations

Infotext - **Checkpoint:** [realisticVisionV51](https://civitai.com/models/4201?modelVersionId=130072) - **Positive Prompt:** `(high quality, best quality), a 4k photo of a cute dog running in the snow, mountains, day, (depth of field, bokeh)` - **Negative Prompt:** `(low quality, worst quality:1.2), [EasyNegative, EasyNegativeV2]` - **Brightness:** `2.5` - **Contrast:** `2.5` - **Alt:** `True` - **Scaling:** `1 - Cos` ```cpp Steps: 24, Sampler: DPM++ 2M Karras, CFG scale: 7.5, Seed: 1257068736, Size: 512x512, Denoising strength: 0.5 Clip skip: 2, Token merging ratio: 0.2, Token merging ratio hr: 0.2, RNG: CPU, NGMS: 4 Hires upscale: 1.5, Hires steps: 16, Hires upscaler: 2xNomosUni_esrgan_multijpg ```

Scaling Settings

By default, this Extension offsets the noise by the same amount every step. But depending on the Sampler and Scheduler used, and whether Alt. was enabled or not, the effects might be too strong during the early or the later phase of the process, which in turn causes artifacts.

Infotext - **Checkpoint:** [realisticVisionV51](https://civitai.com/models/4201?modelVersionId=130072) - **Positive Prompt:** `(high quality, best quality), a 4k photo of a cute cat standing at a flower field in a park, day, (depth of field, bokeh)` - **Negative Prompt:** `(low quality, worst quality:1.2), [EasyNegative, EasyNegativeV2]` - **Alt:** `True` - **Noise:** `Straight Abs.` ```cpp Steps: 24, Sampler: DPM++ 2M Karras, CFG scale: 7.5, Seed: 3515074713, Size: 512x512, Denoising strength: 0.5 Clip skip: 2, Token merging ratio: 0.2, Token merging ratio hr: 0.2, RNG: CPU, NGMS: 4 Hires upscale: 1.5, Hires steps: 12, Hires upscaler: 2xNomosUni_esrgan_multijpg ```

Buttons

Roadmap

API

You can use this Extension via API by adding an entry to the alwayson_scripts of your payload. An example is provided. The args are sent in the following order in an array:

Parameter Type
Enable bool
Alt. bool
Brightness float
Contrast float
Saturation float
R float
G float
B float
Hires. fix bool
ADetailer bool
Randomize bool
Noise Method str
Scaling str

Known Issues

HDR

Discussion Thread

In the Script Dropdown at the bottom, there is now a new High Dynamic Range option:

Settings


Offset Noise TL;DR The most common *version* of **Offset Noise** you may have heard of is from this [blog post](https://www.crosslabs.org/blog/diffusion-with-offset-noise), where it was discovered that the noise functions used during **training** were flawed, causing `Stable Diffusion` to always generate images with an average of `0.5` *(**ie.** grey)*. > **ie.** Even if you prompt for dark/night or bright/snow, the average of the image is still "grey" > [Technical Explanations](https://youtu.be/cVxQmbf3q7Q) However, this Extension instead tries to offset the latent noise during the **inference** phase. Therefore, you do not need to use models that were specially trained, as this can work on any model.
How does this work? After reading through and messing around with the code, I found out that it is possible to directly modify the Tensors representing the latent noise used by the Stable Diffusion process. The dimensions of the Tensors is `(X, 4, H / 8, W / 8)`, which represents **X** batch of noise images, with **4** channels, each with **(W / 8) x (H / 8)** values > **eg.** Generating a single 512x768 image will create a Tensor of size (1, 4, 96, 64) Then, I tried to play around with the values of each channel and ended up discovering these relationships. Essentially, the 4 channels correspond to the **CMYK** color format for `SD1` *(**Y'CbCr** for `SDXL`)*, hence why you can control the brightness as well as the colors.

Vectorscope?

The Extension is named this way because the color interactions remind me of the Vectorscope found in Premiere Pro's Lumetri Color. Those who are experienced in Color Correction should be rather familiar with this Extension.

Yes. I'm aware that it's just how digital colors work in general.

We've come full circle (*ba dum tss) now that a Color Wheel is actually added.