[Pipeline] Stable Signature by Meta

[charchit7]

Model/Pipeline/Scheduler description

Stable Signature Pipeline! I think Stable Signature can be a nice add on to the Diffusers pipeline. Stable Signature creates an invisible watermark that can be used to to distinguish when an image is created by an open source generative AI model or not. It's pretty interesting research and I think Diffusers should have it.

How it works! Stable Signature closes the potential for removing the watermark by rooting it in the model with a watermark that can trace back to where the image was created.

Let’s take a look at how this process works with the below chart.

Alice trains a master generative model. Before distributing it, she fine-tunes a small part of the model (called the decoder) to root a given watermark for Bob. This watermark may identify the model version, a company, a user, etc.

Bob receives his version of the model and generates images. The generated images will carry the watermark of Bob. They can be analyzed by Alice or third parties to see if the image was generated by Bob, who used the generative AI model.

We achieve this in a two-step process:

First, two convolutional neural networks are jointly trained. One encodes an image and a random message into a watermark image, while the other extracts the message from an augmented version of the watermark image. The objective is to make the encoded and extracted messages match. After training, only the watermark extractor is retained. Second, the latent decoder of the generative model is fine-tuned to generate images containing a fixed signature. During this fine-tuning, batches of images are encoded, decoded, and optimized to minimize the difference between the extracted message and the target message, as well as to maintain perceptual image quality. This optimization process is fast and effective, requiring only a small batch size and a short time to achieve high-quality results.

[Note] It does not scale to non-latent generative models.

Open source status

• [X] The model implementation is available
• [X] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Official Links! Official Repository : https://github.com/facebookresearch/stable_signature Blog : https://ai.meta.com/blog/stable-signature-watermarking-generative-ai/ Paper : https://arxiv.org/pdf/2303.15435.pdf

[charchit7]

@patrickvonplaten @sayakpaul what do you guys think? Happy to take this if you guys approve.

[patrickvonplaten]

Hey @charchit7, yes that's a good idea - it'd be great to have this added to diffusers? Would you like to give it a try?

[charchit7]

Thanks @patrickvonplaten, yes, would love to do this!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.