Re: Self-similarity hook

[akshaychawla]

Hi @akshaychawla. Can you share any resources for the SelfSimilarityHook which you have in the deepinversion code? In the original code, it was not present, and I am not able to find relevant papers for it.

Originally posted by @animesh-007 in https://github.com/NVlabs/DIODE/issues/6#issuecomment-979319976

[akshaychawla]

@animesh-007 Here is the code for SelfSimilarityHook https://github.com/NVlabs/DIODE/blob/80a396d5772528d4c393a301b0a1390eb7e7e039/deepinversion_yolo.py#L186

There are no relevant papers because its just the cosine distance between vectors 😃 source: https://scikit-learn.org/stable/modules/metrics.html#cosine-similarity

If we can represent inputs (e.g images) as vectors (e.g x \in \mathcal{R}^n}) that contain semantically relevant features for those inputs, cosine distance tells us how close they are on the unit circle i.e the angle between the 2 vectors, don't care about the magnitude.

We tried to use it to reduce the similarity between the batch of images that are being optimized (with very little success), so that each image in the batch has different colors and textures. We saw that a batch of images optimized by DeepInversion on Yolo-V3 had the same blueish-greenish tinge with strokes as if drawn by a paint brush. I hoped that ssim would increase diversity, but it didn't help too much.

Interestingly, batch similarity is being heavily used by papers for learning backbones with contrastive learning without any labels! e.g https://arxiv.org/abs/2002.05709 uses self similarity between samples in a batch to pull together similar images and pull apart dissimilar images. For generative modelling, you may want to look at GAN papers that might be using self similarity to boost diversity.

[animesh-007]

@animesh-007 Here is the code for SelfSimilarityHook

https://github.com/NVlabs/DIODE/blob/80a396d5772528d4c393a301b0a1390eb7e7e039/deepinversion_yolo.py#L186

There are no relevant papers because its just the cosine distance between vectors 😃 source: https://scikit-learn.org/stable/modules/metrics.html#cosine-similarity

If we can represent inputs (e.g images) as vectors (e.g x \in \mathcal{R}^n}) that contain semantically relevant features for those inputs, cosine distance tells us how close they are on the unit circle i.e the angle between the 2 vectors, don't care about the magnitude.

We tried to use it to reduce the similarity between the batch of images that are being optimized (with very little success), so that each image in the batch has different colors and textures. We saw that a batch of images optimized by DeepInversion on Yolo-V3 had the same blueish-greenish tinge with strokes as if drawn by a paint brush. I hoped that ssim would increase diversity, but it didn't help too much.

Interestingly, batch similarity is being heavily used by papers for learning backbones with contrastive learning without any labels! e.g https://arxiv.org/abs/2002.05709 uses self similarity between samples in a batch to pull together similar images and pull apart dissimilar images. For generative modelling, you may want to look at GAN papers that might be using self similarity to boost diversity.

Thanks, @akshaychawla for the detailed answer. I have looked at one GAN paper ref: https://arxiv.org/abs/2105.08584 in the past that uses contrastive learning too.

[akshaychawla]

@animesh-007 I am closing the issue since it seems to be resolved. If required, please re-open for further discussion on the same topic, else open a new issue.