search

nerdyrodent / VQGAN-CLIP

Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.
Other
2.59k stars 428 forks source link

Sample probability #119

Open abrichr opened 2 years ago

abrichr commented 2 years ago

Given a new image, we would like to determine the probability of an image given the trained model parameters (e.g. as described in http://proceedings.mlr.press/v97/balaji19a/balaji19a.pdf).

Creating a new cog endpoint in predict.py is an obvious starting point. Any suggestions on where to go from there would be greatly appreciated!

A surrogate approach might be to simply sample from the discriminator. If the discriminator estimates the probability that a sample came from the data distribution rather than model distribution, and for a particular image that probability is low, then the image has low probability given the training data. Assuming the GAN has been trained to generate images from the training data, does it follow that the image has low probability given the model parameters?

So instead of determining the probability of an image given the trained model parameters, we are determining the probability of the image given the training data. Is that right?

abrichr commented 2 years ago

Similarly, how can we use the text that was used to generate an image to calculate the image's probability?