Question on using this in conjunction with CLIP / Open_CLIP / VQGAN

[johndpope]

https://github.com/mlfoundations/open_clip/issues/1

@gabrielilharco suggested this research may help with my problem above. Basically want to introspect an image to shed light on what prompts would be appropriate to recreate a similar image.

[rloganiv]

Hi @johndpope,

This is an interesting idea! No guarantees, but this definitely could work, although you will probably get better prompts with multiple query (not sure this is the right word) images.

I think the best way to use AutoPrompt for your application would be to copy the relevant lines of code to the open_clip training script. Pretty much everything you need is contained in https://github.com/ucinlp/autoprompt/blob/master/autoprompt/create_trigger.py. The main things you'll need are:

• The GradientStorage object that registers the backwards hook to store the gradients of the loss w.r.t. the individual prompt tokens.
• The hotflip_attack function to find the updates.
• Something in your training loop that approximates these lines. Basically the steps are: 1. measure the loss, 2. get candidate prompt modifications, 3. use some additional training data to check which candidate is the best. We found step 3 was necessary to getting our prompts to generalize which is why I recommend having multiple query images, but maybe this isn't needed for your application...IDK.

Hope this helps! I'll leave this issue open for a couple days in case you have any follow up questions.

Best,

@rloganiv

[rloganiv]

Oh and to avoid licensing issues I just licensed this code base under Apache 2.0, so you should be free to copy and alter the code for open_clip however you see fit.

[johndpope]

Thanks Robert, going to be deferring to this repo by Jamie Kiros @dzryk https://github.com/dzryk/clip-grams