hnmr293
commented
1 year ago For example, use the following prompt and target tokens. For clarity, I've also added the token number to the prompt.
prompt: a cute girl, blue eyes, red shirt
token idx: 1 2~~~ 3~~~4 5~~~ 6~~~7 8~~ 9~~~~
targets: blue, redNB. The comma is an independent token.
For this prompt and target tokens, Cutoff (this extension) generates the three prompts shown below.
| token idx $t$ | token | applied prompt | prompt id $p$ |
|---|---|---|---|
| 0 | <\|startoftext\|> |
a cute girl, blue eyes, red shirt |
0 |
| 1 | a |
a cute girl, blue eyes, red shirt |
0 |
| 2 | cute |
a cute girl, blue eyes, red shirt |
0 |
| 3 | girl |
a cute girl, blue eyes, red shirt |
0 |
| 4 | , |
a cute girl, blue eyes, red shirt |
0 |
| 5 | blue |
a cute girl, _ eyes, red shirt |
1 |
| 6 | eyes |
a cute girl, _ eyes, red shirt |
1 |
| 7 | , |
a cute girl, blue eyes, red shirt |
0 |
| 8 | red |
a cute girl, blue eyes, _ shirt |
2 |
| 9 | shirt |
a cute girl, blue eyes, _ shirt |
2 |
| 10 | <\|endoftext\|> |
a cute girl, blue eyes, red shirt |
0 |
CLIP converts these three prompts into three corresponding vectors (each a 77x768 matrix):
| prompt id $p$ | 77 vectors of 768 dimensions |
|---|---|
| 0 | $( \quad v^0_0 \quad v^0_1 \quad v^02 \quad ... \quad v^0{76} \quad )$ |
| 1 | $( \quad v^1_0 \quad v^1_1 \quad v^12 \quad ... \quad v^1{76} \quad )$ |
| 2 | $( \quad v^2_0 \quad v^2_1 \quad v^22 \quad ... \quad v^2{76} \quad )$ |
where $v^p_t$ is a 768-dimensional vector which represents the $t$-th token of the prompt $p$.
For instance, $v^0_1, v^1_1, v^2_1$ are the vectors corresponding to the token a in prompts $p=0,1,2$.
Likewise, $v^0_2, v^1_2, v^2_2$ are the vectors corresponding to the token cute in prompts $p=0,1,2$.
The same applies to other $v^p_t$.
From the above, we have a vector to be applied to each token.
| token idx $t$ | token | applied prompt | prompt id $p$ | vector to be applied |
|---|---|---|---|---|
| 0 | <\|startoftext\|> |
a cute girl, blue eyes, red shirt |
0 | $v^0_0$ |
| 1 | a |
a cute girl, blue eyes, red shirt |
0 | $v^0_1$ |
| 2 | cute |
a cute girl, blue eyes, red shirt |
0 | $v^0_2$ |
| 3 | girl |
a cute girl, blue eyes, red shirt |
0 | $v^0_3$ |
| 4 | , |
a cute girl, blue eyes, red shirt |
0 | $v^0_4$ |
| 5 | blue |
a cute girl, _ eyes, red shirt |
1 | $v^1_5$ |
| 6 | eyes |
a cute girl, _ eyes, red shirt |
1 | $v^1_6$ |
| 7 | , |
a cute girl, blue eyes, red shirt |
0 | $v^0_7$ |
| 8 | red |
a cute girl, blue eyes, _ shirt |
2 | $v^2_8$ |
| 9 | shirt |
a cute girl, blue eyes, _ shirt |
2 | $v^2_9$ |
| 10 | <\|endoftext\|> |
a cute girl, blue eyes, red shirt |
0 | $v^0_{10}$ |
The embedding to be applied is: $$( \quad v^0_0 \quad v^0_1 \quad v^0_2 \quad v^0_3 \quad v^0_4 \quad v^1_5 \quad v^1_6 \quad v^0_7 \quad v^2_8 \quad v^29 \quad v^0{10} \quad v^0{11} \quad ... \quad v^0{76} \quad )$$
Sorry for my long text. I used the word "gather" with the intention of the action of picking one specific vector for each $t$ from the $p$ prompts in this way.
asheroin
marson666
NormXU
juxuan27
It would be nice if could give more details about the gather embeddings ? From the image from README.md, it seems that just simple concat the clip embeddings but still got an shape of 768*77, just same as the output from hide all tokens's.