lucidrains
commented
3 years ago @CDitzel Hey Carsten! Glad you are reading the repository and double checking my work!
So I think the best way to make clear is to work through a concrete example. Let's say each text token is just a character, and let's pretend numbers are visual tokens.
say we are working with a multi modal sequence
[c] [a] [t] [4] [1] [2] [3]what we do during training is first append an EOS
[c] [a] [t] [4] [1] [2] [3] [eos]then, for autoregressive, we break this sequence from the range [:-1] to predict [1:]
[a] [t] [4] [1] [2] [3] [eos]
<bottom tokens predict top>
[c] [a] [t] [4] [1] [2] [3]you can see, since the last text token predicts the start of the first visual token, that is why in the logits mask, you see that it is off by one
T mask logit for only text tokens
I mask logit for only image tokens
[T] [T] [I] [I] [I] [I] [I]
[a] [t] [4] [1] [2] [3] [eos]
<bottom tokens predict top>
[c] [a] [t] [4] [1] [2] [3]As for your second question, we are always trying to predict the next token. Even without the masking, the attention net will eventually figure it out. This just makes it a bit cleaner, and makes sure during sampling we don't hit the wrong token.
CDitzel
snoop2head
Been trying to get a grasp of the DALLE code recently. However, there are a couple of things, I cant quite wrap my head around and since the paper is not published yet, I was wondering, if we can maybe clarify them here.
So there is the VAE training which basically features the codebook in the bottleneck and is trained a priori.
Next, Dalle receives text and image pairs, embeds them and adds positional encodings individually to both modalities. However, the image data is not embedded like e.g. in ViT but by downsampling it via the Encoder of the VAE (without accumulating gradients), argmax search within the feature dimension across the downsampled image patches and finally indexing into the previously trained codebook.
The resulting representations of both modalities are then concatenated along the token dimension. And while every word of the text input is one token, the height and width of the VAE-encoded image yields the number of image tokens.
The combined embedding is then passed into a single transformer which calculates self-attention not only intra-modal but also across both modalities if I am not mistaken.
A masking of the form
mask = torch.ones_like(text).bool()results in unmasked attention calculation, right?
A final Mlp maps the transformer output to all potential token possibilities (both text and image).
Then I dont understand the masking
shouldnt there be one more row concerned with the text input and one less row for the image input?
For the following config with 3 text input tokens
the mask looks like this
shouldt it be?
The purpose of the masking is so that image tokens dont contribute to the predictions of text and vice versa. The code proceeds by constructing labels from the text integer tokens and the VAE image embedding pixels by using the codebook indices.
But what is it we are actually trying to predict with this classification task here? It is a 2d CrossEntropyLoss where for each token (either text or image) we are trying to predict ... exactly what? Some I am missing the intuition here I guess...
And then, why is the label vector neglecting the very first label entry but using the EOS enty?
Maybe someone can help me (and others) in understanding better whats going on here. Thank you in advance