kamenbliznashki
commented
4 years ago Hi -
I would read the papers that the code is based on for a reference on autoregressive (AR) models. Also Berkeley has a great class on deep unsupervised learning where you can study the section on AR models — https://sites.google.com/view/berkeley-cs294-158-sp20/home.
In short - take an image of dimension d. The probability of that image is given by the joint probability of all of its pixels p(x_1,...,x_d) where x_i is a pixel (eg for Mnist, an image is 28x28 so d = 2828 = 784; so x_1 is the first top left pixel and so on). An AR model breaks down this joint probability autoregressively so p(x_1,...,x_d) = p(x_1)p(x_2|x_1)...p(x_d|x_d-1,...,x_2,x_1). That means at each pixel the model is only allowed to see the past dims ie the context and you have to block the model from seeing future pixels. This is what the mask achieves.
On Apr 2, 2020, at 08:25, LonglongaaaGo notifications@github.com wrote:
Hi! thank you for your good project! i think your projectis very helpful and beautiful ! Now i am confused by the "attn_mask " in pixelSnail. Besides, i have awared the "attn_mask " also in the pixelSnail of tensorflow version. The corresponding code is shown below: attn_mask = torch.tril(torch.ones(1,1,HW,HW), diagonal=-1).byte() # 1s below diagonal -- attend to context only
so, i hope you that you can give me some hint to understand it ! Thank you for your reply!!!
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.
LonglongaaaGo
Hi! thank you for your good project! i think your projectis very helpful and beautiful ! Now i am confused by the "attn_mask " in pixelSnail. Besides, i have awared the "attn_mask " also in the pixelSnail of tensorflow version. The corresponding code is shown below: attn_mask = torch.tril(torch.ones(1,1,HW,HW), diagonal=-1).byte() # 1s below diagonal -- attend to context only
so, i hope you that you can give me some hint to understand it ! Thank you for your reply!!!