search

tbwxmu / SAMPN

45 stars 14 forks source link

Weights in the Attention Layer? #1

Closed JanoschMenke closed 4 years ago

JanoschMenke commented 4 years ago

The weight matrices W_a and W_b in the attention layer are not mentioned in the paper, but present in the code. I am not so familiar with attention layers, but I was wondering if I missed something in the paper or it is common knowledge to apply those weights in an self attention layer?

JanoschMenke commented 4 years ago

No comment?

whymauri commented 4 years ago

I believe W_a and W_b are W_att and E_g, respectively, in the paper. See equations (4) and (5).

https://jcheminf.biomedcentral.com/articles/10.1186/s13321-020-0414-z

JanoschMenke commented 4 years ago

No in the Code 'att_w' refers to the W_att. and 'att_hidden' refers to E_g. The two weight matrices 'W_a' and 'W_b' are not referenced in the Paper. Maybe because it is assumed that people know that Weights are used in attention, but I feel like this should not be assumed in a Cheminformatics journal.

whymauri commented 4 years ago

Sorry, I got had by poor naming convention in the paper.

In the code, the W_x convention generally means the network weight parameters as a Torch layer. But in the paper, W_att is actually not neural network weight parameters, also called a weight matrix, but the "attention score matrix" instead. att_hiddens is certainly E_G, this is true (although it is just a vector).

So this hasn't enlightened much - why have W_a and W_b? W_a is clear to me: it makes the attention differentiable and therefore "learnable" via the W_a layer weight parameters.

Why must E_g pass through W_b? I don't quite know, but we know what it does:

  1. Bounds all "attention weighted hidden vector," which are really just entries of a vector, above zero.

  2. Adds regularization via dropout.

  3. Makes the attention vector learnable before concatenating with the molecular graph's latent representation.

Why (1-3) are desirable, especially 3, I'm unsure. However, I'm fairly certain W_a is necessary. I agree that the paper should have clarified that soft attention as leveraged in the paper is differentiable and learnable.

Cheers.

JanoschMenke commented 4 years ago

I think W_b just add another transformation but the question if its required. What I also do not understand is that line mol_vec = (cur_hiddens + att_hiddens).

Here att_hiddens= torch.matmul(att_w, cur_hiddens) So the Attention weights are already applied to the activation. Why do I add the activation again to the attention transformed actviations.

It looks kind of like a skip-connection. I think its not intuitively understanable tho, why a skip connection is needed. Why scale teh activation based on attention and then just add the unscaled attention again to it.

tbwxmu commented 4 years ago

Sorry for the late reply. I am busy with my graduation. I have read all your guy's comments. There is no conflict between our paper and source codes. Our paper is just a brief description of the codes. So, please refer to the source codes when you feel conflicted. The code here is optimized for our prediction tasks based on my experience and intuition. If you read our codes carefully, you should note we also use the skip-connection in the message passing steps. Of course, you can change or choose different attention algorithms as there are several variants published by others.

JanoschMenke commented 4 years ago

Hi Thank for the clarification.

Good luck with your graduation.