mrzaizai2k
commented
2 years ago First, I don't know much about the attention, but I'll try to explain to you what I understand This is the source code of attention that I used. It's in Vietnamese so pls use google translate: https://pbcquoc.github.io/vietnamese-ocr/
- Yes it's self-attention. The weights are updated parallelly with CRNN
- Attention: https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html As in seq-2-seq project, you can see that without attention the path encoder connected to decoder is one path. which means information standing far away from each other (the model or LSTM can not remember the information so far away). In my project, CNN is like encoder and RNN is like decoder. It doesn't make sense to use attention after RNN
- for a,c and d I guess it's because of tensorflow (the code is not important). For e) multiply is a x h You can mail me directly through bao.mai@thefuturespast.org
jjsr
Sir the explanation is awesome. Thanks Can you tell please more about attention- 1) What type of attention it is ? Is it self attention (giving the weighted sum of attention vector at every time step) 2) What is the reason for applying it just after CNN not apply it after RNN 3) Sir in code you have done the following steps regarding attention - a) Permute b) Dense layer c) Mean (dimensionality reduction) d) Repeat vector 4) Layers multiply
It would be great help sir . I can understand dense layer is implementing tanH function but unable to understand the reason behind other operations
May I mail you sir .. Thank you in advance . I was unable to understand the reasons behind the code from quite few time