Closed charlenellll closed 1 year ago
I think this may be related to the initialization of coverage attention. I also tried to add train_from: model/1-trm-copy-quickstart/model_step_25000.pt
in the config file to train from existed checkpoint, and this problem still exists.
Or maybe there is something wrong with my configurations. In this case, I hope there could be some guide about how to set the model to use coverage mechanism correctly.
Please let me know if you have any idea about this, thanks!
@pltrdy any idea on this one?
Actually the coverage mechanism isn't implemented for transformer decoders. Coverage comes from See 2018 which is based on RNNs instead (LSTM actually), therefore a single attention head.
It's not clear to me how such a coverage mechanism should behave in a multi-head situation such as transformer decoders (i.e. calculate coverage on all heads? a single head? not sure what to expect from that).
We should at least raise a clearer error in this case.
@charlenellll tell me if you have a precise idea on how the coverage should behave in this case.
@pltrdy I found a paper about using coverage on multi-head attention, in part 3.4: https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012004/pdf But I'm not sure about its effect. Looking forward to your opinion.
Well at least it gives some guidelines to implement coverage in the Transformer. Feel free to implement this paper and open a PR we would review it.
Results show some small improvements in terms of copy and overall quality. However, they compete over Gigaword against seq2seq and vanilla transformer which are not super strong baseline to be honest.
Actually the coverage mechanism isn't implemented for transformer decoders. Coverage comes from See 2018 which is based on RNNs instead (LSTM actually), therefore a single attention head.
It's not clear to me how such a coverage mechanism should behave in a multi-head situation such as transformer decoders (i.e. calculate coverage on all heads? a single head? not sure what to expect from that).
We should at least raise a clearer error in this case.
@charlenellll tell me if you have a precise idea on how the coverage should behave in this case.
Hi, Your consideration is the same as I was thinking, can the replication results on Transformer achieve positive gains without a coverage mechanism ?
@Qnlp Absolutely. And better results as well. Transformers has many heads, has encoder, decoder AND cross-attention (instead of a single cross-attention layer in RNNs) so it may generalize the concept of coverage without trouble.
The fact that no explicit coverage mechanism has been developed for transformers is because they do not really need it.
Hi, I am using OpenNMT-py==2.0.0rc1, and tried to use coverage mechanism while doing copy mechanism. I have tried to add coverage_attn: 'true' and lambda_coverage: 0.2 in my config yaml file, or add -coverage_attn -lambda_coverage 0.2 after the command when training, but either way leads to this error:
My full configs is as follow, I would be grateful if you can tell me where the problem is.